ShoreTel Switch Routing Tables

[mattwray]

How do you check routing tables on the switches and how often do they update?

We have redundant WAN links. Our primary WAN is MPLS with VPN backups at every site. If site A's MPLS goes down it automatically kicks to VPN until the MPLS comes back up.

This works flawlessly for everything EXCEPT ShoreTel. The ShoreTel switches will switch to VPN if the MPLS goes down, but they do not come back to the MPLS without a reboot when it comes back up. This causes communication errors.

Now that I think about it, it only affects ST switch to switch communication. The server can ping all the switches, as can I from my desk. But if you telnet to a switch, they cannot ping to the other switches across the WAN.

Any thoughts?

[cburgy]

How do you check routing tables on the switches and how often do they update?

We have redundant WAN links. Our primary WAN is MPLS with VPN backups at every site. If site A's MPLS goes down it automatically kicks to VPN until the MPLS comes back up.

This works flawlessly for everything EXCEPT ShoreTel. The ShoreTel switches will switch to VPN if the MPLS goes down, but they do not come back to the MPLS without a reboot when it comes back up. This causes communication errors.

Now that I think about it, it only affects ST switch to switch communication. The server can ping all the switches, as can I from my desk. But if you telnet to a switch, they cannot ping to the other switches across the WAN.

Any thoughts?

Enable telnet on the switches and telnet to them. The following are the network/IP commands you can run:

ifShow - Displays the current configured
network parameters.

ipstatShow - Displays IP statistics.
tcpstatShow - Displays TCP statistics.
udpstatShow - Displays UDP statistics.
icmpstatShow - Displays ICMP statistics.
arpShow and arptabShow - Displays the ARP table.
routestatShow - Displays routing statistics.
routeShow - Displays current routing table.
hostShow - Displays the known hosts.

[mattwray]

Enable telnet on the switches and telnet to them. The following are the network/IP commands you can run:

ifShow - Displays the current configured
network parameters.

ipstatShow - Displays IP statistics.
tcpstatShow - Displays TCP statistics.
udpstatShow - Displays UDP statistics.
icmpstatShow - Displays ICMP statistics.
arpShow and arptabShow - Displays the ARP table.
routestatShow - Displays routing statistics.
routeShow - Displays current routing table.
hostShow - Displays the known hosts.

Awesome! Thanks a ton, that is exactly what I was looking for

[mattwray]

Ok, I got this back from my vendor

Matt,


I called ShoreTel, and they said that the switch will not look for another source of connectivity as long as it has a stable connection. So when the MPLS goes down, it will switch to the VPN. But when the MPLS comes back up, it will not switch back because the VPN connection is up and stable. Let me know if you have any questions.

Thanks

Does anyone know, or can think of, a workaround to this situation? The phones and all other equipment take the correct path, just the switches refuse to switch back over to the MPLS.

[devdevil85]

You would think that there would be a default, prioritized connection that ST would always connect to if it were available. Anyone have any answers to his question? I'm wondering why this is myself.

[400degreez]

perhaps a icmp redirect will help, they break switches when they are enabled by changing the switches host route tables. Perhaps if you reverse that and create a scenario that sends the redirect whenever the MPLS comes back online.

[southeasternMI]

There is a couple ways to quick fix. If you reboot the switch when the MPLS comes backup that should fix it, but takes to long and will bring phones down. The second is to add the proper route to the switch. So just telnet to the switch use the command routeAdd "172.0.0.x","192.168.7.x" this will add the proper route of the MPLS and will fix your issue. You don't have to delete the old route just add the new.

[cburgy]

Just disable ICMP Redirects on your switching infrastructure or routers. Won't be a problem....

Or take your firewalls/routers out of the same broadcast domain as the SG switches. Besides, proper network design really dictates those devices should be in a management VLAN any way.

[sschnei]

question.... What is making the decision to use the MPLS or VPN path. Are they using a routing protocol that updates route tables somewhere based a path being up or down, or are they using some other method. I suspect that the ShoreGear switch route tables will not change and he is depending on his default gateway to make any routing decisions.

[cburgy]

The problem is ICMP Redirects. That is what causes the SG switch to be presented with an alternate path. ICMP Redirects are issued when there is a quicker path (less hops) that exist within the same broadcast domain.

[400degreez]

I dont think your problem is ICMP redirects. You could add a router or VLAN in there and use dynamic routing, turn off ICMP redirects and allow the router to make the decision on which gateway it chooses to forward traffic to. Your switches will have a permanent gateway. ICMP redirects can be used in this scenario if you know what you are doing. They dont break the system. They follow orders therefore you still could use this for your DR. You may complex things and people usually dont because of security. Last thing, you should be running a trace on your traffic to get a complete picture of what is going on.

400Degreez.....my time to shine!

[sean]

Is this redundant solution an active active or an active passive setup?

[southeasternMI]

redirects are usually the cause. The default Gatway (route) won't change but redirects are sent to the SG and the host route table is built, which is always SG switches because servers can't recieve redirects. Thats why its always SG to SG that lose connectivity. If you look at the host table you should see the backup route that was sent from the redirect for that particular SG. The SG switches are layer 2, so there is no other way for the SG switch to build routes except for ICMP redirects.

[400degreez]

So your question is, how do you get the switches to revert back to the MPLS gateway from the VPN Gateway after an outage, correct? Put a router in front of them.

[cburgy]

Again, if you have seperate devices terminating your VPN and MPLS connections (or if your VPN device is in the same broadcast domain as the L3 interface of your switch) that reside in the same broadcast domains as your SG switch(es), you need to disable ICMP redirects. You will still have the same problem.

Since there seems to be some confusion on ICMP Redirects and in the interest of saving me time of explaining it, I've attached a Shoretel technical note that finally released on the issue some time back. As you can imagine, a number of companies don't understand network design well and this problem became quite more apparent over the past few years.