Strange Issue

[seannicholas71]

We have a customer that has 5 sites.. .main site with the headquarters server and 4 sites that are all connected via VPN. The remote sites do not have distributed servers, they just have two shoregear 50's and 30's with some analog loop lines. Network is setup with All Adtran 1335's connecting via VPN to each other. Here is the issue:

All sites and call each other extension to extension, until an Adtran needs to be rebooted or the Internet at a remote site goes down, than back up, all of a sudden that site cannot extension dial other sites except the main site. To fix the issues, the ShoreGear switches need to be rebooted, than all is well again. All pinging and routing remain the same, so what could be causing this?

We are on the latest release of 8.1, but we started out on version 8.0 and upgrading has not fixed the issue.

[pathfinder_nick]

I'd also be interested to know what others have to say about these kinds of issues. We have a Cisco VPN setup with our small ShoreTel system right now (Plan to expand the ShoreTel system in the future) and sometimes have weird issues like this.

[eazeaz]

Vpn

A lot of times access lists or vpns can have references to allow "established" traffic. Rebooting the switch may cause the connection to be setup and allowed. It may not work the other way.

If you have access lists, can you temporarily allow all traffic through the tunnels?

I have also seen some key timeouts cause strange problems. I normally raise the key re-negotiation timeout to a much higher than standard setting.

It sounds like once the tunnel goes down, traffic can only flow again after the shoregear switch initiates the traffic/tunnel flow.

[southeasternMI]

What is happening is the SG switch are building the backup routes and not releasing those routes when the main connection comes up.
When this happens again, telenet to the SG switch and routeShow and you will see the route of the backup connection, but not the main route and that there lies the issue.
So what you can do is routeAdd of the main route to the table and then you can delete the old route.
That is the only way to fix this issue with out rebooting equipment. The SG switches aren't releasing routes as they should and trying to delete the routes won't work if the SG switches see the route as being good. So just add the good route and then delete the old. I pretty sure ShoreTel knows this issue exists.

[seannicholas71]

Okay, I will try that. but, just another tidbit of information. even though i can't do extension to extension dialing across sites, quicklook in director shows everything up and running. I can ping every switch from every switch doing the lsp_ping. All communication tests look as if everything should work just fine.

I will check the routes on the switches next time. if I add and delete a route, will that be a persistent route that sticks?

[southeasternMI]

What happens when the internet goes down at a remote site, do the remote sites have a backup connection?
The routes will stick until you lose internet connection again. Its not a fix all, but it will prevent you from rebooting the switches. What I think may be happening is your layer 3 devices are sending redirects to your SG switches when your links go down. For some reason the SG switches don't release the old routes because they see the routes as viable.

[seannicholas71]

no, they don't have a backup connection, so really the routes stay the same, which is where my confusion lies.

[cburgy]

What does lspConList show when this scenario is occurisng?