Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Connection Issue with 90BRI Switch

    Our shoregear 90BRI Switch has lost communication with our HQ server. The 2 devices are seperated by a vpn tunnel created between 2 soniwall pro 2040 firewall.

    I have looked on both Shoretel and Soincwall Forums and there is some information saying that the 2 technologies do not mix very well. Unfortunately there is no clear indication of a fix

    In any case I can telenet and ping the switch from the server, but it seems tcp communication is being refused by the Sonicwall Firewall on port 1024 and upwards.

    Does anyone know what the below output means. After telneting to the SG Switch this output is continually getting displayed


    ShoreTel> DIAG: src=MsCtrl, eventid=0xc100006c , str=(LSP_KEEP_ALIVE 10.4.45.2 s
    um 0x00000582 != 0x00000580 :: resync)
    DIAG: src=MsCtrl, eventid=0xc100006c , str=(LSP_KEEP_ALIVE 10.4.20.9 sum 0x00000
    582 != 0x00000580 :: resync)
    DIAG: src=MsCtrl, eventid=0xc100006c , str=(LSP_KEEP_ALIVE 10.4.45.1 sum 0x00000
    582 != 0x00000580 :: resync)
    DIAG: src=MsCtrl, eventid=0xc100006c , str=(LSP_KEEP_ALIVE 10.4.45.3 sum 0x00000
    582 != 0x00000580 :: resync)
    DIAG: src=Switch, eventid=0xc1000074 , str=(Chiswick 60/12, )
    DIAG: src=Switch, eventid=0xc1000074 , str=(ShoreGear E1, )
    DIAG: src=Switch, eventid=0xc1000074 , str=(Chiswick 120/24, )
    DIAG: src=Switch, eventid=0xc1000074 , str=(SoftSwitch, )
    DIAG: src=Switch, eventid=0x41000075 , str=(SoftSwitch)
    DIAG: src=Switch, eventid=0x41000075 , str=(Chiswick 60/12)
    DIAG: src=Switch, eventid=0x41000075 , str=(ShoreGear E1)
    DIAG: src=Switch, eventid=0x41000075 , str=(Chiswick 120/24)

  • #2
    Originally posted by tyrone powell View Post
    Our shoregear 90BRI Switch has lost communication with our HQ server. The 2 devices are seperated by a vpn tunnel created between 2 soniwall pro 2040 firewall.

    I have looked on both Shoretel and Soincwall Forums and there is some information saying that the 2 technologies do not mix very well. Unfortunately there is no clear indication of a fix

    In any case I can telenet and ping the switch from the server, but it seems tcp communication is being refused by the Sonicwall Firewall on port 1024 and upwards.

    Does anyone know what the below output means. After telneting to the SG Switch this output is continually getting displayed


    ShoreTel> DIAG: src=MsCtrl, eventid=0xc100006c , str=(LSP_KEEP_ALIVE 10.4.45.2 s
    um 0x00000582 != 0x00000580 :: resync)
    DIAG: src=MsCtrl, eventid=0xc100006c , str=(LSP_KEEP_ALIVE 10.4.20.9 sum 0x00000
    582 != 0x00000580 :: resync)
    DIAG: src=MsCtrl, eventid=0xc100006c , str=(LSP_KEEP_ALIVE 10.4.45.1 sum 0x00000
    582 != 0x00000580 :: resync)
    DIAG: src=MsCtrl, eventid=0xc100006c , str=(LSP_KEEP_ALIVE 10.4.45.3 sum 0x00000
    582 != 0x00000580 :: resync)
    DIAG: src=Switch, eventid=0xc1000074 , str=(Chiswick 60/12, )
    DIAG: src=Switch, eventid=0xc1000074 , str=(ShoreGear E1, )
    DIAG: src=Switch, eventid=0xc1000074 , str=(Chiswick 120/24, )
    DIAG: src=Switch, eventid=0xc1000074 , str=(SoftSwitch, )
    DIAG: src=Switch, eventid=0x41000075 , str=(SoftSwitch)
    DIAG: src=Switch, eventid=0x41000075 , str=(Chiswick 60/12)
    DIAG: src=Switch, eventid=0x41000075 , str=(ShoreGear E1)
    DIAG: src=Switch, eventid=0x41000075 , str=(Chiswick 120/24)


    Firewalls can be tricky. When it is occuring, telnet to the switch and run:

    LspConList and see what the connectivity looks like from the SG side.

    If there isn't connectivity, run a lsp_ping "IP_of_other_sg"

    This generates traffic at layer 7 from the SG switch to another. If it doesn't make it, the firewall is blocking something...

    Watch how often your VPN tunnel renegoitiates....

    Comment


    • #3
      Thanks Chris

      Are these commands valid on a SG-90BRI

      The reason I ask is that whatever command I type after login onto the switch via a telenet session I receive "invalid command". I am only presented with these valid options

      1. Show Version
      2. Show Sytem Configuration
      3 Change System Configuration
      4. Reboot
      5 Logout
      Help
      Abort Command

      Thanks

      Tyrone

      Comment


      • #4
        type in:

        gotoShell

        Comment


        • #5
          If it is not connecting running lsp_ping to the SG's in question will tell you a lot as well.

          what you wil want to do is set lsp debug first

          So for each SG set debug as follows

          lsp_debug_level = 4 (TURN OFF Debug with = 0)

          From SG-1 do
          lsp_ping "IP ADDRESS OF SG-2" 100 (In Quotes as Seen, The 100 is the amount of packets, it will send 1000 by default)

          what you will see in Debug is 100 Sends / 100 Receives on each SG

          So lets say You send from SG1 to SG2
          SG2 will display Receive first followed by a send back to SG1
          If you are not seeing any receive, your ports are blocked or routing is not correct.

          If you see Receive and then the Send back to SG1 BUT
          you see only Send from SG1 and not a Receive, then your ports are blocked or routing is not correct.

          You can see where the block is occuring by who is recieving and or sending, it must be a complete circle. To test this, you can also lsp_ping the HQ Server or DVM, turning Debug on in the Server is found in the maint manual
          Last edited by Jlorenz; 02-05-2009, 10:33 AM.

          Comment


          • #6
            Thanks to both of you for your help It seems like the the Problematic switch has been able to ping and receive data from other switches and the HQ server. Using the default 1000 pings it reported that 23 packets were missing. to/from the HQ server and a similar number to/from another SG Switch.

            I am receiving this error when telnetting to the switch with the connectivity issue


            NEC_CONNECT from 10.4.20.9, prior was 10.4.20.9
            tmsd_svc_tms_connection_down(tmsd_svc_run returned, 1): 1 relay_pending 1380
            tmsd_event_pipe_flush()
            ERROR: src=TMSD, eventid=0xc100006c , str=(tmsd_process_necevtproc(NECEVTPROC_SY
            STEM_EVENT)[SNE_ETHER_MODE]; RPC: Unable to receive (Error 4)

            Also I cannot get the LspConList to work I am receiving the following

            C interp: unknown symbol name 'LspConList'.

            Everything points to a routing issue, however I have checked with the firewall/routing config and all is in order. I have also escalated this to our Ferewall Support co and they cannot see anything wrong.

            I'll keep plugging away.

            Comment


            • #7
              [QUOTE=tyrone powell;8315]Also I cannot get the LspConList to work I am receiving the following

              C interp: unknown symbol name 'LspConList'.
              QUOTE]

              try :
              lspConList

              (lower case L at the start)

              Comment


              • #8
                You have a network issue, be it latency, routing or ? there is clearly an issue. The "RPC: Unable to receive (Error 4) is RPC port UDP and TCP 111.

                do a routeShow form the command line, and paste the results/

                Also as ST Dave stated it is lspConList

                And 23 packets is enough packets to loose a connection
                Originally posted by tyrone powell View Post
                Thanks to both of you for your help It seems like the the Problematic switch has been able to ping and receive data from other switches and the HQ server. Using the default 1000 pings it reported that 23 packets were missing. to/from the HQ server and a similar number to/from another SG Switch.

                I am receiving this error when telnetting to the switch with the connectivity issue


                NEC_CONNECT from 10.4.20.9, prior was 10.4.20.9
                tmsd_svc_tms_connection_down(tmsd_svc_run returned, 1): 1 relay_pending 1380
                tmsd_event_pipe_flush()
                ERROR: src=TMSD, eventid=0xc100006c , str=(tmsd_process_necevtproc(NECEVTPROC_SY
                STEM_EVENT)[SNE_ETHER_MODE]; RPC: Unable to receive (Error 4)

                Also I cannot get the LspConList to work I am receiving the following

                C interp: unknown symbol name 'LspConList'.

                Everything points to a routing issue, however I have checked with the firewall/routing config and all is in order. I have also escalated this to our Ferewall Support co and they cannot see anything wrong.

                I'll keep plugging away.

                Comment


                • #9
                  Thanks again guys. I agree it is a firewall/routing issue however there really is nothing that is blocking/preventing the connection according to the configuration of the firewalls at both sites.

                  We have even explicitly allowed the relevant services/ports to and from each site across each of the SonicWall devices.
                  Sonicwall Support can see no issues

                  In any case here is a printout of the lspConList command and routeShow from the switch

                  These commands are extremely useful for troubleshooting - is there a list of command that can be run on the switches?

                  -> lspConList
                  LIST OF CONNECTIONS
                  session for <Chiswick 60/12> 10.4.45.2 - ACTIVE, STATE_IN_SYNC
                  session for <ShoreGear E1> 10.4.45.3 - ACTIVE, STATE_IN_SYNC
                  session for <Chiswick 120/24> 10.4.45.1 - ACTIVE, STATE_IN_SYNC
                  session for <SoftSwitch> 10.4.20.9 - ACTIVE, STATE_IN_SYNC
                  value = 0 = 0x0



                  '.
                  -> routeShow

                  ROUTE NET TABLE
                  destination gateway flags Refcnt Use Interface
                  ----------------------------------------------------------------------------
                  0.0.0.0 10.5.45.254 33619971 5 5 emac0
                  10.5.45.0 10.5.45.4 33554689 2 0 emac0
                  ----------------------------------------------------------------------------

                  ROUTE HOST TABLE
                  destination gateway flags Refcnt Use Interface
                  ----------------------------------------------------------------------------
                  10.4.20.9 10.5.45.254 33685511 2 765 emac0
                  10.4.45.1 10.5.45.254 33947655 0 571 emac0
                  10.4.45.2 10.5.45.254 33947655 0 216 emac0
                  10.4.45.3 10.5.45.254 33947655 0 128 emac0
                  127.0.0.1 127.0.0.1 35651589 0 0 lo0

                  Comment


                  • #10
                    Ok here is your issue

                    Your route show is not correct. Do you have ICMP redirect enabled on your Core Network?

                    In your IPBX log, search for delete and see if there are any route deletes

                    -> routeShow

                    ROUTE NET TABLE
                    destination gateway flags Refcnt Use Interface
                    ----------------------------------------------------------------------------
                    0.0.0.0 10.5.45.254 33619971 5 5 emac0
                    10.5.45.0 10.5.45.4 33554689 2 0 emac0
                    ----------------------------------------------------------------------------

                    ROUTE HOST TABLE
                    destination gateway flags Refcnt Use Interface
                    ----------------------------------------------------------------------------
                    10.4.20.9 10.5.45.254 33685511 2 765 emac0
                    10.4.45.1 10.5.45.254 33947655 0 571 emac0
                    10.4.45.2 10.5.45.254 33947655 0 216 emac0
                    10.4.45.3 10.5.45.254 33947655 0 128 emac0
                    127.0.0.1 127.0.0.1 35651589 0 0 lo0
                    Last edited by Jlorenz; 02-06-2009, 06:44 AM.

                    Comment


                    • #11
                      I'm not sure if we have icmp redirects turned on our Core Network (I assume you mean our Core Switches). I believe icmp redirects should be disabled on the HQ server VIa the registry.

                      1. How can I tell if ICMP redirects are enabled on the network, If they are could this also cause latency issues between sites? How can I turn them off?

                      2. How can I view the ipbx logs?

                      Both you and Dave have been more help than any of our support contracts

                      Thanks so much

                      Tyrone

                      Comment


                      • #12
                        Also have checked with our networking guy - ICMP redirects have been disabled on both the layer 3 Cisco switches (1 at each site)

                        Comment


                        • #13
                          Our Firewall support company have taken several packet captures to and from the server and affected switch.

                          The capture indicates that it is the 90BRI switch that is apparently resetting the connection to the server. This has been shown by a wireshark capture.

                          Does the SG90 BRI have the ability to do this?

                          Comment


                          • #14
                            Checking in the IPBX Logs Where X = install directory
                            X:\Shoreline Data\logs

                            Find the latest IPBX log they are dated and time stamped, best method is to click the data modified to sort by date.

                            Open the File in any text editor (Not Do not SAVE ON CLOSE)

                            Search for Route Delete or Delete see if you find any instances of a Route delete ..... 15 minutes I do not remember the exact syntax.


                            SG Reseting the either

                            Have you replaced the SG? if yes why would it continue then, is the next SG bad as well? The SG does not reset the Port (On purpose or accident, unless its defective)

                            Checking for Switch connectivity
                            The LSP to each switch is a keep alive, it will ping every few seconds and hammer it with requests. If these requests are not answered, then the Shoreware Directory says the SG is OOS or Not Connected. I gave an example of the way you should test for connectivity between SG;s

                            I know the argument that the network guys did there check and everything is good. Believe me, something is not good, this does not mean the SG is not at fault, Please run the tests I described in a few posts up. If you need any assistance, I would be more then happy to assist.

                            /On topic joke Reminder "Its OK on my end"
                            Old Bell Tech gets drafted
                            He is on the shooting range and getting screamed at by the DI
                            "HIT THE TARGET MAGGOT"

                            Bell Tech says, I am sir, I am hitting the target, May I do a test SIR

                            DI SAY, HIT THE TARGET MAGGOT AND DO THE TEST

                            Old Bell Tech licks his finger, puts it in the end of the gun aims and pulls the trigger, blowing his finger off. He smiles at the DI and says, Its ok on my end SIR :gun_bandana:

                            Comment


                            • #15
                              JLorenz

                              1. There is no mention of Route Delete or Delete in the ipbx logs on the Server

                              2. We have tried a second 90BRI which displays the exact symptons.
                              The 90 BRI does works when connected on the network where the Server
                              resides (ie traffic not travelling across our sonicwall firewall devices)

                              3. I have attached copies of the LSP_PINGS to and from the problematic switch (10.5.45.100) 90 bri. I ran the ping to an E1 switch (10.4.45.3) whch is also based at the same site as the HQ server.

                              Ping has not been a problem to or from the server/switch

                              If you think of anything else, let me know.

                              Thankyou
                              Attached Files

                              Comment

                              Working...
                              X