Network Busy - and I don't know why

[JasonC]

More then occasionally, our people will get a "Network Busy" message when trying to make a call. This could be internal extension to extension dialing, or calling out bound to an external number. Even with user's calling user's at the same site.

As far as the network infrastructure goes, everything is cisco, and is monitored for bandwidth usage. The HQ site is running gigabit network, and the sites have at least 1 T1, with some having 2 bounded. The network team says bandwidth never reach's over 75% for the sites on T1's. Our HQ site normally only runs at less then 1% capacity. Almost everything is home-runed to a Cisco 6509. With our server vlan, the backplain on that is 700Gb, but we're only maxing at 9 Gb's. The network team assures me everything on the network side is runnning strong.

Our IP Phones in use is about 323, while the capacity is 625.

HQ: 240 user's
2 SG-T1k & SG-T1
2 SG-220
1 SG-90
1 SG-24a

Site#1:
20 user's, 1 SG50, & SGt1k

Site's #2 thru 5:
approx 15 user's, 1 SG50 & SG-t1k, (EACH site)

Site's 6 thru 9:
approx 10 users, SG50 or SG90 (EACH site)

8 Other locations:
4 - 7 user's each (still connected with T1)

(the user's per site add's up to exceed the total number of user's because user's will migrate from site to site).

In the Director, all the sites bandwidth is set to 768 kb.

Even though the bandwidth is set to 768kb between sites, I don't understand why user's at the HQ calling other user's at the HQ get a network busy.

Does anyone know the total number of connections a director server can handle? Our voicemail server is pretty beefy, daulcore xeon with 4 GB ram & 2 gigabit cards.

Any Idea's?

Thanks,
J

[southeasternMI]

Mirror a phone port and do a couple of captures, one good and one bad.
Compare the two.

[Contractor]

You didn't post the codec(s) you are using.

The only bandwidth in question is the admission control bandwidth. Not real bandwidth.

One possibility is that you are actually using all that admission control bandwidth by using alot of applications that need the HQ server or something of that ilk.

The other possibility is that one of your switches isn't releasing the admission control bandwidth after the call is terminated.

Does the issue fix itself or do you have to do something to "fix" the issue? What happens if you up the admission control bandwidth at the HQ site?

[eazeaz]

HQ

I would up the Admission control on the Headquarters site.

You have 17 sites all competing for a total of 768K from the HQ site? 45k per site total concurrent?

I agree that one would not think that a HQ to HQ call would have a problem due to admission control.

It is a simple change, and its probably a good idea anyway.

[JasonC]

Well, it's not just 1 port, and never seems to be the same port either. The problem goes away on it's own, and sometimes I get it while trying to access the voicemail server, which is at the HQ with me.

Intrasite is set to Medium Bandwidth codecs
Intersite is set to Very Low bandwidth codecs

When asked about applications, are you refering to the Call Manager, or Call Control groups, or other stuff using ECC? We don't use contact center at the moment.

I've doubled the HQ admission control bandwidth to 1536 kb. Should I raise this more to allow each site it's own 768 kb to the HQ?

Since this email, an offsite location emailed me saying they get Network busy often. They are remote with their own SG50 and T1k. When 3 or more people are on the phone, and they try to use a paging group( w/4 people), it will say network busy until the 3rd person get's off the phone. Does this mean the paging groups resides on the Director server and not the local SG switch?

[eazeaz]

HQ

Do all of your Sites T1's come into your HQ site directly?

If so you could realistically set your HQ site to 10000 in/out and not be going too crazy.

With your configuration, I would surely think that you could get away from the VERY LOW codecs between sites as well. Your bottleneck is probably the admission control to the HQ site itself, and once that is increased, the codec could probably get better as well. Sounds like you have the bandwith, it is just restricted currently.

Doesnt admission control log an event to the event viewer on the HQ server? Search the logs and see what you find. I wouldnt be surprised if you find a ton of clipping events.

[JasonC]

Yes, each T1 comes directly into our HQ (over DS3) into Cisco7204. I went ahead and raised the ACB to 5000 kb.

We don't actually have 17 "sites". The small remote locations are configured in the IP PHONE ADDRESS MAP to piggy back off of other sites. Those small remote locations T1's goes to the HQ site too, and not to the sites they are associated with. Strangely, none of those sites have reported any issues.

I haven't checked the logs, but I know Shoretel has gone through our logs 3 different times for different issues over the last month, but no one their mentioned anything about clipping. I'll go though tonight to see if I can find anything.

[eazeaz]

"extra locations"

so your sites:

8 Other locations:
4 - 7 user's each (still connected with T1)

dont have any shoregear switches?

If they dont, I wouldnt have them piggy back off of another remote site.

The data would be going from the small site without a shoregear switch, up the T1 to the headquarters, then back out another different t1 to a remote site with a switch? That is asking for problems.

can you confirm that the 8 "other" locations have no shoretel switches and that those phones go to another "remote" location via a path that takes them through the HQ? There is a lot of lost bandwith there.

[Jason Learmouth]

If there are phones only at a site then make sure there is an IP Phone Address Map for those phones that is set to teleworkers to use the intersite codec to those phones.

[JasonC]

Yes "other" locations have no shoretel switch's, and they do piggy back off another site via T1's going through the HQ. Funny thing is, those phones haven't had issues since QOS was fixed. Eventually everything will be on NPLS.