TMS and talking Telnet to Classic 6000 Codecs

[Shawn Jones]

I have been working with TB tech support on this issue for almost a month, and I have them stumped. The systems operate as usual to the end-user.

My TMS server has been up since May 1, 2006. Around June 1st, I had five Classic 6k codecs, all running B10.0, stop responding to Telnet commands. For the next week, I could contact the unit via HTTP, but with limited usability. After the initial week or so, they stopped responding via HTTP.

The units will ping without issues. I have checked with my network administrators, and there are no Telnet or HTTP ports being blocked to these units. All units are on the same physical network, just different subnets. We do have a network security team that has a server that goes out to all IP addresses and checks for vulneralibilities via FTP, Telnet, etc. I did see the security server hit the IP addresses of the units just prior to them going offline.

If I reboot the unit(s) the issue goes away, but recurs about two months later. This issue seems to only be happening with my 6k classic codecs, the MXP 6k’s see the traffic, but don’t seem to be affected.

Anyone have this issue and like to share their insight or solutions?

Thanks for your help.

[SeanC]

I have been having a similar issue with http access to our codecs. Mostly it has been the B10 and some E5 series codecs. The problem was first noticed back in January after we had upgraded our codecs to the B10 and E5 software. It has been reported to Tandberg but I think they are stumped, as I am still awaiting an answer to this issue.

Cheers,
Sean

[Shawn Jones]

Do you find that it is just certain units doing this? It seems to be the same five or six doing it for me. Don't know if there is a correlation to serial numbers or not...

[SeanC]

It seems to be narrowed down to our systems that are running B10 or E5 software. Now with that being said there are a couple of units that seem to be more problematic then others. Interesting thought about the serial numbers.

As a side note to this I also had this same problem with the MPS that we have. After speaking with Tandberg we upgraded to J3.1 and so far we have not had this issue. I am still monitoring it closely though. I will keep you posted.

Cheers,
Sean

[Jimmy]

Try to downgrade your system to B9.x and see if the fault goes away then upgrade to 10.2 and see if it reoccurs

[Shawn Jones]

What is baffling to me, is that there are only 5 codecs on the network doing this. Oh, and it takes approximately two months for this issue to recur, almost like clockwork.

I have upwards of 20 Classic 6k's, all running B10.0 or higher. :tired:

That's why I am wondering if there is a correlation to serial numbers, perhaps a different chipset was used with the more recent units...

[SeanC]

We have 130 systems and this issue arises with the 2500's and 2 of our 880's. I have downgraded and then upgraded and the issue is still there. Also after we reboot, the system works the way it should but then later in the day we run into the same problem again.

Cheers,
Sean

[Shawn Jones]

Are you running TMS, or another suite where you can check trap logs to see what is happening with the systems?

I think that my primary problem is with our security team's vunerability server querying the 6k's. The system will usually stop responding to telnet following this port scan, according to my trap logs captured by TMS.

[SeanC]

I have both TMS 10 and 11 running as we are migrating from 10 to 11. Trap logs do not show a whole lot in the way of errors. I can ping the systems with <1ms response time and I can telnet to them.

[Sean Lessman]

Are you running TMS, or another suite where you can check trap logs to see what is happening with the systems?

I think that my primary problem is with our security team's vunerability server querying the 6k's. The system will usually stop responding to telnet following this port scan, according to my trap logs captured by TMS.

Is it possible your security scan is opening up telnet sessions with the codecs? They only accept a limited number of telnet connections. When it stops responding, connect a RS232 cable to the codec and issue the 'netstat' command to see how many telnet sessions are open to the codec.

Sean

[Shawn Jones]

The security scans try and open at least four sessions within a matter of seconds. This is only happening on about five codecs.

Would that also explain the web page access slowly degrading from useable, to completely unuseable (no response from the codec) in a matter of days?

The codec still sends traps to TMS, just no Telnet, ftp, or http access. Rebooting the codec returns the ports to active status again.

I cannot stop our security department from scanning my codecs either.

[Sean Lessman]

The security scans try and open at least four sessions within a matter of seconds. This is only happening on about five codecs.

I believe the classic codecs only support 2 simultaneous telnet sessions. You can test this by manually opening up multiple telnet sessions from your PC. If this scanning program is doing the same thing and not properly disconnecting the session, it could be possible the sockets are still open and preventing new telnet sessions. This can be verified by connecting a serial RS232 cable to the endpoint and issuing the command 'netstat'. See if you have multiple connections to www.xxx.yyy.zzz:23 in which www.xxx.yyy.zzz is the IP address of your codec.

Would that also explain the web page access slowly degrading from useable, to completely unuseable (no response from the codec) in a matter of days?

I imagine something similar is causing this too. Probably the best way to troubleshoot is packet sniff and look into the traffic on the IP network going to the device.

The codec still sends traps to TMS, just no Telnet, ftp, or http access. Rebooting the codec returns the ports to active status again.

SNMP traps are different than the other protocols. SNMP uses a different port. And in this case it is outbound from the codec, not inbound like telnet, ftp and http.

Sean

[Shawn Jones]

Would the Telnet sessions being opened explain why the codec would slowly begin to lose the Http page interfaces, with them eventually being unable to respond via http until reboot?

Why would only five of the almost 30 classic codecs with the same software rev respond to the scans in this way?

[Sean Lessman]

Would the Telnet sessions being opened explain why the codec would slowly begin to lose the Http page interfaces, with them eventually being unable to respond via http until reboot?

Not directly as Telnet and HTTP really have nothing to do with each other. However, if you find why one of them is happening, it may lead you to the culprit on the other.

Why would only five of the almost 30 classic codecs with the same software rev respond to the scans in this way?

Its all guesses at this point without doing some troubleshooting. A good starting point is to issue that command I gave in a previous post. When the codec is not accepting connections, find out if there are too many connections already (maybe something is not closing its connections) and where those connections are coming from.

Sean

[Shawn Jones]

Copy of the TMS Log:

7/7/2006 6:10:12 PM System Activity Service: ftp Remote Site: Unknown
7/7/2006 6:09:54 PM System Activity Service: ftp Remote Site: Unknown
7/7/2006 6:09:54 PM System Activity Service: ftp Remote Site: Unknown
7/7/2006 6:09:52 PM Authentication Failure Service: ftp Remote Site: 10.128.128.130:46654
7/7/2006 6:09:50 PM Authentication Failure Service: ftp Remote Site: 10.128.128.130:46654
7/7/2006 6:09:48 PM System Activity Service: ftp Remote Site: Unknown
7/7/2006 6:09:48 PM Authentication Failure Service: ftp Remote Site: 10.128.128.130:46654
7/7/2006 6:09:47 PM System Activity Service: ftp Remote Site: Unknown
7/7/2006 6:09:31 PM System Activity Service: ftp Remote Site: Unknown

The codec is not responding to Telnet again, and the web interface is not functioning. The unit is in another building, about 30 minutes away. I will try and get over there tomorrow to see what is going on.

netstat

OK
--- Routing Table ---
Destination Gateway Mask If Refct Met Flags
127.000.000.000 127.000.000.001 255.000.000.000 lo 21 1 up|sil
010.132.070.000 010.132.070.039 255.255.254.000 et 0 1 up
010.132.070.039 127.000.000.001 255.255.255.255 lo 0 1 up|gw|hst
224.000.000.001 127.000.000.001 255.255.255.255 lo 0 1 up|hst|sil
000.000.000.000 010.132.070.001 000.000.000.000 et 40 1 up|gw
--- Open connections ---
Sckt Proto Local address Foreign address TOS State
[0] tcp 10.132.70.39:21 0.0.0.0:0 00 CONNECTING-BL
[1] tcp 10.132.70.39:23 0.0.0.0:0 00 CONNECTING-BL
[2] tcp 10.132.70.39:57 0.0.0.0:0 00 CONNECTING-BL
[3] tcp 10.132.70.39:80 0.0.0.0:0 00 CONN
[4] udp 10.132.70.39:123 127.0.0.1:123 00 CONNECTING-B
[5] udp 10.132.70.39:962 10.236.64.86:162 00 CONNECTING-B
[6] udp 10.132.70.39:161 10.236.64.86:1767 00 CONNECTING-B
[7] tcp 10.132.70.39:1720 0.0.0.0:0 c0 CONNECTING-BL
[8] udp 10.132.70.39:1719 0.0.0.0:0 c0 CONNECTING
[9] tcp 10.132.70.39:80 10.236.64.86:2570 00 CloseWait
[10] tcp 10.132.70.39:80 10.236.64.86:2611 00 DISCONNECTING
[11] tcp 10.132.70.39:80 10.236.64.86:2614 00 DISCONNECTING
[12] tcp 10.132.70.39:80 10.236.64.86:2615 00 CloseWait
[13] tcp 10.132.70.39:80 10.236.64.86:2617 00 CloseWait
[18] tcp 10.132.70.39:80 10.236.64.86:3655 00 CloseWait
[19] tcp 10.132.70.39:80 10.236.64.86:3673 00 CloseWait
[20] tcp 10.132.70.39:80 10.236.64.86:3686 00 DISCONNECTING
[21] tcp 10.132.70.39:80 10.236.64.86:3693 00 CloseWait
[22] tcp 10.132.70.39:80 10.236.64.86:3699 00 CloseWait
[27] tcp 10.132.70.39:80 10.236.64.86:4725 00 CloseWait
[28] tcp 10.132.70.39:80 10.236.64.86:4753 00 DISCONNECTING
[29] tcp 10.132.70.39:80 10.236.64.86:4763 00 DISCONNECTING
[30] tcp 10.132.70.39:80 10.236.64.86:4772 00 DISCONNECTING
[31] tcp 10.132.70.39:80 10.236.64.86:4783 00 DISCONNECTING
[32] tcp 10.132.70.39:80 10.236.64.86:4200 00 CloseWait
[33] tcp 10.132.70.39:80 10.236.64.86:4205 00 DISCONNECTING
[34] tcp 10.132.70.39:80 10.236.64.86:4763 00 DISCONNECTING
[35] tcp 10.132.70.39:80 10.236.64.86:4783 00 DISCONNECTING
[36] tcp 10.132.70.39:80 10.236.64.86:2611 00 DISCONNECTING
[37] tcp 10.132.70.39:80 10.236.64.86:3686 00 CloseWait
[38] tcp 10.132.70.39:80 10.236.64.86:4763 00 CloseWait
[39] tcp 10.132.70.39:80 10.236.64.86:4772 00 CloseWait
[40] tcp 10.132.70.39:80 10.236.64.86:4205 00 DISCONNECTING
[41] tcp 10.132.70.39:80 10.236.64.86:2614 00 DISCONNECTING
[42] tcp 10.132.70.39:23 10.128.128.130:37654 00 CONNECTED-B
[43] tcp 10.132.70.39:80 10.236.64.86:2611 00 CloseWait
[44] tcp 10.132.70.39:80 10.236.64.86:2614 00 CloseWait
[45] tcp 10.132.70.39:80 10.236.64.86:4783 00 CloseWait
[46] tcp 10.132.70.39:80 10.236.64.86:4753 00 DISCONNECTING
[47] tcp 10.132.70.39:80 10.236.64.86:4205 00 CloseWait
[48] tcp 10.132.70.39:80 10.236.64.86:4206 00 CloseWait
[49] tcp 10.132.70.39:80 10.236.64.86:4207 00 CloseWait
[50] tcp 10.132.70.39:80 10.236.64.86:1394 00 CloseWait
[51] tcp 10.132.70.39:80 10.236.64.86:1402 00 CloseWait
[52] tcp 10.132.70.39:80 10.236.64.86:1408 00 CloseWait
[53] tcp 10.132.70.39:80 10.236.64.86:4753 00 CloseWait
[65] tcp 10.132.70.39:80 10.128.196.81:4354 00 CloseWait
[91] tcp 10.132.70.39:23 10.128.128.130:48283 00 CONNECTED-B
[97] tcp 10.132.70.39:80 10.236.64.86:4196 00 CloseWait
[98] tcp 10.132.70.39:80 10.236.64.86:4200 00 DISCONNECTING
[99] tcp 10.132.70.39:80 10.236.64.86:4205 00 DISCONNECTING
[100]tcp 10.132.70.39:80 10.236.64.86:4206 00 DISCONNECTING
[101]tcp 10.132.70.39:80 10.128.196.81:3968 00 CloseWait
[102]tcp 10.132.70.39:23 10.128.128.130:53512 00 CONNECTED-B
[103]tcp 10.132.70.39:80 10.128.196.81:3969 00 CloseWait
[104]tcp 10.132.70.39:80 10.236.64.86:4207 00 DISCONNECTING
[105]tcp 10.132.70.39:80 10.128.196.81:3977 00 CloseWait
[108]tcp 10.132.70.39:80 10.128.196.81:4182 00 CloseWait
[111]tcp 10.132.70.39:80 10.236.64.86:1394 00 DISCONNECTING
[112]tcp 10.132.70.39:80 10.236.64.86:1402 00 DISCONNECTING
[113]tcp 10.132.70.39:80 10.236.64.86:1405 00 CloseWait
[114]tcp 10.132.70.39:80 10.236.64.86:1406 00 CloseWait
[115]tcp 10.132.70.39:80 10.236.64.86:1408 00 DISCONNECTING


Thoughts?