Your slow telnet response could also be caused by poor DNS
configuration on your DNS servers and their lack of reverse resolution.
I've seen it a ton of times where telnetd's trying to reverse resolve
the client who is trying to attach. It won't do it all the time which
makes it seem strange. A simple way to test it to just move your
/etc/resolv.conf file to a backup file name and see if you still see
the slow connect times on telnet.