On the Linux boxes that host the application I work on, we've started seeing an issue where connection attempts against a port on a remote host are sometimes taking longer to come back.
As a test, a simple script was set up that just tries to telnet against a non-existent remote port in an endless loop.
On most connection attempts, we would get something like the following back near instantaneously:
telnet remoteHost 12345
Trying 192.168.1.1...
telnet: connect to address 192.168.1.1: Connection refused
However, at a fairly regular interval, the connection seems to hang for several seconds before continuing on to the next connection attempt which will go back to being near instant. It will just sit at the "Trying" stage:
telnet remoteHost 12345
Trying 192.168.1.1...
Normally this wouldn't be that much of an issue, but in an application that does thousands of things every second between ~100 servers, these random slow downs are starting to become a problem. Does anyone know what drives this length or time or why it might be inconsistent?
In case it matters, here's some system info:
uname -r -> 2.6.16.60-0.39.3-smp
uname -m -> x86_64
cat /etc/*-release -> SUSE Linux Enterprise Server 10 (x86_64)
VERSION = 10
PATCHLEVEL = 2
tcp_syncookiesflag set? linuxinsight.com/proc_sys_net_ipv4_tcp_syncookies.html