1

I'm getting the error relating to port exhaustion on my server ("Only one usage of each socket address (protocol/network address/port) is normally permitted").

So, I checked with netstat -n, and it lists about 84000 connections.

Based on the local port, and the fact that the foreign IP address is myself, I can see that for example 3654 of them relate to a program running on the same server that I killed over 24 hours ago (yesterday morning, in fact). They're all in the TIME_WAIT state, on foreign ports between 49165 and 65533. At least some of the same foreign port numbers show up as local ports too, for example:

TCP    <my_ip>:14046  <my_ip>:49165  TIME_WAIT
...
TCP    <my_ip>:49165  <other_ip>:80  TIME_WAIT

14046 is the port that a program on my server listens to, and that the program I killed yesterday used to repeatedly connect to. host <other_ip> gives me something at compute-1.amazonaws.com, I don't recognise the IP. I can't get anything out of it from my browser. So I can't quite figure out what on that machine would have connected to port 80 on , but there are various things that have run on the same machine recently that consume web services, so I guess it could be a redirect or similar from one of those.

Separate runs of netstat more than 4 minutes apart show that both connections are sticking around in the list.

Is this expected? I thought sockets were supposed to finally close after 4 minutes or so, but does netstat normally list them for much longer? Are these ports really occupied, and if so is there anything I can do to release them? Preferably without rebooting, but if I have to then I have to.

OS is Windows Server Enterprise, running on a Rackspace VM.

[Edit: hmm, should I have asked this on serverfault?]

[Another edit: a reboot seemed to fix it. All the old connections gone, and new connections of the same type disappearing in a few minutes as expected. I would guess therefore that the network stack somehow got into a malfunctioning state.]

Improve this question
2
  • FWIW, the issue outlined here doesn't seem to be something specific to an enterprise server environment (could happen on any system), so I'd keep it here.
    – Darth Android
    Commented Aug 4, 2011 at 19:31
  • @Darth: that's what I figured - the port exhaustion is because I'm running server-type things, but the fact that the ports stick around may well not be. I just would never have noticed if it had been a few hundred rather than a few tens of thousands.
    – Steve Jessop
    Commented Aug 5, 2011 at 8:40

1 Answer 1

Reset to default
0

Take a look at the following article from Microsoft: Avoiding TCP/IP Port Exhaustion.

One of the suggestions it has is to check your registry setting for the TcpTimedWaitDelay key under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters. That entry controls "the length of time that a connection stays in the TIME_WAIT state when the connection is being closed". Perhaps you've got a very long value somehow .. the default is 240 seconds.

Also, you may want to check and see if you can trace and see which processes are causing the TIME_WAIT connections and see if there is a configuration setting in it to close ports quicker.

Improve this answer
1
  • Thanks, there's no TcpTimedWaitDelay value set. I know exactly what processes caused the connections on port 14046. The port 80 could be a few different things, but they all use Python's urllib or httplib to consume web services. A bug in my code or in Python, not closing the sockets cleanly, I'd sort of understand. I don't know how I'd even go about writing a bug that sets the TIME_WAIT time to (at least) several days, though...
    – Steve Jessop
    Commented Aug 5, 2011 at 8:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .