Notes on programming by Daniel Mendel

Benchmarkers, Beware the Ephemeral Port Limit

Apr 7th, 2013

If you’re benchmarking a web server using tools like ab, weighttp or siege you may encounter an issue that could skew your results – hanging the connection pool. To illustrate this point, let’s look at a couple of benchmarks using siege on OS X:

If the processes are monitored during the second test, both siege and the server process spin up to full capacity for an extended duration as one would expect, but at some point they become totally idle for about 15 seconds before kicking back into action to finish the test.

What gives?

Ephemeral Port Range

To understand what is happening, we have to look at how TCP connections are handled by the operating system. Whenever a connection is made between a client and server, the system binds that connection to an ephemeral port – a set of ports specified at the high end of the valid port range. This is how to reveal what the ephemeral port range is on your system:

The total number of ephemeral ports available on OS X is 16,383 ( on Linux it is usually 28,232 – it is possible to increase the ephemeral port range on OSX ). You might think that this should be more than enough to run our benchmarks since we have only 125 simultanious connections occuring at any given time. However, when one of these ports is closed it does not become immediately available for a new request.

TCP Connection States

During the lifetime of a request, each port goes through a series of states, from SYN_SENT when establishing a connection to ESTABLISHED when communication is actively happening, through a series of closing states eventually culminating in TIME_WAIT after the port has been closed.

During TIME_WAIT the port is held in limbo to ensure any remaining packets are not erroniously provided to a fresh connection. ( Check the current state of ports in use by running netstat -p tcp, get a full overview of the states in the man netstat text )

The duration of the TIME_WAIT state is the Maximum Segment Lifetime and is defined in net.inet.tcp.msl. We can check what it is:

Note that this limitation does not affect real-world requests to a live server because each TCP connection is defined by the tuple of source IP, source port, destination IP and destination port – so the ephemeral port limit only applies to a single client / server pair.