This question came from our site for system and network administrators.

Multiple threads servicing the same TCP connection, or multiple threads each with separate TCP connections? Also are you saying the server is multithreaded, or the client is multithreaded, or both?
–
SpiffMar 25 '11 at 5:40

Well, that's probably because you can only transfer so much data over one connection. However in a multi-threaded program you can have two connections receiving data at the same time and doubling the amount of information you can obtain. There are some limitations to this for example the speed of the server you are downloading from... Hats off two whoever wrote the multi-threaded downloader, those are not easy to write.

Why is it so difficult? You just need to allocate an individual section for each thread and let it write in the appropriate section of the result file. The source of axel seems pretty straightforward to me.
–
André ParamésMar 24 '11 at 12:19

TCP works best when you "keep the pipe full" — when the sending app keeps sending buffers quickly enough to keep the sender TCP stack constantly supplied with data so that it can always have data "in flight" on the network, and when the receiver app keeps reading from the receiver TCP stack fast enough that the receiver TCP window never fills up (again, so the sending TCP stack can always keep data "in flight" on the network).

I could imagine a poorly-written single-threaded sender app that passes one buffer to the TCP stack, waits to hear that it's been fully Acked, and then passes another buffer. That means that once the end of the first buffer is "in flight" on the network, the sending TCP stack is starved for data to send, which means the pipe drains and isn't refilled until after the Ack comes back and the sending app passes it a new buffer.

I could also imagine a poorly-written single-threaded receiver app that doesn't read from the receiving TCP stack fast enough and thus lets the TCP stack's buffers fill up, which means the TCP window fills up, which causes the sending TCP stack to stop sending until the window opens up some. Increasing the receiver's TCP window size may help a little, but the real solution on this end is to read the data faster.