Tackling timeout issues when uploading large files with HttpWebRequest

If you ever had to upload large volumes of data over HTTP, you probably ran into timeout issues. The default Timeout value for HttpWebRequest is 100 seconds, which means that if it takes more than that from the time you send the request headers to the time you receive the response headers, your request will fail. Obviously, if you’re uploading a large file, you need to increase that timeout… but to which value?

If you know the available bandwidth, you could calculate a rough estimate of how long it should take to upload the file, but it’s not very reliable, because if there is some network congestion, it will take longer, and your request will fail even though it could have succeeded given enough time. So, should you set the timeout to a very large value, like several hours, or even Timeout.Infinite? Probably not. The most compelling reason is that even though the transfer itself could take hours, some phases of the exchange shouldn’t take that long. Let’s decompose the phases of an HTTP upload:

Obtaining the request stream or getting the response (orange parts) isn’t supposed to take very long, so obviously we need a rather short timeout there (the default value of 100 seconds seems reasonable). But sending the request body (blue part) could take much longer, and there is no reliable way to decide how long that should be; as long as we keep sending data and the server is receiving it, there is no reason not to continue, even if it’s taking hours. So we actually don’t want a timeout at all there! Unfortunately, the behavior of the Timeout property is to consider everything from the call to GetRequestStream to the return of GetResponse…

In my opinion, it’s a design flaw of the HttpWebRequest class, and one that has bothered me for a very long time. So I eventually came up with a solution. It relies on the fact that the asynchronous versions of GetRequestStream and GetResponse don’t have a timeout mechanism. Here’s what the documentation says:

The Timeout property has no effect on asynchronous requests made with the BeginGetResponse or BeginGetRequestStream method.

In the case of asynchronous requests, the client application implements its own time-out mechanism. Refer to the example in the BeginGetResponse method.

So, a solution could be to to use these methods directly (or the new Task-based versions: GetRequestStreamAsync and GetResponseAsync); but more often than not, you already have an existing code base that uses the synchronous methods, and changing the code to make it fully asynchronous is usually not trivial. So, the easy approach is to create synchronous wrappers around BeginGetRequestStream and BeginGetResponse, with a way to specify a timeout for these operations:

(note that I used the Begin/End methods rather than the Async methods, in order to keep compatibility with older versions of .NET)

These extension methods can be used instead of GetRequestStream and GetResponse; each of them will timeout if they take too long, but once you have the request stream, you can take as long as you want to upload the data. Note that the stream itself has its own read and write timeout (5 minutes by default), so if 5 minutes go by without any data being uploaded, the Write method will cause an exception. Here is the new upload scenario using these methods:

As you can see, the only difference is that the timeout doesn’t apply anymore to the transfer of the request body, but only to obtaining the request stream and getting the response. Here’s a full example that corresponds to the scenario above:

24 thoughts on “Tackling timeout issues when uploading large files with HttpWebRequest”

I want to POST large files to a server using HttpClient. Httpclient has a default timeout of 100 seconds. I do not want to set timeout to infinite. My httpclient should not timeout under normal circumstances and if the httpclient is not able to write data to the request stream it should timeout after say 4 minutes. I want to accomplish exactly the same thing as what you have done with HttpClient. Any ideas on how accomplish this ?
Also any idea on why HttpClient timesout even if I do an asynchronous request like HttpClient.SendAsync() ?

I don’t know how to solve the issue for HttpClient. IIRC, HttpClient.SendAsync uses the BeginGetRequestStream/BeginGetResponse methods internally, so it probably implements its own timeout mechanism, and I don’t think there’s an easy way to disable it…

Hi Thomas,
Thanks for the response. I’m trying to implement a timeout mechanism for my HTTPWebrequest and HttpClient. Towards that I have another question.

In your article you state :
“you can take as long as you want to upload the data. Note that the stream itself has its own read and write timeout (5 minutes by default), so if 5 minutes go by without any data being uploaded, the Write method will cause an exception”

MSDN documentation for read/write timeout states
“Gets or sets a time-out in milliseconds when writing to or reading from a stream. The number of milliseconds before the writing or reading times out.”

If I understand MSDN documentation correctly, your ENTIRE read/write operation must be completed within 5 minutes or else you will time out.
In your article you state “so if 5 minutes go by WITHOUT any data being uploaded” you will time out.
Is it not contradictory or have I misunderstood something ?

Actually the description on MSDN is a little ambiguous… From what I have observed, it works the way I described it. Or perhaps it’s the time before a call to Write times out… (when uploading a large volume of data, you typically do many calls to Write, not just one)

Yeah, Microsofts documentation is misleading I think.
The ReadWriteTimeout actually only times out if there is no data written/read from the stream for that time.
So I think it is also a good Idea to decrease that timeout since 5 minutes seems too large.

Just to say thank you. I’ve been struggling with a GET request provided by a supplier that takes more than 100 seconds to return. I’ve not been able to set the WebRequest timeout to more than 100 seconds, but finally this technique worked. Thank you.

The solution I explain in this post only addresses timeout issues, not other kinds of issues that can occur during a long transfer. In your case, it looks like the server actively closed the connection, so it’s not a timeout.

My previous comment still stands, your error has nothing to do with a timeout (or perhaps the server itself has a timeout for completing the request, but nothing you do from the client side can change that)

First I tried it as described here, without timeout during writing to the stream. That didn’t work, i.e. it timed out.

So I read the MSDN docs and understood it differently as what is described above. Most especially: one cannot write to the stream without a timeout because there is a default of 5 minutes (ReadWriteTimeout) which, contrary to above, is the timeout for actual reading/writing, and not the waiting of the start of transmission.

Given this, I modified my code so that I set ReadWriteTimeout property. That still did not work, because the long ReadWriteTimeout I set for writing to the stream was not honored because GetResponseWithTimeout() timed out first, somehow aborting the writing to the stream.

I didn’t mention the ReadWriteTimeout, because it’s related to a different issue. Basically, if you have ReadWriteTimeout of 5 minutes, it means that it will timeout if you’re unable to read or write anything for 5 minutes. If that happens, it probably means the connection is lost. GetResponseWithTimeout does have a timeout (as the name implies), but at this point ReadWriteTimeout doesn’t come into play. ReadWriteTimeout is used to set the ReadTimeout/WriteTimeout on the request and response streams, so it only makes sense once you have a stream, i.e. when you’re uploading the request body or downloading the response body.

Thanks for the great article. Really helped me understand some timeout issues. However I got one problem with your approach.

In order to get it working, the HttpWebRequest.AllowWriteStreamBuffering = false is mandatory. Otherwise the upload will take place together with the GetResponce-call.
That’s not an issue, but in order to set AllowWriteStreamBuffering = false, I also need to set HttpWebRequest.SendChunked = true. Thats required, otherwise I get an exception.But when I do that, uploads are much slower then before. So far I have no Idea why that is. Any ideas?

What is the exception you’re getting if you don’t send SendChunked to true? I have used AllowWriteStreamBuffering=false without setting SendChunked to true for a long time, and it worked fine. Perhaps it’s a server-side problem?

I’m getting a System.Net.ProtocolViolationException:
When performing a write operation with AllowWriteStreamBuffering set to false, you must either set ContentLength to a non-negative number or set SendChunked to true.

However, I just figured out that the default Buffersize of Streamwriter (which I’m using) is 4kb. That was killing me. Increasing it to 40kb solved the problem 🙂

But just found out another strange thing. When setting keepAlive = false together with AllowWriteStreamBuffering = false I don’t get the system.Net.ProtocolViolationException. In that case I get System.NotSupportedException: The stream does not support concurrent IO read or write operations.

When I just also add SendChunked= true, everything works. I’m really confused now 🙂

I think it is incorrect that the blue part does actually sending and could take very long time, the sending happens when calling GetResponse, from my experiment, GetRequestStream and writing to request stream are very fast, the timeout actually happens in GetResponse().

When AllowWriteStreamBuffering is false, as in my example, calling GetRequestStream immediately sends the headers, and writing on the stream sends directly to the network. When AllowWriteStreamBuffering is true, GetRequestStream returns immediately, and writing to the stream writes to an in-memory buffer, which is sent only when you call GetResponse