The draft draft-ietf-http-v10-spec-00.txt contains the following text:
7.2.2 Length
When an Entity-Body is included with a message, the length of that
body may be determined in one of several ways. If a Content-Length
header field is present, its value in bytes (number of octets)
represents the length of the Entity-Body. Otherwise, the body
length is determined by the Content-Type (for types with an
explicit end-of-body delimiter), the Content-Transfer-Encoding (for
packetized encodings), or the closing of the connection by the
server. Note that the latter cannot be used to indicate the end of
a request body, since it leaves no possibility for the server to
send back a response.
Note: Some older servers supply an invalid Content-Length
when sending a document that contains server-side includes
dynamically inserted into the data stream. It must be
emphasized that this will not be tolerated by future
versions of HTTP. Unless the client knows that it is
receiving a response from a compliant server, it should not
depend on the Content-Length value being correct.
Packetized C-T-Es are not specified yet.
Most body types (all but the multiparts) do not have an explicit
end-of-body delimiter.
This means that the closing of the connection is THE method left when
you don't depend on content-length.
I've seen this blow up on me: When a particular server had very heavy
load, the forked server crashed, closing the TCP connection, after
sending the headers, but before sending the full body.
Result: The client thinks that it has a complete object, and in fact
CACHES this invalid object - and there is no algorithmic way to tell
it that it is in the wrong.
This worries me.
Suggested fix: Add language that says that clients SHOULD warn the user
when content-length indicates MORE data than actually received. I've
seen lots of cases where the browser says "got 10245 of 9433 bytes",
which probably means that content-length was LESS, but I don't know if
the reverse ever happens.
Harald T. Alvestrand