Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: ftp
Group: bad behaviour
Status: Open
Resolution: None
Priority: 6
Private: No
Submitted By: Kris (tinker105)
Assigned to: Daniel Stenberg (bagder)
Summary: very long ftp replies confuse curl

Initial Comment:
On an upload using either the curl command line or a client using the "easy" API, if the FTP reply to the STOR command exceeds about 16K (e.g. due to a custom server including debug info), curl will stop reading it and will subsequently falsely report that the connection is dead. The rest of the long reply will be emitted (if verbose set) after subsequent commands and curl will create a new connection (to perform a 2nd transfer).

I tried to fix the problem by doubling the size of UrlState.buffer and the (BUFSIZE) limit checks in ftp_readresp and Curl_GetFTPResponse but encountered a different "reckoned dead" problem, later, after the 2nd STOR, this one emitted from Curl_retry_request.

I've found a bug in the FTP response function and on a system that showed
the bug a quick fix cured the problem. I'm now working on a nice fix for it
and I hope to commit it within a few hours. I'll get back.

I haven't gotten very far trying to figure this bug out. I looked at the
terms in Curl_retry_request that cause curl to reckon the connection as
dead but don't know enough about how the code works to know if the
(differing)values make sense or not (e.g. bytecount == 0, reuse == 1).

I also couldn't figure out how to create a libcurl.so with debug info (on
Solaris w/ gcc). For example, I tried manually adding -g to the link line
but that didn't help. A 'file' says the library isn't stripped but both
dbx and gdb tell me there are no debugging symbols. Any ideas?

Yeah, it seems Curl_GetFTPResponse() sometimes fail to get the end of the
response and then it times out and marks the connection as "dead"...

Can you single-step through it in one of these situations and see if you
get any bright ideas about what's going on? The test cases work for me all
the time and I've tried a couple of things on "live" servers as well
without being able to repeat this problem.

Well, I've got good news and bad news. The good news is that the initial
problem is solved. The bad news is that curl now reckons the connection
dead during the 2nd put. This looks like the same problem I described in
my initial post (after hacking the code). Here's some verbose output:

ok, I now committed my take at unifying the code so that there's now only
one place that do the actual response reading logic so this should
hopefully finally address this problem...

I didn't create a new test case but since they all now use the same
function for reading the responses I think the existing ones should be good
enough. The test suite runs fine for me and I've tested the binary on a few
servers in the wild!

I'll work up a test case and see if I can unify these functions better.
This latter one being a left-over from libcurl's blocking days while the
other is for non-blocking operations but I should be able to do this in a
nicer way...

ok, any chance you can help us by trying to craft up a test case that
repeats your problem? I can't see how your case is different than the ones
we've fixed recently...

See test case 1003, 1005 and 1006 and use as basis for an attempt. They
are all doing different tests with very long ftp server responses.

Your server isn't responding this in a very slow manner, is it?

The "left intact" message means that libcurl aborted the transfer and left
the connection "alive". And yes, leaving such a connection alive assuming
it is okay will mess up the subsequent reconnection quite a bit so that is
a plain wrong action.

Still no joy here (using the 8/24 snapshot). I'm including some snippets
of the verbose output to show what's happening. To elaborate, after about
16K of reply, curl stops reading the reply, emits the "left intact" msg,
there's a short delay, then it emits the "dead" msg, the "QUIT", and (most
of) the remainder of the reply. It then establishes a new connection to
handle the 2nd put in the script.

I've added test case 1005 that tests many short lines from the server, and
it passes, too. However, if you shorten the reply in test case 1005 by 583
characters, then the test hangs. curl hits a boundary condition when it
truncates the long line, and ends up parsing a line with a code of
"250250", which is an unexpected code.