2009/7/26 Julian Fitzell <jfitzell at gmail.com>:
> whew... ok, so the problem was caused by one of the blogs we aggregate
> generating an infinite redirection. Obviously that's a lame reason and
> several pieces of code along the way should have prevented that from
> taking the whole site down but that's what it was. The blog feeds are
> updated periodically and that feed happened to have gone stale and
> every incoming request tried to trigger an update, which never
> returned.
>> Removing that URL essentially solved the problem, though Pier's
> transaction log made it pretty tricky for me to figure out how to even
> get the change to stick (I uploaded a new image with the offending URL
> removed but pier simply reloaded the old value from the transaction
> log). I ended up having to generate a transaction log with only that
> change on my laptop and then concatenate that onto the end of the
> transaction log on the server.
>> The infinite redirection is caused by an unusual (but according to the
> HTTP spec valid) implementation choice in HTTPSocket that seems to
> confuse web servers (one that I've run into before... see
>http://tiny.cc/suckygooglesupport for how Google won't fix their
> servers). HTTPSocket always includes the port in the Host: header when
> making an HTTP request, even when it is the default port. In Google's
> case, the server obviously fails to normalize the URL and realize that
> port 80 is the default port and is therefore the same as the URL
> without the port. In the case of this blog feed, the server also fails
> to normalize. It correctly redirects
>>http://old.feed.address/ to: http://new.feed.address/>> but incorrectly redirects
>>http://old.feed.address:80/ to http://old.feed.address/>> Since HTTPSocket submits the latter address back as
>http://old.feed.address:80/ you can see how we carry on ad infinitum.
>> Diagnosis:
> 1) Obviously HTTPSocket should catch the infinite redirection.
> 2) Pier's blog component also probably shouldn't be allowing a slow
> blog feed update to prevent answering requests for the site.
> 3) Maybe someone should also change HTTPSocket to normalize URLs
> before submitting... it seems to be causing recurring problems
> 4) Maybe we should install the CURL plugin on seaside.st?
AFAIK the aggregation code supports the CURL plugin, because
HTTPSocket is such an abomination.
Cheers
Philippe