Requesting http://www.netflix.com would result in being redirected to http://www.netflix.com/Default?tcw=1&cqs=, which would result in being redirected back to netflix.com and so on forever. Drakma actually handled the situation very gracefully. After 6 redirect attempts, it threw an exception saying it had exceeded its redirection limit. I just couldn’t figure out why.

I tried messing with my user agent. Writing scrapers has always made me a little paranoid that the developers on the other end know what I’m up to, are locked in a struggle-to-the-death to prevent my unauthorized uses and of course use the most sophisticated and obvious option available to them: my User-Agent header. Drakma has a remarkably convenient built-in facility for spoofing major browser versions. It’s literally a 20 character change to the function call. I tried that. No dice. Apparently the Netflix developers are as overworked as everyone else and bots just aren’t a big enough problem to justify snuffing them out by filtering User-Agent headers.

The redirection eventually terminated in Firefox. Otherwise I and millions of other users wouldn’t be able to see the page at all. Maybe Firefox has a built in limit to redirection, at which point it just says “fuck it, I’m just gonna go with this page”. But that doesn’t make any sense either, becuase 302’s don’t come with any HTML to display just in case the browser decides it’s tired of bouncing from page to page.

Firebug’s net panel showed that with cookies on, I didn’t redirect at all, netflix.com returned a 200 OK. With cookies off, I redirected once to /Default, then back to the homepage and that time, a 200 came back with gloriously renderable HTML.

I sent Drakma’s HTTP headers to *standard-output* with a handy little one-liner:

Huh, the issue was in Drakma. I would request their homepage with no cookies, and Netflix would respond with a redirect to a different URL and a bunch of Set-Cookie headers. Drakma would follow, dutifully passing a Cookie header, and be redirected back to the original URL, also with Set-Cookie headers. Drakma would again follow, but this time not include cookies in the request.

Netflix is doing the redirects — I think — to ‘prime the pump’. If you don’t send cookies to the second URL, you get redirected to a you need to turn cookies on page. If you do, Netflix figures you’ll end up back at their homepage with a bunch of initialized cookies. It’s weird behavior on the part of Drakma that’s breaking the system.