Last viewed:

Python httplib2 - Handling Cookies in HTTP Form Posts

Submitted by Anonymous (not verified) on Tue, 11/09/2007 - 17:33

I often need to automate tasks in web based applications. I like to do this
at the protocol level by simulating a real user's interactions via HTTP. Python
comes with two built-in modules for this: urllib (higher
level Web interface) and httplib (lower
level HTTP interface).

However, I usually don't use either of these. I prefer to use Joe
Gregario's excellent httplib2 module (btw,
I really wish this could make its way into Python's Standard Library).
It is a much richer library and has a lot of nice features for dealing with HTTP.

When automating something, you often need to "login" to maintain some sort of session/state
with the server. This is usually achieved with form-based authentication. You
post a form to the server, and it responds with a cookie in the incoming HTTP header.
You need to pass this cookie back to the server in subsequent requests to maintain
state or to keep a session alive.

Here is an example of how to deal with cookies when doing your HTTP Post.

First, lets import the modules we will use:

import urllib import httplib2

Now, lets define the data we will need: In this case, we are doing a form post with
2 fields representing a username and a password.

At this point, our "response" variable contains a dictionary of HTTP header fields
that were returned by the server. If a cookie was returned, you would see a "set-cookie"
field containing the cookie value. We want to take this value and put it into the
outgoing HTTP header for our subsequent requests:

headers['Cookie'] = response['set-cookie']

Now we can send a request using this header and it will contain the cookie, so the
server can recognize us.

So... here is the whole thing in a script. We login to a site and then make another
request using the cookie we received: