7 Answers
7

You can also pass the headers as a dictionary when creating the Request itself, as the docs note:

headers should be a dictionary, and will be treated as if add_header() was called with each key and value as arguments. This is often used to “spoof” the User-Agent header, which is used by a browser to identify itself – some HTTP servers only allow requests coming from common browsers as opposed to scripts. For example, Mozilla Firefox may identify itself as "Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11", while urllib2‘s default user agent string is "Python-urllib/2.6" (on Python 2.6).

With named parameters you can do this in two lines. Remove the first line and replace the second with this: req = urllib2.Request('www.example.com', headers={'User-Agent': 'Mozilla/5.0'}). I prefer this form for making just a single request.
–
Iain ElderOct 8 '13 at 17:14

All these should work in theory, but (with Python 2.7.2 on Windows at least) any time you send a custom User-agent header, urllib2 doesn't send that header. If you don't try to send a User-agent header, it sends the default Python / urllib2

None of these methods seem to work for adding User-agent but they work for other headers:

opener.addheaders should probably be [('User-agent', 'Custom user agent')]. Otherwise all these methods should work (I've tested on Python 2.7.3 (Linux)). In your case it might break because you use the proxy argument wrong.
–
J.F. SebastianSep 20 '12 at 4:40