Python modify user-agent

Posted by
daniel

May
20

How to generate user-agent header for web requests in Python

In previous post, we saw how to modify user-agent header in wget, curl and httpie programs. In this post, I will show you how to modify user-agent header in Python’s popular requests module. There are several reasons for modify user agent, one of which is to trigger a different response from a website. Many website offer different content based on user-agent header. You can find user-agent header details here.

In Python, one of the most popular libraries to query web servers is the requests module. The requests module allows you to pass header information using the headers option –

1. Simplest use case, without a header

import requests
requests.get('http://linuxfreelancer.com/status')

And this is how it is logged on the web server side, Apache in this case –

This allows us to generate from random to specific valid user-agent header information. We can then pass this randomly generated user-agent text to the requests module header option, and we will view our web server logs to validate –