Important

Always check for if you can extract information via an API first if they provide one, RSS / Atom feeds are also a great option if provided.

Prerequisites

We'll be using two external python libraries primarily.

Requests

We'll be using the requests library instead of urllib2. It's better in every way. I could go into details and explain why, but I think the requests page sums it all up in this short paragraph:

Python's standard urllib2 module provides most of the HTTP capabilities you need, but the API is thoroughly broken. It was built for a different time — and a different web. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks.

lxml

lxml is a XML/HTML parser, so we'll be using it to extract the data from the responses we are returned. Some people prefer BeautifulSoup but I'm confident at writing my own xPaths and so would prefer to stick with lxml for raw speed.

Crawling / Spidering

Crawling is exactly the same as a "general" scraper, except that you find the next URLs to scrape on each page that you parse. Crawling is a technique used to create large databases of information.

Examples:

Crawling an eBay category to gather product information. (Don't - they have an API).

Crawling a category in the yellow pages for the contact information of a certain profession.

etc...

Storing seen URLs

You'll want to store the URLs you've already visited, so that you don't scrape them multiple times. For a small scrape(< 50k URLs), I would suggest simply using a set for this. You can then simply check for the presence of a URL in the set before fetching it / adding it to the "To Fetch Queue".

If you're going to crawling a large site, then the above technique of simply using a set won't hold up. The set will grow huge and will start using a ton of memory. The solution to this is a Bloom Filter. I won't go into much detail, but a bloom filter essentially lets us store which urls we have seen, then ask the question "Have I seen this URL before?" andprobably (very likely) returning the correct answer. It can do this usingvery little memory. pybloomfiltermmap is a good Bloom Filter implementation for Python.

Important: Make sure your normalize your URL before inserting it into the set / bloom filter. Depending on the site you may want to remove all query params etc. You don't want to be storing the same URL a ton of times.

CSS Selectors

If you're coming from a Javascript background, you might be more comfortable querying the DOM with CSS selectors rather than xPath. lxml provides a way to do this, it has a css_to_xpath method. Check out this page for a short guide on how to use it:

Concurrency

We all know you want your data now, but hitting the server with 200 concurrent requests is going to make for one angry webmaster. Don't be that guy. Do everybody a favour and stick to a reasonable amount. 5 is usually enough.

To send requests concurrently easily with requests, check out thegrequests library. It allows you to make your code concurrent in pretty much a line or two.

rs =(grequests.get(u)for u in urls) responses = grequests.map(rs)

Robots.txt

Almost every guide out there will tell you to always follow robots.txt rules. I'm not going to do that. The simple fact is they are usually really restrictive and don't even reflect what the webmasters restrictions would be. Most sites just use the default robots.txt for their framework / that their SEO plugin generates. Just make sure you're reasonable.

An exception to this rule is if you are writing a general spider. Something like Google/Bing. In this case, I would follow the rules the disallowed directories and crawl delay. If they don't want something indexing, then don't index it in your search engine or whatever.

Avoiding Detection

Sometimes you don't want to get caught. I get that. It isn't actually that difficult to achieve. The first thing to do it randomize your user agent. Your IP address will still be the same for all the requests, but that isn't unusual. Places like universities / businesses often have a ton of computers that all route traffic via 1 IP.

The next step is to use proxies. You can pick up shared proxies at a very small monthly cost, and when you're using randomized user agents and sending requests from 100 different IP's spread across different IP ranges and even different datacenters, they're gonna have a hard time detecting you.

Common Problems

Installing lxml

You may run into some trouble when trying to install lxml, especially on a linux distro. lxml requires the following packages to be installed before:

apt-get install libxml2-dev libxslt-dev python-dev lib32z1-dev

You can then run the pip install lxml command as per usual.

The site uses AJAX, I can't scrape it!

This is simply incorrect. Often if a site uses AJAX, it can make your job even easier. Use the Network tab in Chrome Developer Tools to find the AJAX request, you'll usually be greeted by a response in json. This saves you the effort of having to parse the page, you can just hit the AJAX endpoint directly.

The site is showing different content to my scraper

Check the page source in your browser and make sure the information you want to scrape is actually there. It could be that the content was injected into the page via javascript and our scraper doesn't run javascript. If this is the case, your only real option is to run something like PhantomJS. You can control Phantom from Python though, just use something like selenium or splinter.

Sites often serve different content to different user agents. You should check the user agent that you are sending in your browser and set your scraper to have the same.

It's possible that the site is serving different content to different locations using geotargeting. If you're running your script from a server, then this could be affecting you.

I'm getting 403 errors / rate limited

The simple solution to this is don't hit the site so damn hard. Be conservative with the amount of concurrent requests you use. Do you really need the data within the hour, can you really not wait 24 hours for it?

If you really need to keep sending requests to the site, then you're going to have to invest in some proxies. You can then simply create a simple proxy manager class that makes sure you only use the same proxy once every X interval so that it doesn't get blocked.

Check out the Concurrency and Avoiding Detection sections above.

Code Examples

The following examples aren't really of any use on their own. They're simply here as a reference for you.

Single Threaded Crawler (basic)

This crawler just the basic principle behind crawling. It uses a dequeue so you can pop from the left of the queue and therefore fetch URLs in the order you find them. It doesn't do anything useful with the pages, just printing out the page title.