Menu

Distributed Scraping With Multiple Tor Circuits

Multiple Circuit Tor Solution

When you rapidly fetch different web pages from a single IP address you risk getting stuck in the middle of the scraping. Some sites completely ban scrapers, while others follow a rate limit policy. For example, If you automate Google searches, Google will require you to solve captchas. Google is confused by many people using the same IP, and by search junkies. It used to be costly to get enough IPs to build a good scraping infrastructure. Now there are alternatives: cheap rotating proxies and Tor. Other options include specialized crawling and scraping services like 80legs, or even running Tor on AWS EC2 instances. The advantage of running Tor is its widespread network coverage. Tor is also free of charge. Unfortunately Tor does not allow you to control the bandwidth and latency.

All navigation performed when you start a session on Tor will be associated with the same exit point and its IP addresses. To renew these IP addresses you must restart Tor, or send a newnym signal, or as in our case study you can run multiples Tor instances at the same time If you assign different ports for each one. Many SOCKS proxies will then be ready for use. It is possible for more than one instance to share the same circuit, but that is beyond the scope of this article.

IMDB: A Case Study

If you like movies, Internet Movie Database is omnipresent in your daily life. IMDB users have always been able to share their movies and lists. Recently, however, the site turned previously shared public movie ratings private by default. Useful movie ratings disappeared from Internet with this change, and most of those that were manually set back to public are not indexed by search engines. All links that previously pointed to user ratings are now broken since the URLs have changed. How can you find all the public ratings available on IMDB?
If you use IMDB’s scraping policy it will take years, since the site contains tens of million of user pages. Distributed scraping is the best way to solve this issue and quickly discover which users are sharing their ratings. Our method just retrieves the HTTP response code to find out whether the user is sharing his rating.

Our code sample has three elements:

Multiple Tor instances listening to different ports. The result is many SOCKS proxies available for use with different Tor circuits.

A Python script that launches multiple workers in different threads. Each worker uses a different SOCK port.

MongoDB to persist the state of the scraping if the process fails or if you want to stop the process and continue later.

Python Script

The script below stores its results on MongoDB on the “imdb” db under the “imdb.ratings” collection. To handle the number of simultaneous workers you can change the “Discovery.NWorkers” variable. Note that the the number of workers must be equal to or less than the number of Tor instances.

This script will gather users with public ratings using the following MongoDB query: db.imdb.ratings.find({‘last_response’: 200})
Try exporting the movies ratings. This the easiest part because it is now a comma separated value file and you don’t need an XPath query.

Additional observations

We are not just using MongoDB because it is fancy, but also because it is very practical for quickly prototyping and persisting data along the way. The well-known “global lock” limitation on MongoDB (and many other databases) does not significantly affect its ability to efficiently store data.

We use SocksiPy to allow us to use different proxies at the same time.

If you are serious about using Tor to build a distributed infrastructure you might consider running Tor proxies on AWS EC2 instances as needed.

Do not forget to run Tor instances in a secure environment since the control port is open to everyone without authentication.

Our solution is easily scalable.

If you get many 503 return codes, try balancing the quantity of proxies and delaying each worker’s activity.

Post navigation

thanks for this, it helped me get python connecting with TOR which I was finding difficult :)

Bin Wang

Thanks a lot for your explanation of building multiple TOR by either multiple threads or the round robin method. However, i have a hard time forking your code since I am using mysql instead of mongodb, I am wondering could you please maybe talk about how to use multiple tor and connect with mysql, thanks !

http://blog.databigbang.com Sebastian Wain

Hi Bin, what are the difficult issues that you find converting from mongodb to mysql?

bin

Hi Sebastian, I have already figured that out, and the reason is that you have to rewrite every command that updates the database, from mongodb syntax to mysql… btw, your code sometimes cannot finish after I changed the croupier to run only one time. I am pretty sure there is only one element in the queue, which is supposed to be none. However, it keeps printing Elements in queue 0: Active threads:xx Exception Counter :1. … Seems like the monitor cannot stop…. have that happened to you.

Sid

Do we really need to have the control port passwd be blank. Dont see it being used in the python script anywhere.

Irenicus

i am tring to connect .onion throught python, i can connect to tor successfully (got into check.torproject.org and saw Congratulations text) but when i try .onion sites my program says cant connect and says “name or service is not known”, do you have any idea ? where do i do wrong ?

http://bounteo.us/ Rob

That sounds like python is doing DNS resolution directly instead of through Tor. One of the main differences between SOCKS4 and SOCKS5 is that the latter added support for UDP in addition to TCP. Python is probably using UDP for DNS (most software does) so it’s not getting sent to the Tor router. Change socks.PROXY_TYPE_SOCKS4 to socks.PROXY_TYPE_SOCKS5 and that should fix the problem.

Robert

As you mentioned, I’ve been trying to automate Google searches, but when I try to connect to google using tor their server detects that I’m using tor and they don’t accept the request. Do you know any way of bypass such scheme?

http://bounteo.us/ Rob

Google is extremely good at preventing scraping. You’re getting caught by Google’s rate limiters – remember there are lots of other people using these same exit nodes. As far as I know there’s no way around this – you’ll likely need to find a different proxy service entirely.