Project Description

djangospider is light web crawling framework, it have a few code, but
can do high speed crawling, it support three modes to crawl: multithreading,
tornado IOloop, and twisted rector.you can easily to understand to how to use
async crawler.

Requirement:

Python2.7
Works on Linux

Install:

you can download the zip package in github. then unpack the zip package,
find the path of setup.py, Execute the command:
$sudo python setup.py install

The entry function: Start(start_urls,mode)

start_urls parameter: is a list, and it’s element is tuple:

the first of the tuple is url which you will crawl,
the second of the tuple is the callback for url.