Get started

Install

$ npm install crawler

Basic usage

varCrawler=require("crawler");varc=newCrawler({maxConnections:10,// This will be called for each crawled pagecallback:function(error,res,done){if(error){console.log(error);}else{var$=res.$;// $ is Cheerio by default//a lean implementation of core jQuery designed specifically for the serverconsole.log($("title").text());}done();}});// Queue just one URL, with default callbackc.queue('http://www.amazon.com');// Queue a list of URLsc.queue(['http://www.google.com/','http://www.yahoo.com']);// Queue URLs with custom callbacks & parametersc.queue([{uri:'http://parishackers.org/',jQuery:false,// The global callback won't be calledcallback:function(error,res,done){if(error){console.log(error);}else{console.log('Grabbed',res.body.length,'bytes');}done();}}]);// Queue some HTML code directly without grabbing (mostly for tests)c.queue([{html:'<p>This is a <strong>test</strong></p>'}]);

Slow down

Use rateLimit to slow down when you are visiting web sites.

varcrawler=require("crawler");varc=newCrawler({rateLimit:1000,// `maxConnections` will be forced to 1callback:function(err,res,done){console.log(res.$("title").text());done();}});c.queue(tasks);//between two tasks, minimum time gap is 1000 (ms)

Custom parameters

Sometimes you have to access variables from previous request/response session, what should you do is passing parameters as same as options:

preRequest

If you want to do something either synchronously or asynchronously before each request, you can try the code below. Note that direct requests won’t trigger preRequest.

varc=newCrawler({preRequest:function(options,done){// 'options' here is not the 'options' you pass to 'c.queue', instead, it's the options that is going to be passed to 'request' module console.log(options);// when done is called, the request will startdone();},callback:function(err,res,done){if(err){console.log(err)}else{console.log(res.statusCode)}}});c.queue({uri:'http://www.google.com',// this will override the 'preRequest' defined in crawlerpreRequest:function(options,done){setTimeout(function(){console.log(options);done();},1000)}});

Advanced

Send request directly

In case you want to send a request directly without going through the scheduler in Crawler, try the code below. direct takes the same options as queue, please refer to options for detail. The difference is when calling direct, callback must be defined explicitly, with two arguments error and response, which are the same as that of callback of method queue.

Work with bottleneck

Control rate limit for with limiter. All tasks submit to a limiter will abide the rateLimit and maxConnections restrictions of the limiter. rateLimit is the minimum time gap between two tasks. maxConnections is the maximum number of tasks that can be running at the same time. Limiters are independent of each other. One common use case is setting different limiters for different proxies. One thing is worth noticing, when rateLimit is set to a non-zero value, maxConnections will be forced to 1.

varcrawler=require('crawler');varc=newCrawler({rateLimit:2000,maxConnections:1,callback:function(error,res,done){if(error){console.log(error)}else{var$=res.$;console.log($('title').text())}done();}})// if you want to crawl some website with 2000ms gap between requestsc.queue('http://www.somewebsite.com/page/1')c.queue('http://www.somewebsite.com/page/2')c.queue('http://www.somewebsite.com/page/3')// if you want to crawl some website using proxy with 2000ms gap between requests for each proxyc.queue({uri:'http://www.somewebsite.com/page/1',limiter:'proxy_1',proxy:'proxy_1'})c.queue({uri:'http://www.somewebsite.com/page/2',limiter:'proxy_2',proxy:'proxy_2'})c.queue({uri:'http://www.somewebsite.com/page/3',limiter:'proxy_3',proxy:'proxy_3'})c.queue({uri:'http://www.somewebsite.com/page/4',limiter:'proxy_1',proxy:'proxy_1'})

Normally, all limiter instances in limiter cluster in crawler are instantiated with options specified in crawler constructor. You can change property of any limiter by calling the code below. Currently, we only support changing property ‘rateLimit’ of limiter. Note that the default limiter can be accessed by c.setLimiterProperty('default', 'rateLimit', 3000). We strongly recommend that you leave limiters unchanged after their instantiation unless you know clearly what you are doing.

Server-side DOM options

Object Use cheerio with default configurations to inject document if true or “cheerio”. Or use customized cheerio if an object with Parser options. Disable injecting jQuery selector if false. If you have memory leak issue in your project, use “whacko”, an alternative parser,to avoid that. (Default true)

Charset encoding

options.forceUTF8: Boolean If true crawler will get charset from HTTP headers or meta tag in html and convert it to UTF8 if necessary. Never worry about encoding anymore! (Default true),

options.incomingEncoding: String With forceUTF8: true to set encoding manually (Default null) so that crawler will not have to detect charset by itself. For example, incomingEncoding : 'windows-1255'. See all supported encodings

Cache

options.skipDuplicates: Boolean If true skips URIs that were already crawled, without even calling callback() (Default false). This is not recommended, it’s better to handle outside Crawler use seenreq

Http headers

options.rotateUA: Boolean If true, userAgent should be an array and rotate it (Default false)