http-queue | Queue HTTP Requests to be Made at Set Intervals

A lot of APIs have rate limits, and might block or charge you if you make too many requests in a short period of time.

These are issues I’ve had with a number of projects. One API I was using to scrape global weather data, I accidentally hit their 10,000 requests per month limit in an afternoon.

More recently, I was looking to do a little investigative work using the Reddit API which limits requests to 60 per minute. But, to get the data I needed, and build the picture I wanted, I had to make hundreds of requests to different endpoints in succession.

I didn’t want to get blocked for making too many requests too quickly, and I didn’t want to worry about limiting my requests all over the place, so I cannibalised a little piece of functionality from an older project, edited it a bit, and published my first NPM module.

http-queue

The module queues https requests to be fired in order, at set intervals, and forgotten. So, in my case, I was able to make a request to a specific subreddit for the authors of top posts, and get the most recent 100 posts for each author. That means 101 requests, fired without regard for the limit would see me blocked after the first 60.

Instead, each request is fired at 1000ms intervals, meaning the whole script finishes scraping after 101 seconds and my API privileges are left intact.

And, the best part is, I’ve made it super easy to use.

More info can be found on NPM or Github

https://www.npmjs.com/package/http-queue/

https://github.com/ChrisWoollon/http-queue/

Usage

1

npm install http-queue

You can add requests to the queue using the newRequest method.

The method takes three parameters;

The request URI or object (required)

A response callback, which returns the data as a parameter (optional)

An error callback, which returns the error as a parameter (optional)

Currently, only GET requests over https are supported

1

2

3

4

5

6

7

8

9

10

11

12

let httpQueue=require('http-queue');

constx=1000;// ms between requests

httpQueue=newhttpQueue(x);

// Runs on init

httpQueue.newRequest('https://google.co.uk',function(data){

console.log(data);

});

// Runs 1000ms later

httpQueue.newRequest('https://github.com',function(data){

console.log(data);

});

Alternatively, newRequest takes an object as its first parameter;

1

2

3

4

5

6

varoptions={

url:https://jsonplaceholder.typicode.com/posts/1

}

queue.newRequest(options,function(data){

console.log(data);

});

You can also pass functions as variables for multi-use, and handle errors in their own function