Project description

If you know Python, you know
`Requests <http://docs.python-requests.org/>`__. Requests is love.
Requests is life. Depending on your use cases, you may come across
scenarios where you need to use Requests a lot. Services you consume
may have rate-limiting policies in place or you may just happen to be in
a good mood and feel like being a good Netizen. This is where
requests-respectful can come in handy.

*requests-respectful*:

Is a minimalist wrapper on top of Requests to work within rate
limits of any amount of services simultaneously

Can scale out of a single thread, single process or even a single
machine

Enables maximizing your allowed requests without ever going over set
limits and having to handle the fallout

fromrequests_respectfulimportRespectfulRequesterrr=RespectfulRequester()# This can be done elsewhere but the realm needs to be registered!rr.register_realm("Github",max_requests=100,timespan=60)response=rr.get("http://github.com",params={"foo":"bar"},realms=["Github"],wait=True)

importrequestsfromrequests_respectfulimportRespectfulRequesterrr=RespectfulRequester()# This can be done elsewhere but the realm needs to be registered!rr.register_realm("Github",max_requests=100,timespan=60)request_func=lambda:requests.get("http://github.com",params={"foo":"bar"})response=rr.request(request_func,realms=["Github"],wait=True)

Installation

Configuration

Default Configuration Values

Configuration Keys

redis: Provides the host, portand database of the
Redis instance

safety_threshold: A rate-limited exception will be raised at
(realm_max_requests - safety_threshold). Prevents going over the
limit of services in scenarios where a large amount of requests are
issued in parallel

requests_module_name: Provides the name of the Requests
module used in the request lambdas. Should not need to be changed
unless you import Requests as another name.

Overriding Configuration Values

With requests-respectful.config.yml

The library auto-detects the presence of a YAML file named
requests-respectful.config.yml at the root of your project and will
attempt to load configuration values from it.

Example:

requests-respectful.config.yml

redis:host:0.0.0.0port:6379database:5safety_threshold:25

With the configure() class method

If you don’t like having an extra file lying around, the library can
also be configured at runtime using the configure() class method.

Either of these registers 3 realms: * Google at a maximum requesting
rate of 10 requests per second * Github at a maximum requesting rate
of 100 requests per minute * Twitter at a maximum requesting rate of
150 requests per 5 minutes

Updating a Realm

rr.update_realm("Google",max_requests=25,timespan=5)

This updates the maximum requesting rate of Google to 25 requests per
5 seconds.

Getting the maximum requests value of a Realm

rr.realm_max_requests("Google")

This would return 25.

Getting the timespan value of a Realm

rr.realm_timespan("Google")

This would return 5.

Unregistering a Realm

rr.unregister_realm("Google")

This would unregister the Google realm, preventing further queries
from executing on it.

Unregistering multiple Realms

rr.unregister_realms(["Google","Github","Twitter"])

This would unregister all 3 realms in one operation, preventing further
queries from executing on them.

Requesting

Using Requests HTTP verb methods

The library supports proxying calls to the 7 Requests HTTP verb
methods (DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT). This is
literally a Requests method so go crazy with your params, body,
headers, auth etc. kwargs. The only major difference is that a
realm kwarg is expected. A wait boolean kwargs can also be provided
(the behavior is explained later).

The kwarg realm has been deprecated on requesting instance methods.
It will still work with a warning until 0.3.0

Handling exceptions

Executing these calls will either return a requests.Response object
with the results of the HTTP call or raise a
RequestsRespectfulRateLimitedError exception. This means that you’ll
likely want to catch and handle that exception.

The wait kwarg

Both ways of requesting accept a wait kwarg that defaults to False. If
switched on and the realm is currently rate-limited, the process will
block, wait until it is safe to send requests again and perform the
requests then. Waiting is perfectly fine for scripts or smaller
operations but is discouraged for large, multi-realm, parallel tasks
(i.e. Background Tasks like Celery workers).

Tests

FAQ

Whoa, whoa, whoa! Redis?!

Yes. The use of Redis allows for requests-respectful to go
multi-thread, multi-process and even multi-machine while still
respecting the maximum requesting rates of registered realms. Operations
like Redis’ SETEX are key in designing and working with rate-limiting
systems. If you are doing Python development, there is a decent chance
you already work with Redis as it is one of the two options to use as
Celery’s backend and one of the 2 major caching options in Web
development. If not, you can always keep things clean and use a Docker
Container or even build it from
source. Redis has kept a
consistent record over the years of being lightweight, solid software.

How is this different than other throttling libraries?

Most other libraries will ask you to specify an interval at which to
send requests and will literally loop over
request()...time.sleep(interval). This one will allow to send as
many as you want, as fast as you want, as long as you are under the
maximum requesting rate of your realm.

Other libraries don’t have the concept of realms and separate
requesting rate rules.