Executing HTTP requests in parallel with asyncio

Neel Somani - June 9, 2018

If you’re making multiple HTTP requests in Python, it’s clear that you’d want to perform them asynchronously. The alternative would be to wait for each blocking request to complete before starting the next one, a waste of resources.

Making the requests in parallel

As an example, let’s see how long performing 5 requests synchronously would take:

On my computer, this gives about 1.86 seconds. To make things asynchronous, we only need to do a few things. We need to create a Future for each HTTP request, which will run on our own ThreadPoolExecutor, so we can specify the number of threads. To wait on these Futures, this will have to be done within an asynchronous function, which we’ll have the event loop run until completion.

Benchmarking

The single request takes about .375 seconds. As we can see, running five requests asynchronously is marginally slower (maybe due to minor overhead in scheduling a task with the ThreadPoolExecutor, or a small delay by asyncio.wait), but it’s substantially better than performing the 5 requests synchronously.

About the Author

Neel Somani, a student at the University of California, Berkeley, is the founder of Apptic LLC. In addition to computer science, he's interested in philosophy and entrepreneurship. You can follow him on Google+, LinkedIn, and Twitter.