Example code can be found on GitHub. All code on this post is licensed under MIT.

Mayhem Mandrill Recap

The goal for this 5-part series is to build a mock chaos monkey-like service called “Mayhem Mandrill”. This is an event-driven service that consumes from a pub/sub, and initiates a mock restart of a host. We could get thousands of messages in seconds, so as we get a message, we shouldn’t block the handling of the next message we receive.

Making synchronous code asyncio-friendly

I’m sure that as folks have started to use asyncio, they’ve realized that async/await starts permeating everything around the code-base; everything needs to be async. This isn’t necessarily a bad thing; it just forces a shift in perspective.

Although sometimes, you’ll have a need to call synchronous code in your beautiful, asynchronous monster. To make it non-blocking, it may be as easy as using a threadpool executor:

So for our code to work with this, we need to rework our asynchronous consumer and our __main__ scope:

asyncdefconsume(executor,queue):loop=asyncio.get_running_loop()whileTrue:msg=awaitloop.run_in_executor(executor,consume_sync,queue)ifnotmsg:# could be Nonecontinueasyncio.create_task(handle_message(msg))# <-- snip -->if__name__=='__main__':loop=asyncio.get_event_loop()# May want to catch other signals toosignals=(signal.SIGHUP,signal.SIGTERM,signal.SIGINT)forsinsignals:loop.add_signal_handler(s,lambdas=s:asyncio.create_task(shutdown(s,loop)))queue_sync=queue.Queue()executor=concurrent.futures.ThreadPoolExecutor(max_workers=5)publisher_coro=handle_exception(publish(executor,queue_sync),loop)consumer_coro=handle_exception(consume(executor,queue_sync),loop)try:loop.create_task(publisher_coro)loop.create_task(consumer_coro)loop.run_forever()finally:logging.info('Cleaning up')loop.stop()

Pretty easy actually; very similar to before with the save_sync example.

Aside: There’s a handy little package called asyncio-extras which provides a decorator for synchronous functions/methods. You can avoid the boilerplate of setting up an executor and just await the decorated function.

But sometimes, third-party code throws a wrench at you…

Making threaded code asyncio-friendly

If you’re lucky, you’ll be faced with a third-party library that is multi-threaded and blocking. For example, Google Python library for its Pub/Sub makes use of gRPC under the hood which is implemented with threading, but is also blocks when we’re consuming from a publisher. The library also requires a non-asynchronous callback for when a message is received. To visualize, here’s a simple script which uses this library (if you’re running this yourself, be sure to use their local emulator):

#!/usr/bin/env python3"""Notice! This requires: google-cloud-pubsub==0.35.4 (latest at the time of writing)"""importjsonimportloggingimportosimportrandomimportstringfromgoogle.cloudimportpubsublogging.basicConfig(level=logging.INFO,format='%(asctime)s,%(msecs)d%(levelname)s: %(message)s',datefmt='%H:%M:%S',)TOPIC='projects/europython18/topics/ep18-topic'SUBSCRIPTION='projects/europython18/subscriptions/ep18-sub'PROJECT='europython18'CHOICES=string.ascii_lowercase+string.digitsdefget_publisher():client=pubsub.PublisherClient()try:client.create_topic(TOPIC)exceptExceptionase:pass# already createdreturnclientdefget_subscriber():client=pubsub.SubscriberClient()try:client.create_subscription(SUBSCRIPTION,TOPIC)exceptException:pass# already createdreturnclientdefpublish_sync():publisher=get_publisher_client()formsginrange(1,6):msg_data={'msg_id':''.join(random.choices(CHOICES,k=4))}bytes_message=bytes(json.dumps(msg_data),encoding='utf-8')publisher.publish(TOPIC,bytes_message)logging.debug(f'Published {msg_data["msg_id"]}')defconsume_sync():client=get_subscriber_client()defcallback(msg):msg.ack()data=json.loads(msg.data.decode('utf-8'))logging.info(f'Consumed {data["msg_id"]}')future=client.subscribe(SUBSCRIPTION,callback)try:future.result()# blockingexceptExceptionase:logging.error(f'Caught exception: {e}')if__name__=='__main__':# safety net, wouldn't want to do anything in prodassertos.environ.get('PUBSUB_EMULATOR_HOST'),'You should be running the emulator'publish_sync()consume_sync()

In particular, looking at consume_sync, the returned future is a StreamingPullFuture. It’s pretty handy: it will asynchronously pull for messages from the publisher, allowing us to forgo the while True loop to periodically pull ourselves. The StreamingPullFuture also makes use of some convenient features, including managing the message deadlines.

To illustrate, here’s how we can use loop.run_in_executor for this blocking code. I’ve made a helper coroutine function (run_pubsub) to setup an executor, use it to kick off the synchronous consumer, and pass it off to my async publisher to use for its non-async work:

# <-- snip -->importasyncioimportconcurrent.futuresimportsignal# <-- snip --># updated func to take in the loop as an argumentasyncdefpublish(executor,loop):publisher=get_publisher()whileTrue:awaitloop.run_in_executor(executor,publish_sync,publisher)awaitasyncio.sleep(.1)defcallback(msg):msg.ack()data=json.loads(msg.data.decode('utf-8'))logging.info(f'Consumed {data["msg_id"]}')defconsume_sync():client=get_subscriber()# remove the try/except around the returned future for nowclient.subscribe(SUBSCRIPTION,callback)asyncdefrun_pubsub():loop=asyncio.get_running_loop()executor=concurrent.futures.ThreadPoolExecutor(max_workers=5)consume_coro=loop.run_in_executor(executor,consume_sync)asyncio.ensure_future(consume_coro)loop.create_task(publish(executor,loop))asyncdefshutdown(signal,loop):logging.info(f'Received exit signal {signal.name}...')loop.stop()logging.info('Shutdown complete.')if__name__=='__main__':# safety net, wouldn't want to do anything in prodassertos.environ.get('PUBSUB_EMULATOR_HOST'),'You should be running the emulator'loop=asyncio.get_event_loop()# one signal for simplicityloop.add_signal_handler(signal.SIGINT,lambda:asyncio.create_task(shutdown(signal.SIGINT,loop)))try:loop.create_task(run_pubsub())loop.run_forever()finally:logging.info('Cleaning up')loop.stop()

I’d like to also prove that this is now non-blocking, so let’s add a dummy coroutine function, run_something_else, to be ran alongside run_pubsub. We’ll add two coroutine functions to a general run, helper, and update the __main__ section:

# snipasyncdefrun_something_else():whileTrue:logging.info('Running something else')awaitasyncio.sleep(random.random())asyncdefrun():coros=[run_pubsub(),run_something_else()]awaitasyncio.gather(*coros)if__name__=='__main__':assertos.environ.get('PUBSUB_EMULATOR_HOST'),'You should be running the emulator'loop=asyncio.get_event_loop()# for simplicityloop.add_signal_handler(signal.SIGINT,lambda:asyncio.create_task(shutdown(signal.SIGINT,loop)))try:loop.create_task(run())loop.run_forever()finally:logging.info('Cleaning up')loop.stop()

We see we have the MainThread which is the asyncio event loop. There’s also five Mandrill_-prefixed threads that were created by our threadpool executor. There’s five because we limited the number of workers when creating the executor. It looks as if the subscription client has its own threadpool executor named ThreadPoolExecutor-ThreadScheduler; Thread-MonitorBatchPublisher is from the publisher; and some gRPC/bidirectional streaming going on for consuming pub/sub with the rest of the threads (heart beater, lease maintainer, etc).

All in all, though, the approach to threaded code isn’t any different than the non-async code.

Until you release you need to call asynchronous code from a non-async function that’s within another thread.

Making threaded code asyncio-friendly tolerable

Obviously we can’t just ack a message once we receive it. We need to restart the required host and save the message in our database.

This is deceptive. We’re lucky it works. Once we share some data between the threaded code in the callback and the asynchronous code when handling the message, we’ll see this only works because of happenstance.

To illustrate what I mean, let’s share a simple intermediary queue between the threaded code and the event loop, and try to cancel the task that loop.create_task returns.

GLOBAL_QUEUE=asyncio.Queue()asyncdefget_from_queue():whileTrue:pubsub_msg=awaitGLOBAL_QUEUE.get()logging.info(f'Got {pubsub_msg.message_id} from queue')asyncio.create_task(handle_message(pubsub_msg))asyncdefadd_to_queue(msg):logging.info(f'Adding {msg.message_id} to queue')awaitGLOBAL_QUEUE.put(msg)defconsume_sync(loop):client=get_subscriber()defcallback(pubsub_msg):logging.info(f'Consumed {pubsub_msg.message_id}')task=loop.create_task(add_to_queue(pubsub_msg))task.cancel()# attempt to cancel the task given to another threadclient.subscribe(SUBSCRIPTION,callback)

It may look like things are serially processed, but it’s just the streaming pull future that the Google Pub/Sub library returns (just take a look at the milliseconds!).

Recap

It’s pretty simple to get around synchronous code using a ThreadPoolExecutor and loop.run_in_executor. However, one can easily get tripped up when needing to use threads with asyncio. With that, there are a few _threadsafe APIs within the asyncio library that it’s good to get familiar with.