How to route a Celery task to a dedicated queue

By default, Celery routes all tasks to a single queue and all workers consume this default queue.
With Celery queues, you can control which Celery workers process which tasks. This can be useful if you
have a slow and a fast task and you want the slow tasks not to interfere with the fast tasks. Or if you
need to send tasks from one microservice to another microservice.

Step 1: Configure Celery via task_routes

Celery can be configured on a per-task basis which queue a task gets sent to:

Note the --queues command line arguments. They your workers subscribe to particular queues.
For subscribing to more than one queue, use a comma-separated list, like so --queues=feeds,filters.
For further information, have a look at the Celery docs.

Check the docker-compose logs to follow the task flow through the two workers:

docker-compose logs -f

The docker-compose.yml stack also comes with a flower instance.
Flower is a tool for monitoring Celery workers and tasks. Check out your browser on http://localhost:5555.

Summary

In this blog post you learned how to configure Celery to route tasks to dedicated queues and how to make Celery
workers subscribe to certain queues.

In order to achieve that, you need to define routes per task. This approach works well for a simple setup.
However it does not scale very well for an application with many Celery tasks or for a micro-services/Docker environment
where multiple services communicate via the same message broker.

In the next blog post, you will learn all about dynamic task routing which is a programmatic and scalable solution to
overcome the limitations described above.