Search

Meta

This tutorial shows how to distribute messages to multiple servers for processing. We will be using RabbitMQ which is based on AMQP.

The way messaging works is that you have producers producing messages and consumers consuming them. We are going to have 1 producer generating random integers and 2 consumers: 1 consuming the odd integers and 1 consuming the even integers.

The producer sends messages to the exchange and the consumers create and bind queues to the exchange to receive the messages they are interested in. The way messages are routed is using routing keys for the messages and binding keys for the queues.

The producer will be running on the server “producer.messaging.com”, 1 consumer will be running on “consumer1.messaging.com” and the other one on “consumer2.messaging.com”.

Requirements

We need to install RabbitMQ server on producer.messaging.com and py-amqplib on the 3 servers.

After installing rabbitmq-server, the server should be up and running.

Producer

First thing to do is to create a Python class for the producer. The constructor will initiate a connection to the RabbitMQ server. The constructor also takes as argument the name of the exchange which will receive all the messages from the producer.

We also need a method to publish a message to the exchange. This method takes the message and its routing key as arguments. The content type of the message here is text but you can change it. Also, we use persistence mode for the message (delivery_mode = 2) so best-effort is used for delivery.

Next is a method to create an exchange. We use the exchange type “direct” which means the broker routes the message using a straight match on the routing key. If a we bind a queue using the key ‘test’, only the messages with the routing key ‘test’ will be sent to it.

def declare_exchange(self, exchange_name, durable=True, auto_delete=False):
"""
Create exchange.
@param exchange_name name of the exchange
@param durable will the server survive a server restart
@param auto_delete should the server delete the exchange when it is
no longer in use
"""
self.exchange_name = exchange_name
self.channel.exchange_declare(exchange=self.exchange_name,
type='direct', durable=durable, auto_delete=auto_delete)

Followed by a method to create a queue and bind it to the exchange. We need to indicate the binding key so the exchange knows which messages to send to the queue based on their routing key.

def declare_queue(self, queue_name, routing_key, durable=True, exclusive=False, auto_delete=False):
"""
Create a queue and bind it to the exchange.
@param queue_name Name of the queue to create
@param routing_key binding key
@param durable will the queue service a server restart
@param exclusive only 1 client can work with it
@param auto_delete should the server delete the exchange when it is
no longer in use
"""
self.queue_name = queue_name
self.routing_key = routing_key
self.channel.queue_declare(queue=self.queue_name, durable=durable,
exclusive=exclusive, auto_delete=auto_delete)
self.channel.queue_bind(queue=self.queue_name,
exchange=self.exchange_name, routing_key=self.routing_key)

At this point, we can get define a method to start a consumer and use a callback for each message delivered. The next method allows us to do that.

def start_consuming(self, callback, queue_name=None, consumer_tag='consumer'):
"""
Start a consumer and register a function to be called when a message is consumed
@param callback function to call
@param queue_name name of the queue
@param consumer_tag a client-generated consumer tag to establish context
"""
if hasattr(self, 'queue_name') or queue_name:
self.channel.basic_consume(queue=getattr(self, 'queue_name',
queue_name),
callback=callback, consumer_tag=consumer_tag)

Last, we need a wait method to ask the consumer to be patient and wait for activity on the channel: ie: messages being delivered.

def wait(self):
"""
Wait for activity on the channel.
"""
while True:
self.channel.wait()

Example

Let’s put our classes into action. We are going to define a producer and 2 consumers as described at the beginning of this tutorial.

First, our producer will generate random integers and send them to the exchange. If the integer is odd, it will use the routing key “odd” and if the integer is even, it will use the routing key “even”. This code will be running on “producer.messaging.com”.

Next is our first consumer running on “consumer1.messaging.com”. We declare an exchange named “integers” and create a queue and bind it to receive messages with the routing key “odd”. If the exchange and queue already exist, calling declare_exchange() and declare_queue() does nothing.

We define a callback function which prints the odd integer and acknowledge the message so the exchange knows it has been delivered properly.

I hope you enjoyed this tutorial. You can see that RabbitMQ is really easy to use and very powerful. Do not hesitate to add a comment if you have any feedback. If you need help with a project written in Python or with building a new web service, I am available as a freelancer: LinkedIn profile. Follow me on Twitter @laurentluce.