Discover how you can get APIs and microservices to work at true enterprise scale.

This article talks about RabbitMQ, a message broker middleware recommeded by OpenStack for cloud deployments. It complies to AMQP standards and developed in Erlang. The code examples are developed using Python and PIKA library.

1. Message Broker

A message broker is a software component that enables communication across applications in the enterprise application cluster. It also known as Message Oriented Middleware(MOM) in Service Oriented Architecture (SOA). The applications in the enterprise cluster use the message broker like a mail-exchange or a post-office to send messages to other applications.

RabbitMQ complies to AMQP standard, which is open standard for business messages between applications and organizations. It is a binary protocol rather than an interface specification. AMQP standard enables messaging as a cloud service, advanced publish-subscribe pattern, custom header based rourting and programming language independent.

2. RabbitMQ Model

The RabbitMQ model consits of various components. They are: Producer (sender), Consumer (receiver), Exchange , Bindings and Message queues. These components work together as explained below:

The producer sends a message to an exchange

Exchange forwards the message to the queues based on the bindings

The bindings are set by queues to attach to an exchange

Consumers (receiver) receives the messages from their respective message queues.

RabbitMQ ECO System

The producer and consumer of the messages are external entities. The Exchanges, Queues and Bindings are internal entities of the message broker.

2.1 Exchanges

Exchanges are core of RabbitMQ. The producer sends messages to exchanges to forward it to the right consumer. Exchanges make forwarding decision based on exchange type, routing_key in the message, custome header fields in the message, and the routing_key registered by the queue with the exchange.

Exchanges can be configured durable, so that they will survive restarts of the RabbitMQ server. Exchanges can be “internal” to server, so that it can be published by other exchanges inside the server. The exchanges can be configured to auto-delete, while no more queues are bound to it.

2.2 Queues

Queues are used to forward the messages to target consumers and ensures orderly message delivery. The consumer bind itself to a queue with a callback function, so that consumer will be notified on receiving a message.

Queues can be configured durable, so that they will survive restarts of the RabbitMQ server. It can also be configured “exclusive” to a connection, once the connection is closed the queue will be deleted. Auto-delete queues are deleted when no consumer is subscribed to the queue.

2.3 Bindings

Bindings are used by the exchanges to make the routing decision for a message. Basically bindings connects the exchanges with the queues and it uses exchange-type, routing-key set by the queue, custom message headers and routing-key present in the message sent by the producer. If there are not enough bindings to make a forwarding decision for a message, the message can be dropped or sent back to the producer, based on the properties set in the message.

3. Producer and Consumer

In a typical banking system, when a customer withdraws money from the ATM or swipes his credit card at a shop, the customer is notified with a SMS or an email to prevent frauds. In this use case, the core banking system will send a message to messaging sub-systems and continue process other customer requests. The messaging sub-systems listening on the queues will be notified and they take appropriate actions based on subscriber preferences. In this example, the core banking system is the producer of the messages and messaging sub-system are consumer of the messages.

3.1 Producer

As per RabbitMQ design, the producer sends the message to an exchange. The behaviour of the exchange defined by the exchange type, route_key & bindings by the message queues. Let us see how to create a producer using Python pika library. If you do not have RabbitMQ setup, please refer the Appendix-A section to install and configure RabbitMQ.

In the above code segment, first we establish connection to server, where RabbitMQ is hosted. RabbitMQ allows to establish multiple channel within the existing connection and all the communications are operations are tied with that channel. The 3rd line, we create an exchange and named it “SMS-Exchange” and its type is “direct”. In case if the exchange is already created, this statement will not create an error it simply returns it unless there is no conflict in the exchange type.

The function “channel.basic_publish” is used to send a message to the exchange. The producer has to mention that name of exchange and routing_key along with the actual message. In “direct” exchange the “routing_key” will be used to make message forwarding decision. Finally close the connection, if not needed any more.

3.2 Consumer

As per RabbitMQ design, the consumer is the target application where the messages are intended. The consumer must register itself to a queue and bind it with the exchange. If more than one consumers registers to a queue, then RabbitMQ sends the messages to consumers in round-robin fashion.

The connection establishment is exactly same for both producer and consumer. But the consumer has to declare a queue and bind the queue with the exchange along with routing_key. In the above code, the “SMS-Queue” is created and bound with “SMS-Exchange” with routing_key “SMS-Alert”. This instructs the SMS-Exchange to forward the message with routing_key “SMS-Alert” to “SMS-Queue”.

The callback function registered with the queue will be called once message is received on the queue. The list statement “channel.start_consuming()” is a blocking call, where the consumer awaits for any messages on the registered queue.

3.3 Remote Connection

In the above examples we have connected to RabbitMQ server which is present in the localhost. Lets see the code snippet, how to connect to a remote RabbitMQ server:

The above code, first creates a credits object by setting up the user credentials. Then we create a parameters object by setting up the IP address, port number, virtual host path and credential object. This parameter is simply passed to BlockingConnection method to establish connection with the given parameters. Once you establish a connection, the rest of code is exactly same for creating a producer and consumer.

4. Exchange Types

In this section we will learn how to develop various types of exchanges.

4.1 Direct Exchange

A direct exchange delivers messages to queues based on the message routing key. A queue binds to the exchange with a routing key. When a new message with that routing key arrives at the direct exchange, that message is routed to that queue. Direct exchanges are used to distribute tasks between multiple workers in a round robin manner. RabbitMQ load balances the consumers, when multiple consumers listens on the same queue. A direct exchange is ideal for the unicast routing of message.

Producer: The below code segment creates an exchange named “Direct-X” and set the exchange type to “direct”. The postMsg method sends the message to this exchange with routing_key set to “Key1”

Creates a queue “Faount-Q1” and binds to exchange “Fanout-X”, which is of fanout exchange type. Even though the consumer registers a routing_key, it will not have any effect on exchange’s forwarding decision, because it is a fanout exchange.

Now you will see that the message sent by the publisher will be received by both consumers.

4.3 Topic Exchange

Topic exchange route messages to one or more queues based on the message routing key and key-pattern used to bind a queue to an exchange. This is exchange is used to implement publish/subscriber pattern. Topic exchange is suitable for multicast.

Producer creates an exchange named “Topic-X” of type “topic” and sends various messages with different key values.

4.4 Headers Exchange

Header exchanges route message based on attributes in message header and it ignores the routing-key. If a message header attributes matches with the queue binding parameters, then the message is forwarded to those queues.

Producer can set key, value pair (dictionary) in the header of the message sent.

The above producer, creates an exchange named “Header-X” and set the type to “headers”. Then it sends a message with header key set to “Source” and the respective value set to “Core-Banking”. Basically the message header is bundled with the originator of the message, in this example it is “Core-Banking”.

The below Consumer creates a queue named “Header-Q1” and binds it with “Header-X” along with header information it is interested. If you refer the post Method, it

Note that the queue_bind statement takes additional argument, named “arguments” which takes the key-value pair. If there are more than one key-value pair present, x-match:any will ensure that even if a single entry matches the message will be delivered to this queue.

5. Summary

We have seen how RabbitMQ works, understand various componenets and messaging models. RabbitMQ supports high available clustering environments, which will ensure zero down time, higher throughput and increased capacity, which makes it suitable for cloud based installations.

Python 2.7.10 is available https://www.python.org/downloads/. Please install version 2.7.10, since pika support is not yet available for 3.x and it is currently under development.

Pika(v0.9.13) can be used using Python pip.

pip install pika

A.2 Install RabbitMQ Management Plugin

RabbitMQ comes with a web based management console. The software is bundled along with the RabbitMQ installation software we have installed. To enable management console, execute the below command in your linux machine:

After enabling the management plug-in, you can open the http://ipaddress:15672/ in your browser to see the Web Management Interface of RabbitMQ.

A.3 Configuring RabbitMQ (vhost/user/privileges)

Once you have installed RabbitMQ it is ready for production. But we have to make few mandatory configurations in order to allow remote clients, because the default configuration and user privilages will not allow the remove clients to access the server.

The idea here is to create a virtual environment with the RabbitMQ server and provide access to a user, where he can create a Message Queue, Exchange, Read and Write into those resources. So the users are restricted to a virtual environment and resources are contained within the environment.

A.3.1 Create a Virtual Host

rabbitmqctl add_vhost <virtual_host_name>

Note : RabbitMQ will have a default virtual host named “/”.

A.3.2 Create a New User with password

rabbitmqctl add_user <username> <password>

Eg. rabbitmqctl add_user scott tiger

Note: RabbitMQ will have a default user named “guest”, but it cannot used for remote access by external clients. But it allows when the server is acessed locally.