Federated Queues

In addition to federated exchanges, RabbitMQ supports federated queues. This feature provides a way of balancing the load of a single queue across nodes or clusters.

A federated queue links to other queues (called upstream queues). It will retrieve messages from upstream queues in order to satisfy demand for messages from local consumers. The upstream queues do not need to be reconfigured and they do not have to be on the same broker or in the same cluster.

The following diagram demonstrates several federated and unfederated queues in two RabbitMQ nodes connected using queue federation.

All of the configuration needed to establish the upstream links and the federated queue is in the broker with the federated queue (downstream).

Typical Use Cases

The typical use would be to have the same “logical” queue distributed over many brokers. Each broker would declare a federated queue with all the other federated queues upstream. (The links would form a complete bi-directional graph on n queues.)

Such a logical distributed queue is capable of having rather higher capacity than a single queue. It will perform best when there is some degree of locality; i.e. as many messages as possible are consumed from the same queue as they were published to, and the federation mechanism only needs to move messages around in order to perform load balancing.

Using Federated Queues

Federated queues are declared just like any other queue, by applications. In order for RabbitMQ to recognize that a queue needs to be federated, and what other nodes messages should be consumed from, downstream (consuming) nodes need to be configured.

Configuration is done by declaring policies. A policy is a pattern that queue names are matched against. Matching queues will be federated. A federated queue can belong to only one policy. If multiple policies match a queue name, the policy with the highest priority is applied. When two policies have the same priority, the matching one is chosen non-deterministically.

Policies can be configured manually using RabbitMQ management UI, programmatically with rabbitmqctl or over HTTP API.

A federated queue can be ‘upstream’ from another federated queue. One can even form 'loops’, for example, queue A declares queue B to be upstream from it, and queue B declares queue A to be upstream from it. More complex multiply-connected arrangements are allowed.

How It Works

The federated queue will connect to all its upstream queues using AMQP (optionally secured with SSL). When declaring or configuring the federated queue each upstream queue is listed with the connection properties to be used to establish the link.

The federated queue will only retrieve messages when it has run out of messages locally, it has consumers that need messages, and the upstream queue has “spare” messages that are not being consumed. The intent is to ensure that messages are only transferred between federated queues when needed. This is implemented using consumer priorities.

If messages are forwarded from one queue to another then message ordering is only preserved for messages which have made exactly the same journey between nodes. In some cases messages which were adjacent when published may take different routes to the same node to be consumed; therefore messages can be reordered in the presence of queue federation.

Each individual queue applies its arguments separately; for example if you set x-max-length on a federated queue then that queue will have its length limited (possibly discarding messages when it gets full) but other queues that are federated with it will not be affected. Note in particular that when per-queue or per-message TTL is in use, a message will have its timer reset when it is transferred to another queue.

Unlike with federated exchanges, there is no limit to how many times a message can be forwarded between federated queues. In a set of mutually-federated queues, messages will move to where the spare consuming capacity is - so if the spare consuming capacity keeps moving around then so will the messages.

Limitations

basic.get cannot retrieve messages over federation if there aren’t any in a local queue (on the node the client is connected to). Since basic.get is a synchronous method, the node serving a request would have to block while contacting all the other nodes to retrieve more messages. This wouldn’t sit well with federation’s availability promise (partition tolerance).

Brokers running different versions of RabbitMQ can be connected using federation. However, since queue federation requires consumer priorities, it is not possible to federate a queue with a broker running a RabbitMQ version prior to 3.2.0.

Pitfalls

Federated queues cannot currently cause messages to traverse multiple hops between brokers based solely on need for messages in one place. For example, if you federate queues on nodes A, B and C, with A and B connected and B and C connected, but not A and C, then if messages are available at A and consumers waiting at C then messages will not be transferred from A to C via B unless there is also a consumer at B.

It is possible to bind a federated queue to a federated exchange. However, the results may be unexpected to some. Since a federated exchange will retrieve messages from its upstream that match its bindings, any message published to a federated exchange will be copied to any nodes that matching bindings. A federated queue will then move these messages around between nodes, and it is therefore possible to end up with multiple copies of the same message on the same node.