What is RabbitMQ?

RabbitMQ is a message bus, which describes itself as “a messaging broker -
an intermediary for messaging. It gives your applications a common platform to
send and receive messages, and your messages a safe place to live until
received”.

How does Sensu use RabbitMQ?

Sensu services use RabbitMQ (the default Sensu transport) to communicate
with one another. Every Sensu service requires access to the same instance of
RabbitMQ or a RabbitMQ cluster to function. Sensu check requests and check
results are sent over RabbitMQ to the approprate Sensu services.

The RabbitMQ AMQP connection heartbeat in seconds. Enabling can help in
early detection of disrupted TCP connections causing the RabbitMQ client to
attempt re-connection to the server much earlier than if left disabled.
_NOTE: if this setting is not defined or set to 0 then RabbitMQ client
heartbeats are disabled.

required

false

type

Integer

example

"heartbeat": 30

prefetch

description

The RabbitMQ AMQP consumer prefetch value, setting the number of
unacknowledged messages allowed for the channel. This attribute can be used
as a flow control mechanism, to tune message throughput performance.
NOTE: an increased prefetch value should be used if you are experiencing a
backlog of messages in RabbitMQ while the Sensu server(s) load remains low.
Increasing the prefetch value will effect the distribution of messages in
Sensu configurations with more than one Sensu server.

required

false

type

Integer

default

1

example

"prefetch": 100

ssl

description

A set of attributes that configure SSL encryption for the connection. SSL
encryption will be enabled if this attribute is configured.

required

false

type

Hash

example

"ssl": {}

ssl attributes

The following attributes are configured within the "ssl": {} RabbitMQ
definition attribute scope.

cert_chain_file

description

The file path for the chain of X509 SSL certificates in the PEM format for
the SSL connection.

Configure RabbitMQ

Standalone configuration

For standalone configurations, no additional configuration changes are required
beyond what is documented in the RabbitMQ installation guide.

Distributed configuration

For distributed configurations (e.g. where RabbitMQ is running on dedicated
systems), RabbitMQ may need to be configure to listen for connections from
external systems using the tcp_listeners configuration directive.

To enable support for external connections, please ensure that your
/etc/rabbitmq/rabbitmq.config file contains the following configuration
snippet:

[{rabbit, [{tcp_listeners, [{"0.0.0.0", 5672}]}]}].

High-availability configuration

What is a RabbitMQ cluster?

A RabbitMQ broker is a logical grouping of one or several Erlang nodes, each
running the RabbitMQ application and sharing users, virtual hosts, queues,
exchanges, bindings, and runtime parameters. Sometimes we refer to the
collection of nodes as a cluster.

In high availability configurations, all of the Sensu processes need to be able
to communicate with a member of a single RabbitMQ cluster.

High-availability hardware requirements

The Sensu transport does not require message persistence, making throughput the
primary concern for RabbitMQ (i.e. memory is more important than disk
performance). When starting with a RabbitMQ cluster it is important to use
systems with sufficient compute, memory, and network resources. Although it’s
challenging to provide “recommended hardware requirements” for every possible
Sensu installation, we have found that starting with three nodes equivalent to
AWS EC2 m3.medium instances generally provides a solid baseline for
monitoring 1000+ servers, each reporting 10-20 checks (& metrics) at a 10-second
interval.

For the best results when configuring RabbitMQ for high availability
applications, we recommend a three (3) node RabbitMQ cluster, as running fewer
or more nodes introduces additional failure modes.

Install RabbitMQ

Before configuring a RabbitMQ cluster, RabbitMQ must be installed on all of the
systems you will use to provide the RabbitMQ services. For RabbitMQ installation
instructions, please refer to the Sensu RabbitMQ installation guide. Once
RabbitMQ has been installed and started, you may proceed to configure the
RabbitMQ cluster.

NOTE: a RabbitMQ cluster offers several methods of handling network
partitions. A RabbitMQ cluster should not span regions (e.g. WAN links), as
doing so would increase latency and the probability of network partitions.

Set the RabbitMQ erlang cookie on all 3 selected instances to the same value
(e.g. cookiemonster). Create and/or edit the file located at
/var/lib/rabbitmq/.erlang.cookie so that it contains the following
contents:

coookiemonster

WARNING: if this file is edited while RabbitMQ is running it may prevent the
RabbitMQ process from stopping and/or restarting gracefully, which is why
step #1 of these instructions indicates that the RabbitMQ services should be
stopped before making configuration changes.

Start RabbitMQ on all three RabbitMQ systems.

NOTE: The service command will not work on CentOS 5, the
sysvinit script must be used, e.g. sudo /etc/init.d/rabbitmq-server start

sudo service rabbitmq-server start

Reset the RabbitMQ nodes on all 3 selected instances in preparation for
clustering.

sudo rabbitmqctl stop_app
sudo rabbitmqctl reset

Start the RabbitMQ application on one of the three RabbitMQ systems, and
obtain the IP address of this system for instructing the remaining systems on
where to join the RabbitMQ cluster.

sudo rabbitmqctl start_app
ifconfig

NOTE: this step effectively starts the RabbitMQ “cluster”, which the other
RabbitMQ systems will join in the following steps. for the purposes of this
guide we will assume 10.0.1.6 is the IP address of this system.

Join the other two RabbitMQ systems to the RabbitMQ cluster started in step
#6 using the rabbitmqctl utility, and the IP address obtained in step #6.

NOTE: by default, queues within a RabbitMQ cluster are located on a single
node (the node on which they were first declared). This is in contrast to
exchanges and bindings, which can always be considered to be on all nodes in
a RabbitMQ cluster. Sensu requires specific queues to be mirrored across all
RabbitMQ nodes in a RabbitMQ cluster. The Sensu results and keepalives
queues MUST be mirrored. Sensu client subscription queues do not need to
be mirrored.

WARNING: RabbitMQ uses policies to determine which queues are mirrored. The
command provided above will create a RabbitMQ policy to mirror the Sensu
results and keealives queues in the RabbitMQ vhost named /sensu. If
you have modified your configuration to use a vhost other than /sensu,
please proceed accordingly.

Configure Sensu to use the RabbitMQ cluster

Sensu services (e.g. sensu-client) can be configured with the connection
information for each RabbitMQ node in a RabbitMQ cluster without the need for
load-balancing middleware (e.g. HAProxy). Sensu will randomly sort the
configured RabbitMQ connections and attempt to connect to one of the nodes in
the cluster. If Sensu is unable to connect to a node in a RabbitMQ cluster, or
if Sensu loses connectivity to a node in a RabbitMQ cluster, it will attempt to
connect (or reconnect) to the next node in the RabbitMQ cluster.

To configure Sensu to connect to a RabbitMQ cluster, the "rabbitmq"
configuration Hash can be replaced with an Array of RabbitMQ connection
configurations (i.e. "rabbitmq": [] instead of "rabbitmq": {}).

The following is an example RabbitMQ definition that configures Sensu to
connect to a RabbitMQ cluster.

NOTE: the RabbitMQ nodes must first be successfully clustered in order for
Sensu to operate.

Securing RabbitMQ

Sensu leverages RabbitMQ access control and SSL for secure communication. Sensu
was created to deal with dynamic infrastructure, where it is not feasible to
maintain strict firewall rules. It is common to expose RabbitMQ’s SSL port
(5671) without any restrictions, if certain conditions are met. Removing the
default RabbitMQ user guest is mandatory and using a generated user name,
password, and vhost is highly recommended. Enabling SSL peer certificate
verification will ensure only trusted RabbitMQ clients with the correct private
key are able to connect.

RabbitMQ and SELinux

If SELinux is enabled on the machine(s) reponsible for running RabbitMQ, you may
need to make minor policy changes in order for RabbitMQ (and Erlang) to run
successfully.

To list the available SELinux booleans, run the following command:

sudo getsebool -a

For some reason, enabling the NIS boolean allows RabbitMQ to bind to its TCP
socket and operate normally.

sudo setsebool -P nis_enabled 1

RabbitMQ SSL

RabbitMQ SSL and Erlang 18.3

For users of Erlang version 18.3, the SSL implementation that RabbitMQ relies on
changed in such a way that additional configuration parameters are needed for
SSL encrypted communication between Sensu and RabbitMQ (specifically affecting
Sensu Enterprise users). This does not apply to Erlang 19.0+

In order to enable SSL communication between RabbitMQ installations running on
Erlang 18.3 and Sensu, it is necessary to configure the specific TLS version
(i.e. {versions, ['tlsv1.2']}) and ciphers (i.e. {ciphers,
[{rsa,aes_256_cbc,sha256}]}) that RabbitMQ will accept, and to reject clients
with no certificate (i.e. {fail_if_no_peer_cert,true}).

Newsletter

Subscribe to the newsletter to get product updates about Sensu, including notifications when new releases are available. No more than one email per week, no less than one email per month. #monitoringlove.