A blog about servers and junk

Getting Started With Sensu Using Puppet. For Real.

Nagios. So familiar. I feel like I’ve run Nagios at every job I have ever had.

Talk to most ops people, even at really big places, and they will probably admit
to using it.

Puppet’s exported resources
takes away some of the pain, but sometimes I think to myself, there must be a
better way to do this. Sensu might be that better way.

Let’s try it out, but gosh, I am SO lazy. I cannot be bothered to read the
installation instructions. All I want to do is install the puppet module,
add a couple of lines to my manifest, and let puppet do the rest. Then I
can run puppet agent in debug mode so when my boss comes by it looks like
I’m REALLY busy.

Step 1: Game plan

I’ve got a test server I know I want to be my sensu server. I know I’m going to
have enable the sensu client run on the servers I want monitored. Here are my
goals:

Have sensu-server configured on my server (call it mon1)

Have sensu-client configured on my client (call it client1)

I want a dashboard

I want a an email alert

I don’t want to have to ssh to my clients to do anything. (I have puppet to do that for me, duh.)

Step 2: Puppet Module

My puppet master is not mon1, but it doesn’t matter. I run on the puppetmaster

Ok, good start. So… the “For Real” part in the blog post title is about those
other things that most howto’s don’t mention. Unless you already have
RabbitMQ and Redis installed, you will need those modules. Don’t know how to
run Redis or configure RabbitMQ? It’s ok, neither do I.

Step 2A: SSL Certs

Yea, I know what you are thinking. Kyle, I already have SSL certs for my
infrastructure, do I have make another set? Yes. I think so. I’m not smart
enough to use existing certs.

Joe Miller has made a pretty easy script to generate some. For RabbitMQ
you can basically use a single client and server key and let puppet distribute them:

Step 5B: Your first server-side check

Sometimes you need to have the servers do the checking. Not everything can be
a client-side check. Sometimes you really do want your monitor server to be
able to ping your clients (or check http, etc).

The Sensu documentation doesn’t seem to have examples of this. The only way
I know how to do it is with stored configs
with something like
puppetdb:

In this case, the @@ in front of the sensu check tells puppet to not actually
make it, just store it. Then the «||» operator on the server side will take
those stored configs, and make them.

Conclusion

Sensu is still new, but it shows a lot of promise. It is built from the ground
up to be configured by machines, not by humans. It is also designed to scale,
allowing you to grow your RabbitMQ cluster and your Sensu-servers at will.

Absent from Sensu (at the time of this writing) is the infrastructure for
complicated time periods, escalations, etc. Maybe it is better that way? It
does feel a little more unixy, with each individual Sunsu piece handling a
very particular function.