A blog about servers and junk

I recently finished my Intermediate Sensu Training on Udemy.
It was a ton of work but I’m glad I got it all together. Part of that training
includes how to deploy and configure Sensu with four
of the most popular open-source configuration management tools:
Puppet,
Chef, Ansible, and
Salt.

The Sensu Decree

In order to do the training I had to learn each of these tools enough so I could install
a baseline Sensu installation. Here is what I reproduced with each iteration:

A Sensu client, Server, and API Setup and Running

RabbitMQ Server, User, and Sensu Vhost ready for use. (no SSL)

Redis installed and running for state

A Sensu check (check_disk and/or check_apache)

The Sensu Mail handler to send emails for alerts

The Uchiwa Dashboard

All on one host (localhost)

This was no small feat, and required using a non-trivial number of features of each configuration management system to get the job done.

Here were some other guidelines that I followed in this exercise:

Always use 3rd party modules/cookbooks/etc. Use official ones if possible.

Use the local-execution mode provided by the configuration management tool
(no client/server setup)

Follow official docs when available for general guidelines for things like
installation.

Differences in things like config file names or versions of Redis are
inconsequential. As long as Sensu behaved the same I considered it complete.

No considerations for security (out of scope for this exercise)

Review of Each Tool

Puppet

Puppet In General

Puppet is my “native language” when it comes to configuration management, so it
is a little hard for me to imagine what it is like to not know what it is
like to know exactly how it works.

Puppet has a custom DSL to describe configuration in terms of “types”. These are
the primitives that you can build infrastructure upon, things like “files, “package”,
and “service”. Third party modules can extend that language with custom types,
allowing you to abstract over the “raw” types. For example, the RabbitMQ has a
type for providing rabbitmq_users, which do not correspond to a particular
config file or anything, but instead can only be added by special invocations
of the rabbitmqctl command.

Puppet strongly emphasizes code-reuse. The Puppet Forge is the registry where you can upload and
share modules. The Forge has a number of methods to help indicate code quality.
It also exposes “officially supported” and “officially approved” modules, for
extra approval stamps. While the forge may have a very “long tail” of modules
that do very common tasks, the set of officially-supported and
officially-approved modules leaves behind a good selection of high-quality
modules ready for re-use.

A common criticism of Puppet is that it does not apply resources in the order that
they are declared in the manifest. Instead, Puppet internally calculates a directed
graph of resources and their dependencies, and executes them in a dependent order, which
is not necessarily in the order in which they are parsed. This is similar to how Linux
package managers install packages. If you run apt-get install apache libc libssl,
the packages will not necessarily get installed in the order that they were
specified on the command line.

Puppet also comes with Hiera,
a convenient hierarchical key/value store. This store allows users to override
and set site-specific settings to Puppet modules without having to fork or modify
them. Hiera encourages custom hierarchies that meet your business needs, allowing
users to specify settings in a way that makes the most sense for their environments.
And example hierarchy might look something like:

Then Hiera looks up parameters from most-specific (hostname) to least-specific
(common), and returns the first value that is available.

Review of the Sensu Puppet Module

The sensu-puppet module is a
first-class citizen in the Sensu world. It has native types for the Sensu JSON
files that it manages, as well as a sensu-gem type for easily installing
rubygems with the embedded Sensu ruby.

The Sensu Puppet module only manages Sensu, and has no integration with any
other RabbitMQ, Redis, or any other module. To me this is expected, in the
Puppet world it would be the job of a profile to combine the Sensu module
with RabbitMQ and other things. For the most part this integration is left as
an exercise to the reader.

The Sensu Puppet module also doesn’t manage Uchiwa. That requires a
different puppet module. Again to me
this is a good thing, I hate it when tools try to do too much.

The actual codebase is actively maintained and reasonably active, with a few
releases per year. The Puppet Forge rates it almost perfectly for module quality.
The code has excellent unit test and acceptance test coverage. As far
as Puppet modules go, the Sensu Puppet module is a great example of a
well-maintained piece of code.

One downside the “completeness” of the module is that sometimes new features of
Sensu are released, and the puppet-module will lag. The configuration inputs to
the puppet module are well-typed, and not just free-form hashes. This gives a
lot of guardrails and helps ensure config files are correct before they hit
the disk, but it means that some features are not usable until the Puppet
module can account for them.

Although the code worked, there was a significant bug that prevented the
module from ever converging. This was annoying but allowed me to test the code.
This bug looks to be fixed in master.

Chef

Chef in General

Chef is not as old as Puppet, but is certainly a mature product. Chef is “just
ruby” when it comes to its configuration language. The upside to this is that
Ruby developers can theoretically dive in and hack on stuff. The downside to
this is that being “just ruby”, “leaves a lot of rope to hang yourself”.

One nice feature provided by the Chef company is their hosted chef solution, which
allows people to get started without hosting a Chef-server.

The Chef toolset also comes with the knife command, which is a great command
line tool for interacting with the Chef-server. It also is a parallel-ssh tool,
manipulates chef cookbooks, and can also launch ec2 (and other) instances. (did
they take the kitchen-sink metaphor too far?)

The Chef Supermarket serves as the public
registry for Chef cookbooks. There are not too many quality indicators to see,
to help find which cookbooks are any good. The best metric I could see is just
sorting by “followers”. This is made up by the fact that there are over
a hundred officially supported cookbooks.

Probably the most difficult aspect of Chef for me to understand was how
attributes interact. This confusion is probably most obvious when you look at
Chef’s 15 levels of attribute precedence. It seems
to me that there should be a more obvious way for intent to flow, but I could
be just spoiled by Puppet’s Hiera.

Review of the Chef-Sensu Cookbook

The Sensu Chef Cookbook is also a
first-class citizen in the Sensu-world. Chef is the “native config language” of
Sean Porter, the main author of Sensu. This
gives a lot of credibility to the Cookbook, and shows in the contributor page.

The Cookbook itself is feature complete, with recipes for installing and
configuring all aspects of Sensu.

The scope of the cookbook includes all Sensu related technologies, including
RabbitMQ, Redis, and Uchiwa. It is certainly “batteries included” and on by
default. It even downloads and compiles Redis from source for you.

Another example of this “batteries included” design is the RabbitMQ module
settingApt attributes. Like the
above Redis example, this behavior surprised me, but technically it is not related to
the Sensu chef cookbook.

At the same time, wrapper cookbooks are recommended as a
method to combine multiple cookbooks together in a coherent way. I think in
general I just expected the wrapper cookbooks to do more and the main Sensu
cookbook to do less.

The cookbook does have an integration test suite, but it is not run via Travis.
The code is under active development, and does multiple releases a year. It has
native support for Chef data bags for
transporting the RabbitMQ SSL support, which is a nice touch (Not tested in
this review).

Ansible

Ansible in General

Ansible is a relative newcomer to the configuration management space. Ansible
uses yaml files to define desired state. The yamls files are a nice way to
represent things, but it would be misleading to think that Ansible is just yaml
files. Ansible has its own DSL and uses Jinja2 templating, which is parsed
over the contents of the yaml.

The Ansible Galaxy is the community registry for
uploading shared roles. You can sort by rating
to try to get a better idea about which roles are potentially higher quality
than others.

There doesn’t seem to be any official roles/playbooks. The closest there is to
official roles is the
ansible-examples
repository. But click the link and look at the lamp_simple example. There is
no code-reuse at all! Every example re-invents how to install apache, install
ntp, configure iptables, etc. What’s up with that?

While the yaml files may make it very easy for beginners to make playbooks that
get things done quickly, I don’t think they will work out great as
infrastructure expands. The abstractions just are not there

Another sign, to me, that Ansible has the wrong abstractions is that
so many roles are distro specific.
Not many have the necessary code to work on both “CentOS” and “Debian”. There
is a generic package
type, but very few roles use it? Check out the original author’s opinion on the subject.
Look at the examples! They all only work on yum based distributions.

I’ve read lots of posts of people migrating to Ansible and loving it.
Personally, I don’t get it. The abstractions are too low-level. If you are lucky,
then the Ansible core has a Module to manipulate the resources on the host,
like RabbitMQ stuff. If you are
unlucky, then the only primitives you have available are yaml files and
running commands and parsing stdout.
Or you can write your own module.

Ansible Sensu Playbook Review

There is no official Sensu Ansible playbook. I was not able to find any playbooks
that support RedHat-based distributions.

The sensu_check module is part of the “Extras”, but it is only a very small
part of deploying Sensu, and it has no cohesion with the playbook that actually
deploys Sensu itself. There is no way to extend sensu_check without forking
ansible-modules-extras. It
can’t consume arbitrary check metadata.

In the end, to meet my needs I had to construct hashes myself and deploy them
to disk as JSON. The playbook-provided way to deploy sensu checks is to have
them all contained in the single
sensu_checks
variable.

Salt

Salt in General

Salt is also a relative new-comer to the configuration management world. As a
user, Salt feels very similar to Ansible. They both use yaml files to represent
the desired state of the system. Both use Jinja templates. Both require the
“advanced” system interaction to happen with the core stuff, and the
Salt formulas can be just yaml with no real code.

Salt takes a different approach to sharing community code compared to the
other configuration management systems. Salt keeps all the official formulas in
one GitHub project. The docs recommend
forking the formula
for your own use. On the plus side, having “canonical” formulas for common
tasks reduces duplication and encourages code re-use. The downside is that… it
encourages forking? These formulas in general are not that extensive. They
don’t have releases or any kind of testing in place.

Salt’s Pillar is a powerful tool for separating configuration from code.
It is similar to Puppet’s Hiera. Pro: separate config from code; keep
the site-specific variables in a separate folder than the formulas. Con:
formulas have to be “pillar-aware”. There is no equivalent to Puppet’s
automatic parameter lookup.

Sensu Salt Formula Review

For my testing, I used the official
Salt-formula. There is
a sensu-salt repo on the official Sensu project, but it is not really suitable
for production use in my opinion.

For the most part, the formula did what it said on the tin. Of course, like
Ansible, the only way I was able to deploy checks in a flexible way was to
construct my own Hashes and deploy them as JSON directly. There is no such thing
as a sensu_check type in Salt.

I was not able to get rid of the hard-coded cron check.
I guess goes with the idea that they expect you to fork the repo and make your
own local changes to meet your needs. I thought I should maybe open an issue for
this, but the file has been there for a year and nobody else has complained. I
figured it was just me, and maybe I should get over myself and accept the fact
that I got a free cron check!

In my own testing, I used the native gem provider with a special path to
Sensu’s gem binary to install Sensu gems. But then I discovered that the
formula did this too, but in twodifferent ways,
using the cmd.run method instead of the native gem method. I didn’t really
like this, but at the same time, this is the first time I’ve ever used Salt.

As far as I can tell, to do more advanced Sensu config things, like filters or
mutators, you are expected to fork the formula and drop in the json file into
the right directory.

Comparison

A rough opinionated comparison between the tools, with regards the tool itself
and the tool in conjunction with Sensu. “High” doesn’t necessarily mean “good” here:

Puppet

Chef

Ansible

Salt

Review of The Config Management Tool in General

Version used

3.4.3

12.4.1

1.5.4

2015.5.3

Third Party Module Easy of Use

High

High

Medium

Low

Official Sensu Support for the Tool

High

High

Low

Low

Reproducibility

High

High

High

High

Easy of use getting started

Medium

Medium

High

Medium

Language extensibility

High

High

Low

Low

Separation between config data and code

Hiera

Databags/Attributes

just variables?

Pillar

Module re-usability?

High

High

Low

Low

Review of the Sensu Module/Cookbook/Etc

Version of the module Used

1.5.5

2.10.0

0.1.0

c6324b3

Sensu Module Feature Completeness

High

High

Medium

Medium

Sensu Module Integration with Other Modules

Low

Extreme?

None

None

Sensu Module Flexibility

High

High

Medium

Low

Sensu Module Re-usability

High

High

High

Low

How Opinionated Was It?

Low

High

Low

Medium

Usability with Sensu’s Embedded Ruby

Yes

Yes

Not natively

Sorta

Conclusion

The way I see it, there are two camps. Chef and Puppet both provide a rich
language to build modules with. For example, the PuppetLabs RabbitMQ module contains all the code
to interact with RabbitMQ. The main Puppet codebase doesn’t know anything about
RabbitMQ. The same goes for Chef. Both Chef and Puppet also have their own DSL.
Puppet uses yaml files for Hiera, but they are for config only, unlike
Ansible/Salt.

In the other camp is Ansible and Salt. They have a simplified config language,
and require the help from the core software to do the “heavy lifting” of the
raw types. For example, the Salt RabbitMQ formula
requires the help of core Salt RabbitMQ module
to provide the primitives.

Final Thoughts

Puppet

Directed graph dependency ordering, not parse-order driven

Type/Provider system and defined types provide the right abstraction layers to build upon.

Hiera provides a good separation of config/code, making it easier to reuse modules without modification.

Strong culture of testing

Lots of good supported modules

High deployment overhead and language learning curve

Chef

LWRP system provides the right abstraction layers to build upon.

Knife tool does do a lot of cool stuff

Lots of good supported cookbooks

Strong culture of testing

“Just ruby”

15 levels of attribute precedence is insane

Ansible

Low deployment overhead and low learning curve

“Just yaml files”

Lack of type/providers means that playbooks use “apt” and “yum” directly, which kinda sucks

Salt

Pillar provides a nice separation of config/code, which is good for formula-reuse, if the formula is pillar-aware

Centralized formulas emphasize consolidated development effort

No strong state testing emphasis or framework

Going Further

If you want to know more about Sensu, of course you can take my training course: