binford2k.com

Sharing secrets with Puppet, secretly.

If you’ve used Puppet for anything non-trivial, you’ve almost certainly used it
to configure something secret. Perhaps you’ve configured an application with a
database password. Perhaps you’ve configured a local maintenance user account
with a private SSH key. Something that might seem obvious in retrospect is that
these secrets exist in the catalog–and by extension all reports and any other
tooling that uses them. Anyone with access to the catalog or raw reports also
has access to your secrets. All your secrets.

Before we go any further, let’s set the stage a bit and make sure we’re not
fearmongering here. Secrets have to be transmitted somehow and we’ve been doing
this for years. I’m sure some of you remember trying to figure out how to
protect your SSL certificates with a passphrase without having to manually type
it in every time you restarted Apache. Most of us gave up and just didn’t use a
passphrase. It’s the classic bootstrapping problem. If your secrets weren’t in
your Puppet catalog, they’d be in a Kickstart script or the like, or you’d be
manually provisioning machines. As a result, in a well architected
infrastructure, Puppet catalogs and reports are considered to be sensitive
information and access to them is strictly limited.

But it would sure be nice if it didn’t have to be.

Transparent encryption:

My node_encrypt module
allows you to securely transmit secrets to any Puppet agent in your
infrastructure in a safe and secure fashion, with no configuration or setup
required. It will reuse and piggyback on top of the existing Puppet PKI
infrastructure, meaning that each node has secrets encrypted for it and for it
alone. Let’s look at it in practice. Once you’ve installed the module1, you can
declare an encrypted resource like as the following:

This manages a file on disk using most of the same attributes as a standard file
resource type. Aside from content, the parameters actually are passed directly
to a file resource. This means that you can use file attributes as documented.
The Puppet master will encrypt the content of the file using that agent’s public
key. Only that agent will be able to decrypt it–using its private key, of
course. The actual plain-text content of the file will never exist in the
catalog or in any reports. Instead, the catalog will look something like this:

When the agent enforces the catalog, the ciphertext will be decrypted using the
agent’s private key and validated against the Puppet CA certificate chain. And
that’s it.

Plug and play catalog encryption.

Advanced usage:

Ok, I lied. There are a few situations in which it doesn’t just magically work
out of the box. But even these are pretty easy to set up and use. Most of the
time you won’t have to bother with this section.

Command line encryption:

If you’d like to pre-generate your ciphertext instead of encrypting on each
catalog compile, you can do that with the puppet node encrypt subcommand. It
requires a target node and will encrypt text from the the command line.

This cipher text can be pasted directly into a manifest, or (better) saved as
Hiera data and passed to the node_encrypt::file resources as the
encrypted_content parameter. In this way you can slightly lower the load on your
master, or you can generate catalogs with secrets on nodes that are not the CA.

Using with compile masters:

Speaking of non-CA nodes, let’s take a moment to discuss certificates. Because
the CA node signs all agent certificates, it has the public certificate for each
node in the Puppet infrastructure. It is the only node with these certificates.
That means that out-of-the-box, the CA is the only node which can encrypt
secrets.

Clearly this won’t work for many infrastructures. If you’ve got more than 800 or
so nodes, you’re almost guaranteed to have some compile masters. The
node_encrypt::certificates class was designed for this use case. It will
synchronize certificates across your infrastructure so that encryption works
from all compile masters. Simply include this class on all your masters,
including the CA or Master of Masters. It will make sure that all compile
masters have all agents’ public certificates by creating a fileserver mount on
the CA node and then copying certificate files from this mount point to each
compile master.

If you’d like to limit access to the certificate mountpoint, you can pass a
comma-separated list of compile master nodes as the whitelist parameter. Note
that in either case, only public certificates are ever available in this
fashion.

Once Puppet has run on the CA node and then each of the compile masters, they
too will be able to encrypt secrets for any node in the infrastructure. This
certificate synchronization will stay current, meaning that certificates for any
new nodes will automatically be synchronized to compile masters.

Parser function:

The encryption is performed by a parser function during catalog compilation. You
can call this function yourself to manually generate ciphertext. This will
mostly be useful if and when third-party modules adopt this encryption scheme.
For example, with this pull request
to _rc’s datacat module, you can encrypt values in the catalog before they’re
constructed on the node. I’ll do a followup post soon on using this encryption
in your own module.

datacat{'/tmp/test':template_body=>'Decrypted value: <%= @data["value"] %>',show_diff=>false,}datacat_fragment{'encryption test':target=>'/tmp/test',data=>{value=>node_encrypt('This string will not be included in the catalog.'),},}

Ecosystem:

Other tools certainly exist in this space. One of the most well known in the
Puppet ecosystem is Hiera Eyaml,
which is intended to protect your secrets on-disk and in your repository. With
Hiera eyaml, you can add secrets to your codebase without having to secure the
entire codebase. Having access to the code doesn’t mean having access to the
secrets in that code. But the secrets are still exposed in the catalog and in
reports. You should be protecting them as well using something like
node_encrypt.

You can (and should) use eyaml to store your secrets on disk, while you use
node_encrypt to protect the rest of the pipeline.

Another tool is Conjur. This is an entire secret
management platform, with all the benefits and drawbacks that come along with
it. Once it’s configured across your infrastructure, you can pass secrets
out-of-band so that the Puppet master never even sees them. It requires some
configuration and each node requires authentication to the secret server–the
standard bootstrapping problem again. It’s not as transparent, and it’s
certainly not zero-config. Nevertheless, it’s worth investigating due to its
integration with many other tools in your infrastructure and its richer control
over secret dissemination.

GitHub Project:

I’d love it if you tried out the module and gave me some feedback. Is there
something that it’s missing? Is there an edge case that I’m not catching? Did it
eat your puppy and make faces at your baby? File an issue or send me a pull
request!