Managing Secrets With Chef Vault

Sep 10th, 2013

Two years ago, I wrote a post about
using Chef encrypted data bags
for SASL authentication with Postfix. At the time, my ISP didn’t allow
non-authenticated SMTP, so I had to find a solution so I could get
cronspam and other vital email from my servers at home. I’ve since
switched ISPs to one that doesn’t care so much about this, so I’m not
using any of that code anymore.

However, that doesn’t mean I don’t have secrets to manage! I actually
don’t for my personal systems due to what I’m managing with Chef now,
but we certainly do for Opscode’s hosted Enterprise Chef environment.
The usual suspects for any web application are required: database
passwords, SSL certificates, service API tokens, etc.

We’re evaluating chef-vault as a possible solution. This blog post
will serve as notes for me so I can remember what I did when my
terminal history is gone, and hopefully information for you to be able
to use in your own environment.

Chef Vault

Chef Vault is an
open source project
published by Nordstrom. It is distributed as a
RubyGem. You’ll need it installed on your local workstation so you can
encrypt sensitive secrets, and on any systems that need to decrypt
said secrets. Since the workstation is where we’re going to start,
install the gem. I’ll talk about using this in a recipe later.

1

% gem install chef-vault

Use Cases

Now, for the use cases, I’m going to take two fairly simple examples,
and explain how chef-vault works along the way.

A username/password combination. The vaultuser will be created on
the system with Chef’s built-in user resource.

A file with sensitive content. In this case, I’m going to use a
junk RSA private key for vaultuser.

Secrets are generally one of these things. Either a value passed into
a command-line program (like useradd) or a file that should live on
disk (like an SSL certificate or RSA key).

Command-line Structure

Chef Vault includes knife plugins to allow you to manage the secrets
from your workstation, uploading them to the Chef Server just like
normal data bags. The secrets themselves live in Data Bags on the Chef
Server. The “bag” is called the “vault” for chef-vault.

After installation, the encrypt and decrypt sub-commands will be
available for knife.

Mode: Solo vs Client

I’m using Chef with a Chef Server (Enterprise Chef), so I’ll specify
--mode client for the knife commands.

It is important to note the MODE in the chef-vault knife plugin
commands affects where the encrypted data bags will be saved. Chef
supports data bags with both Solo and Client/Server use. When using
chef-solo, you’ll need to configure data_bag_path in your
knife.rb. That is, even if you’re using Solo, since these are knife
plugins, the configuration is for knife, not chef-solo. I’m using a
Chef Server though, so I’m going to use --mode client.

Create a User with a Password

The user I’m going to create is the arbitrarily named vaultuser,
with the super secret password, chef-vault. I’m going to use this on
a Linux system with SHA512 hashing, so first I generate a password
using mkpasswd:

Create the Item

The command I’m going to use is knife encrypt create since this is a
new secret. I’ll show two examples. First, I’ll pass in the raw JSON
data as “values”. You would do this if you’re not going to store the
unencrypted secret on disk or in a repository. Second, I’ll pass a
JSON file. You would do this if you want to store the unencrypted
secret on disk or in a repository.

The [VALUES] in this command is raw JSON that will be created in the
data bag item by chef-vault. The --search option tells chef-vault
to use the public keys of the nodes matching the SOLR query for
encrypting the value. Then during the Chef run, chef-vault uses those
node’s private keys to decrypt the value. The --admins option tells chef-vault
the list of users on the Chef Server who are also allowed to decrypt
the secret. This is specified as a comma separated string for
multiple admins. Finally, as I mentioned, I’m using a Chef Server so I
need to specify --mode client, since “solo” is the default.

Here’s the equivalent, using a JSON file named secrets_vaultuser.json. It has the content:

As we can see, I have two nodes that are API clients with access to
decrypt the data bag items. These values are all generated by
chef-vault, and I’ll talk about how to update the list and rotate
secrets later in this post.

Manage a User Password

Let’s manage a user resource with a password set to the value from our
encrypted data bag using Chef Vault.

First, I created a cookbook named vault, and added it to the base
role. It contains the following recipe:

ChefVault::Item.load takes two arguments, the “vault” or data bag,
in this case secrets, and the “item”, in this case vaultuser. It
returns a data bag item. Then in the
user resource, I use
the password:

The important resource attribute here is password, where I’m using
the local variable, vault and the vaultuser key from the item as
decrypted by ChefVault::Item.load. When Chef runs, it will look like
this:

Finally, manage the content of the private key file with a file
resource and the content resource attribute. The value of
vault_ssh["vaultuser-ssh-private"] will be a string, with \n’s
embedded, but when it’s rendered on disk, it will display properly.

Managing Access to Items

There are three common scenarios which require managing the access to an item
in the vault.

A system needs to be taken offline, or otherwise prevented from
accessing the item(s).

A new system comes online that needs access.

An admin user has left the organization.

A new admin user has joined the organization.

Suppose we have a system that we need to take offline for some reason,
so we want to disable its access to a secret. Or, perhaps we have a
user who has left the organization that was an admin. We can do that in a
few ways.

Update the Vault Item

The most straightforward way to manage access to an item is to use the
update or remove sub-commands.

Remove a System

Suppose I want to remove node DEADNODE, I can qualify the search to
exclude the node named DEADNODE:

There’s a bit of a “Chicken and Egg” problem here, in that a new node
might not be indexed for search if it tried to load the secret during
a bootstrap beforehand. For example, if I create an OpenStack instance
with the base role in its run list, the node doesn’t exist for the
search yet. A solution here is to create the node with an empty run
list, allowing it to register with the Chef Server, and then use
knife bootstrap to rerun Chef with the proper run list. This is
annoying, but no one claimed that chef-vault would solve all
problems with shared secret management :–).

Remove an Admin

The admins argument takes a list. Earlier, I only had my userid as an
admin. Suppose I created the item with “bofh” as an admin too:

If you’re familiar with Chef Server’s authentication cycle, you’ll
know that until that private key is copied to the node, it will
completely fail to authenticate. However, once the
/etc/chef/client.pem file is updated with the content from the knife
command, we’ll see that the node fails to read the Chef Vault item:

Note I say this is heavy-handed because if you make a mistake, you
need to re-upload every single secret that this node needs access to.

Removing Users

We can also remove user access from Enterprise Chef simply by
disassociating that user from the organization on the Chef Server. I
won’t show an example of that here, since I’m using Opscode’s hosted
Enterprise Chef server and I’m the only admin, however :–).

Backing Up Secrets

To back up the secrets, as encrypted data from the Chef Server, use
knife-essentials (comes with Chef 11+, available as a RubyGem for
Chef 10).

12345

% knife download data_bags/secrets/
Created data_bags/secrets/vaultuser_keys.json
Created data_bags/secrets/vaultuser.json
Created data_bags/secrets/vaultuser-ssh-private_keys.json
Created data_bags/secrets/vaultuser-ssh-private.json

Since these are encrypted using a strong cipher (AES 256), they should
be safe to store in repository. Unless you think the NSA has access to
that repository ;–).

Conclusion

Secrets management is hard! Especially when you need to store secrets
that are used by multiple systems, services, and people. Chef’s
encrypted data bag feature isn’t a panacea, but it certainly helps.
Hopefully, this blog post was informative. While I don’t always
respond, I do read all comments posted here via Disqus, so let me know
if something is out of whack, or needs an update.