Keeping application secrets with Vault

I’ve been talking to one of our security guys recently about providing my piece of software with secret certificate and in the meanwhile keeping that certificate out of my hands. Apparently, managing application secrets is not an easy task. Later that day I checked out one of the tools that supposed to make such tasks simper – HashiCorp Vault – and was quite impressed. I didn’t realize how big the problem domain is, and how many tools and tricks you have to consider in order to build a solution for that. Today I want to go through the basics of managing secrets with Vault and hopefully highlight few things what impressed me the most.

What is Vault

Vault is a command line tool and a RESTful service that’s designed to safely keep application secrets such as logins, passwords, tokens or certificates. Obviously, along with keeping it also provides the secrets back on demand. What’s more, vault can work with custom secret backends (e.g. AWS, databases, PKI) and generate temporary credentials on a fly. All requests for secrets will be authenticated and audited, so security stuff should always be able to tell who requested what.

Installation

Almost everything that HashiCorp produces is downloadable as simple zip archive and Vault itself is not an exception. I downloaded it on Mac, so all examples will have unixy flavor. However, running it on Windows wouldn’t be much different.

“Hello world!”

Let’s try to run something that will give us a feeling of what we’re going to deal with. As vault involves command line tool and a service, we need that service to start first. For demo purposes, vault server –dev will do the trick:

Start development server

Shell

1

2

3

4

5

6

7

8

9

10

11

12

vault server-dev

#==> Vault server configuration:

#...

#The only step you need to take is to set the following environment variables:

#

# export VAULT_ADDR='http://127.0.0.1:8200'

#...

#Unseal Key: JMe7A0cECXKLnSP8pQmKIrku3if1NWzgrTiLSeRQcPA=

#Root Token: 18924580-71bd-f5e0-a218-50bfada86741

#

#==> Vault server started! Log data will stream in below:

#...

Another thing we need to do is to configure command line vault to talk to the server via HTTP, not HTTPS, as it would try to do by default. We can do that by setting VAULT_ADDR environmental variable and vault server –dev already wrote the whole command for that:

Running a real server would require us to authenticate first, but demo server takes care of that for now. However, talking to vault server via HTTP requires valid authentication token even now. The following call will fail:

In real life it would work similarly: somebody would’ve put the secrets into the store and then applications would request them back, each using its own authentication token.

But what we’ve done so far doesn’t look that much impressive. After all, simple Consul key-value store with enabled authentication could’ve done the same. In order to understand what makes the tool so cool we need to dive a little bit deeper.

A little bit more realistic example

Let’s try to do the same example, but in more production-like fashion.

Starting the server

You can’t just start vault server without configuring it first. Configuration’s bare minimum should at least involve persistent storage for encrypted secrets and HTTP(S) endpoint details. There’s whole number of possible storages for secrets (Consul, S3, PostgreSQL, Azure to name a few), but I’ll use regular file system for storage and will keep the same HTTP settings as in –dev mode.

config.hcl

JavaScript

1

2

3

4

5

6

7

8

storage"file"{

path="./secrets"

}

listener"tcp"{

address="127.0.0.1:8200"

tls_disable=1

}

By the way, HashiCorp came up with its own configuration format called HCL (you just saw it above), which is a more human readable version of JSON. I’m quite skeptical when it comes to inventing new languages, but what do I know. On the bright side, we indeed can start the server now:

Start vault server

Shell

1

vault server-config config.hcl

Unsealing the vault

That’s really cool idea. The vault can be in two states: sealed (default), where all secrets are in persistent storage and vault has no idea how to decrypt them, and unsealed (desired), when all the secrets finally get into the memory. The process of unsealing the vault is brilliant.

When you initialize vault the first time it creates a master key for decryption and splits it into several pieces (e.g. 3). Sealed vault never knows the key, so in order to unseal it you have to provide at least few pieces (e.g. 2) of the master key in order to restore it. If different people own different pieces, single person can’t unseal the vault – somebody else must participate in the process. But in case of emergency anyone can seal it back.

Let’s see that in action. I’m going to initialize newly created vault and tell it to split master key into three pieces and require at least two of them to unseal the vault:

Initialize the vault

Shell

1

2

3

4

5

6

7

vault init-key-shares3-key-threshold2

#...

#Unseal Key 1: tQatbtCzOeX5f2mL4Mbd30drim5/CdIOODdBlZ0TxLQ/

#Unseal Key 2: cHCFu9rvz6vwk9jH6dtg+7Ct0EvjN02FRFCPI/dFVFlO

#Unseal Key 3: r3jt+sDaW1UYg1psAQcHdoOAGmEEPlikVmHt+YRqVu4e

#Initial Root Token: 5565017e-4b50-bb5f-073e-3def713ba4f1

#...

Having the keys, we need to call vault unseal as many times as key-threshold options was set to. Obviously, each time with different keys.

Unlock the vault

Shell

1

2

3

4

5

6

7

8

9

10

vault unseal

#Key (will be hidden):

#Sealed: true

#Key Shares: 3

#Key Threshold: 2

#Unseal Progress: 1

#Unseal Nonce: 4fd58cbe-222a-2b05-57f3-5b3736676540

vault unseal

#...

Now vault is ready to share with its data.

Authentication

Of cause, unauthorized reading from and writing to a vault is nonsense. Caller have to identify itself with a token, and the only token we have by default is root token (returned by vault init). Let’s try it.

Authenticate as root

Shell

1

2

vault auth18924580-71bd-f5e0-a218-50bfada86741

#Successfully authenticated! You are now logged in.

While we’re authenticated, let’s also add few secrets to the vault. We’ll need them later.

Write some secrets

Shell

1

2

vault writesecret/db-production login="1"password="1"

vault writesecret/db-staging login="dev"password="pass"

Creating a policy and a new token

Using ‘root’ account is not particularly safe. We could’ve created new, less privileged token, but as we’re logged in as ‘root’, new token would become a child token of ‘root’ and without providing custom policy it also would inherit all ‘root’ permissions. We need to create custom policy first.

A policy is basically another HCL file that says what paths a token will have an access to. If I wanted to create some sort of development policy that provides read-only access to secret/db-staging secrets and zero access to everything else, I’d come up with something like this:

dev-policy.hcl

JavaScript

1

2

3

path"secret/db-staging"{

capabilities=["read"]

}

After saving the policy to e.g. dev-policy.hcl file and writing it to the vault with vault policy-write development dev-policy.hcl command, we finally could create a new token with limited access:

Create new token with development policy

Shell

1

2

3

4

5

6

vault token-create-policy development

#Key Value

#--- -----

#token 77ab8047-34e2-34ac-1245-ce98c00d9c12

#...

#token_policies [default development]

It’s easy to test that the new token indeed has limited permissions:

Test access restrictions

Shell

1

2

3

4

5

6

7

8

9

10

11

12

vault auth77ab8047-34e2-34ac-1245-ce98c00d9c12

vault readsecret/db-staging

#Key Value

#...

#login dev

#password pass

vault readsecret/db-production

#Error reading secret/db-production: Error making API request.

#...

#* permission denied

Using custom secrets backend

So far we’ve been using Key-Value secrets backend, which is enabled by default and mounted to secrets path. That’s why all our keys had to start with secrets/. However, that’s not the only backend we could’ve used. Imagine, how cool it would be if instead of hardcoded logins and passwords we provided temporary ones, generated on demand?

Custom secrets backends can do that. As we already started with database credentials example, let’s use built-in database backend to generate read-only MySQL credentials on the fly. Before we begin, let’s switch back to root token:

Authenticate as root

Shell

1

vault auth18924580-71bd-f5e0-a218-50bfada86741

Then, mount database backend:

Mount database backend

Shell

1

2

vault mount database

# Successfully mounted 'database' at 'database'!

At this point we need MySQL server. I have Docker installed on my machine, so we can get one in seconds:

Start MySQL container

Shell

1

2

3

4

docker run--rm-d\

-p3306:3306\

-eMYSQL_ROOT_PASSWORD=rootpwd\

mysql

This command will also open port
3306 and assign
rootpwd password to MySQL
root account, both of which we’ll provide to database backend later.

At this point we can configure database backend and define what exactly read-only role (I called it
viewer ) means for us:

It basically boils down to CREATE USER SQL statement with “SELECT” permission for whoever tries to get credentials for viewer role. I didn’t specify how long that newly created user should exist, so vault will default to some value on its own.

Let’s test if the whole thing works:

Generate database credentials

Shell

1

2

3

4

5

6

7

8

vault readdatabase/creds/viewer

#Key Value

#--- -----

#lease_id database/creds/viewer/d9ec67ae-07c9-c6ba-ae5a-9e49f74a8c78

#lease_duration 768h0m0s

#lease_renewable true

#password A1a-9xstpvu59txsx6qw

#username v-root-viewer-pxutp8uzw7t1z53w97

And after getting into MySQL container:

Test generated credentials

Shell

1

2

3

4

5

6

7

mysql-uv-root-viewer-pxutp8uzw7t1z53w97-pA1a-9xstpvu59txsx6qw

#mysql: [Warning] Using a password on the command line interface can be insecure.

#Welcome to the MySQL monitor. Commands end with ; or \g.

#Your MySQL connection id is 6

#Server version: 5.7.19 MySQL Community Server (GPL)

#

mysql>

It all works! Isn’t that cool? Every application, every process could have its own temporary login and password, which would be useless to steal, and with proper logging we always could track down what process run what query.

Conclusion

The vault has other pluggable backends: another secret providers, authentication backends (e.g. AWS or github) and ones for auditing. We barely scratched the surface here.

I’m still impressed how much one can learn from just playing with the tool: algorithms, concepts, smart separation of pluggable components, some other thoughts. Even if I had no intent to get back to Vault ever again, it was worthwhile to sit with documentation and experimenting with the tool just because of that ideas.