Hiding Secrets in Terraform

Terraform is a popular tool amongst DevOps folks that allows you to set up your
infrastructure as code. It is powerful and fairly easy to get to grips with.

One of the features that makes it really nice is you can make changes and run a
check to see what physical changes your code changes will make. By default
Terraform will make the least amount of changes it can get away with.

This means if you are setting up 5 different AWS things (some EC2 instances, an
Elasticache instance, and maybe some S3 buckets) and the change you make only
effects one of the EC2 instances, then Terraform isn’t going to touch your
Elasticache stuff.

Local tfstate stores sensitive information

How Terraform goes about the business of comparing the changes in code to the changes in your infrastructure is that it generates what it calls a tfstate.

This is just a set of files that state exactly what is currently running, if what your code would generate changes what is running, it will know that is a thing that needs to be changed.

Unfortunately, in order to set up most of these services you need usernames and passwords to be set - and since you can potentially change these passwords via Terraform then it stands to reason that Terraform is going to need be able to compare your old credentials with possible new ones.

To facilitate this it stores all settings, including usernames, passwords, port numbers and literally everything else in these tfstate files, in plain text.

This wasn’t something I’d have expected as the default behaviour. The documentation does suggest that you use a thing called Remote State (more on that later)

Terraform also puts some state into the terraform.tfstate file by default. This state file is extremely important; it maps various resource metadata to actual resource IDs so that Terraform knows what it is managing. This file must be saved and distributed to anyone who might run Terraform. It is generally recommended to setup remote state when working with Terraform. This will mean that any potential secrets stored in the state file, will not be checked into version control

But that wasn’t always the case, an earlier version of the documentation had the wording;

Terraform also put some state into the terraform.tfstate file by default. This state file is extremely important; it maps various resource metadata to actual resource IDs so that Terraform knows what it is managing. This file must be saved and distributed to anyone who might run Terraform. We recommend simply putting it into version control, since it generally isn’t too large.

No mention of doing anything special to remove secrets from this file.

Lots of people have credentials in Github

I’m not the only person to be surprised by this, in fact there has been an open ticket since 2014 with people stating their surprise and sharing how they’ve worked around the issues.

What follows is basically me distilling a lot of the great information in that ticket into some things you should consider, and adding my own personal opinion to each suggestion.

I’ve already spoiled the ending by sharing that Remote State is the way forward, but there is every chance that the solution you need doesn’t require Remote State, or for some reason precludes you from using it.

Use a dummy password on create and update after

One solution is to only ever use a dummy password and then execute some at a later date which will update the password to the one you actually care about.

Here is a sample script showing how you could do that;

You could even have the code generate a random password to use to update these credentials and then have the random password securely stored somewhere.

Use something like Git Crypt

Several folk in that Github Issue had suggested Git Crypt. I think this is a lovely solution for the more general problem of “I have some files which I would quite like to version control but would really rather not have their contents exposed to the world by mistake”.

From their README;

git-crypt enables transparent encryption and decryption of files in a git repository.
Files which you choose to protect are encrypted when committed, and decrypted when checked out. git-crypt lets you freely share a repository containing a mix of public and private content. git-crypt gracefully degrades, so developers without the secret key can still clone and commit to a repository with encrypted files. This lets you store your secret material (such as keys or passwords) in the same repository as your code, without requiring you to lock down your entire repository.

I’m not sure why you would favour this over Remote State, but I’m certainly going to keep this project in mind for future use.