What is Terraform?

Terraform is a tool to enable the abstraction of infrastructure configurations into archivable, version-controlled code just like software code. This code then drives the provisioning process such as instantiating servers, databases, network topologies and other resources.

Configuration Management (CM) tools, like Chef and Puppet, typically assume the pre-existence of certain bare-metal nodes and components onto which the server/application provisioning process is executed. Terraform handles the previous stage of setting up the nodes and underlying infrastructure required for these CM tools to run on. While doing so, Terraform records the infrastructure it created in state files so it can manage and update it later.

We can then keep track of past changes to the infrastructure and preview upcoming ones, opening it up to the deployment lifecycle along with staging and testing. The Terraform declarative code essentially becomes our infrastructure documentation, simplifying reuse across teams and onboarding of new team members.

Infrastructure configuration files

In each of our EB-based microservice's Git repositories, we include a sub-directory with the following Terraform configuration files:

The scripts/application/ directory contains the Terraform code necessary to create the EB application for the microservice called ovo-microservice-example. An EB application is a shell inside which multiple application versions and environments are created and launched.

The scripts/application/create_application.tf file is written in a JSON-compatible domain-specific language (DSL) called Hashicorp Configuration Language (HCL). In our case, it is sparse as all our infrastructure resource declarations (such as EC2 instances, ELB load balancers and Auto Scaling groups) are in the environment configuration files.

When launched with scripts/application/terraform apply, Terraform will execute all .tf files in the current directory (but not its sub-directories) so the naming of .tf files is arbitrary. On each run, Terraform will create or update the terraform.tfstate and (optionally) terraform.tfstate.backup state files to keep track of the created resources. This is the core of Terraform's infrastructure state management . These auto-generated files help Terraform manage, version and incrementally update these resources on subsequent updates by minimising fully removing and rebuilding the infrastructure.

For safety, it is best to preview potential changes to the resources by running scripts/application/terraform plan first which outputs a list of what resources Terraform intends to modify, or delete (or leave alone).

After ensuring the Terraform installation folder is on our GoCD agents' $PATH environment variable and that the AWS EB CLI is correctly installed, we add Terraform commands to the ovo-microservice-example micro-service pipeline configuration as follows.

Terraform Caveats

No support for EB application versions

As of version 0.7.2, Terraform does not currently support EB application versions, although support is actively being worked on. This does mean that one cannot specify a healthcheck URL in the EB environment definition such as in scripts/environment/uat/create_application.tf.

This is because without a version, the EB environment is spun up with the EB sample application (which does not support the /ping healthcheck endpoint we use), so the environment can never be healthy if the healthcheck URL is defined in the .tf declaration file.

Currently, we must configure the health check URL manually after the creation of the environment - so only once.

Another temporary solution would be to execute this script to upload and deploy the application using the local-exec provisioner in the .tf file.

Storing state files in Git

The additional pushing to git by push_tf_scripts.sh executed after each Terraform command in the pipeline can sometimes cause of git merge conflicts, especially if multiple developers are simultaneously working on the same repo. To get unstuck, one option is to make the changes to the EB environment manually via the AWS EB web UI or command line, then run terraform refresh (to update local state file against the actual running EB instance), terraform apply (to update the local state files) followed by git push on the state files.

To avoid such issues, we are switching to storing the .tfstate files remotely to an encrypted and versioned AWS S3 bucket using the suggested [terraform remote](Configure remote state storage) command:

S3 versioning will facilitate rollbacks in case of deployment errors and encryption will help keep any environment variables and secrets stored in the state files private and outside of the repo (at least if we use variables - see later).

Since Terraform does not provide locking, if terraform apply is run at the same time then a S3-stored state file might be overwritten by another and the changes lost. Terragrunt was developed to address this although we have not found this to be an issue.

We are in the process of switching to Apache Zookeeper and/or Hashicorp Vault to store configuration settings or securely store secrets to avoid storing them in the tf or .tfstate files completely, or passing them as variables in the build process.

Importing existing infrastructure

It can be laborious to declare all the settings necessary to configure an EB environment in the HCL syntax. One option is to first create the EB environment manually via the AWS web UI. Then using the EB CLIconfig command, execute:

This YAML file can then be used to manually create the Terraform .tf file in HCL. Alternatively, as of the recent version 0.7.0, Terraform now supports the import command to import the .tfstate file for an existing EB instance:

e-xxxx is the instance ID (which is displayed in the AWS EB UI or via command eb status).

Unfortunately, the import command does not yet support the creation of .tf creation files, so you'll need to reconstruct it from the .tfstate which can be tricky and error-prone.

Updating the EB instance outside of Terraform

Once your CI pipeline integrates Terraform, changes made manually via the AWS UI or AWS CLI to the EB environment will be overwritten by the state stored in the repo. It is essential to consider this aspect to avoid nasty surprises during releases.

One approach is to only make changes to EB instances via Terraform .tf file updates and to disallow changes via the UI or CLI (at least if one expects them to remain).

An intermediate approach is to modify the environment via the UI then store the changes into the repo by running the refresh command: terraform refresh which reconciles the current live EB environment with the stored state files to yield an up-to-date understanding of the current infrastructure.

If multiple changes have been made and you would like to just extract the HCL parameters to add to your .tf file, then via the UI, first modify your DEV EB environment or a clone of UAT. Then run terraform refresh and do a diff between the old and new state files to highlight the changes. Once these changes are ported to your .tf file(s), push them to your repo and execute your CI pipeline to update your environment(s).

On a side note, it appears AWS does not support changing the tags on a running EB environment. This also might be the case for certain security group and VPC changes. The environment needs to be re-created from scratch to surface these changes. To maintain uptime, one option is to clone the current EB environment, swap the CNAMEs of the live environment with its clone, destroy and recreate the originally live environment then swap back the CNAMEs.

Elastic Beanstalk config backups

In addition to having backups of the EB environment in your repo as Terraform config files, you can supplement this with saved configurations directly available within the AWS UI and stored in S3. This will allow you to quickly spin up a previous or current version of your environments, such as if Terraform is causing issues or in a case of emergency by support staff without requiring Terraform console access or knowledge thereof.

The following bash script called eb-config-save.sh (stored on the build server) will store the given EB environment both to local YAML files and remotely into up to 4 recycled slots in the AWS EB UI:

We would add the eb-config-save.sh ovo-microservice-example-uat pipeline task after a successful UAT environment deploy. This is followed by the execution a custom bash file similar to the earlier push_tf_scripts.sh to push any changes in say myconfigs/ (and more specifically themyconfigs/.ebelasticbeanstalk/saved_config/ directory) to the Git repo (requires modifying the pipeline to ignore triggers initiated by Git commits for these file):

Future Improvements

Instantiate other resource types

In addition to using Terraform to provision our EB micro-service instances, we are looking at using it to provision supporting infrastructure such as databases, security groups, VPCs, S3 buckets, Lambdas, roles and more.

This would help reduce the large amount of uncontrolled, unfamiliar and manually deployed state in our infrastructure. Since these systems are seldom configured or modified, people tend to be reticent to tinker with them and even less to test them appropriately.

More modular design

Breaking up the existing design

Instead of duplicating the .tf files for each environment within an application (eg. Prod, UAT and Dev) and maintaining them separately, we will switch to modules so that a single parameterised configuration module defines all environments. Only a few parameters would be necessary to define an environment such as the environment name, cname, instance type and numbers.

If we extracted a common template for an OVO EB micro-service into a separate module, the new file structure defined in directory scripts/environment/ovo-microservice-base/ would look like this:

Note that variables can also be injected from the server environment, via a .tfvars file or via the command line with the -var option.

Using remote modules

Another advantage is that modules can be stored remotely (like in a Git repo), so that updates made to a shared module can automatically affect all environments based on said module.

For example, security changes to a common micro-service template module from Devops would propagate to all micro-services on the next release of each service. Each service would be in compliance without any explicit code changes from each developer teams maintaining their own services.

To call a remote module hosted on Github, we simply need to modify the source field in scripts/environment/prd/main.tf to source = "github.com/ovotech/ovo-microservice".

Terraform alternative: CloudFormation

AWS CloudFormation is Amazon's tool to automatically provision almost every service and resource offered on AWS. Elastic Beanstalk even uses CloudFormation under-the-hood to launch its resources.

Like Terraform, its infrastructure-as-code configuration files are defined in a somewhat more verbose JSON syntax. The Ruby DSL called cfn-dsl, the Python library Troposphere and, the AWS CloudFormation Designer drag-and-drop web interface are available to simplify the declaration process.

The JSON infrastructure definitions can also be pushed to Git to track changes, reuse, and easily revert to known good configurations. These definition files, referred to as templates, are then uploaded to CloudFormation which then takes care of the creation, updating, and deleting of AWS resources described in a stack (a set of JSON templates).

Like Terraform, CloudFormation can make incremental changes to infrastructure and preview those changes before proceeding to determine if they are in line with expectations using ChangeSets (similar to the terraform plan command). Unlike Terraform, it does already support EB application versions via SourceBundles and creating usable templates from existing AWS resources with CloudFormer.

However, CloudFormation does not explicitly support states and only supports the AWS ecosystem compared to Terraform which supports many platforms such as Microsoft Azure, Google Cloud, Heroku etc.

Conclusion

Our experience systematising our EB provisioning process with Terraform has shown great promise, although not without some growing pains. We intend to expand this ability to obtain visibility on infrastructure changes, reproducibility, testability and reusability over to all our constantly-evolving and disparate infrastructure.