Reconfigure a Terraform Backend for Rotated AWS Access Keys

Chris Wahl · Posted on2020-05-192020-05-07

I leverage several different Amazon Web Sevices (AWS) regions for my lab environment resources. Each region has a region-scoped IAM account with programmatic access keys. This limits the attack surface area to a single region should one of the access keys leak. HashiCorp Terraform uses these regional access keys to maintain a declarative state by way of numerous Terraform plans.

I rotate the regional access keys after no greater than 90 days and store the new keys in HashiCorp Vault. During this process, a few Terraform plans produced the error shown below:

Initializing the backend...
Error: error using credentials to get account ID: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
status code: 403, request id: 1234567890

This error indicates that the state file stored in AWS Simple Storage Service (S3) was unable to be retrieved. Without valid credentials, Terraform fails gracefully because the state file cannot be validated before refreshing resource information. In short: the plan could not check the previous state of my cloud resources in order to see what had changed.

For some reason, the credentials were declined by AWS as revealed by the status code “403 Forbidden” coupled with “The security token included in the request is invalid.“

I manually tested the new access keys with awscli without any issues. My next idea was to look at my source code and see if state was being retained. I soon realized that the backend state was, in fact, being stored locally!

This particular Terraform plan was run prior to setting up an S3 backend. For some reason, my local state file persisted with a Terraform backend block. Inside the state file was the old access keys. My next thought was to look for a method to nullify the state values.

Terraform Init with Reconfigure

It turns out that reconfigure is the parameter to clean up my backend configuration. This will “reconfigure the backend, ignoring any savedconfiguration.”

I ran terraform init -reconfigure and noticed the local state file change in git. The serial, access_key, and secret_key values were modified as shown below:

With the access keys cleared from the local state file, Terraform once more looked to my .aws credentials to gather the current (and valid) access keys. Success!

If you find yourself in a similar situation, I hope this post provides assistance.