Incident report: Platform as a Service for government

This post outlines a recent production issue on the Platform as a Service for government (PaaS) and how it was resolved.

What happened

At 3pm on Friday 16 September 2016, the ssl private key was published in a location accessible to all PaaS team members. It was also made available to a small number of people with ‘read access’ to our terraform repository on our private github enterprise installation.

Only users with access to the team’s encrypted credential store should have access to the ssl private key.

People who have ‘read access’ to the aws-account-wide-terraform repository on github.gds, or those with github.gds admin access, could have got access to the SSL private key.

No users were affected

Although no users were affected, users of live applications hosted on the PaaS could have been at a slightly greater risk of being directed to a ‘fake’ app, although this didn’t happen.

An attacker would have had to reconfigure the GOV.UK infrastructure to send traffic to the spoofed site to affect end users.

Potentially the certificate could have been used to spoof other applications running on the PaaS, but Trade Tariff is the only ‘live’ application.

How we responded

We removed the certificate from aws-account-wide-terraform, and rotated the ssl private key.

Why it occurred

The terraform provider for the api gateway did not hide (sha1) the contents of the private key when storing in the state file.

State files are only checked by the reviewer of a pull-request on aws-account-wide terraform.

What we’re doing to prevent this from happening again

For now, we’ll check the state file contents before committing it to github.

In the future we’ll stop storing the state file in the github repository.

When we kick off stories we’ll flag up the need to take extra care when working with terraform and sensitive data, and when reviewing state files.

We’ll move our support manual, which includes our incident process, to our team manual which is publicly accessible.

We’ll speed up and document our process for rotating and revoking SSL certificates.

Detailed timeline for developers

16 September 10:10am
State file containing the private key commited into aws-account-wide-terraform

12 October 3pm
Code pair notice this while working on adding other SSL private key to aws-account-wide terraform

12 October 4pm
Discussion on how and when the fix can be scheduled as the next priority story, and triggering the incident process.
Decision made not to be fixed straight away and can be played as next highest priority story

13 October 9am
Work begins on removing the certificate from aws-account-wide-terraform

Share this page

About Government as a Platform

This blog is about our work with departments and agencies to build common components that can be shared and reused to make it easy for service teams across government to design, assemble and build services.