This Is What Happened When I Leaked My AWS Secret Key

Keep your secret keys out of your source code; it could cost you thousands

A couple of months ago I was going through old code and using it to fill out my Github. It was about 9PM at night when I published the source code for spookd.me, a rails app I had thrown together some months ago.

The next morning I awoke to this not-great email:

$4900 a day seemed a bit excessive, considering I usually spend about $2 a day.

Upon investigating, I discovered somebody had spun up about 20 c3.8xlarge EC2 instances in each region. According to my bill, these were costing $3-4 an hour each, and they managed to rack up about $2600 of computation in just one night. Having done a bit of research it seems these machines were likely mining some cryptocurrency. Sneaky.

My Mistakes

1. Duh

My first mistake is obvious. Spookd.me is an application which accepts image uploads, so it needed a private key for access to an S3 bucket. I had stupidly committed the key in some past source -- it wasn't even present in the most recent version of the code, but it still made it to Github. Apparently there are bots that trawl Github for access keys while testing them with the AWS API. Try searching secret_access_key: on github and imagine how things may have looked for the first person to try this.

To reiterate: don't paste private keys into source code. If you accidentally commit an access key, read this AWS blog post, and I'd rm -rf .git if I were you. For each key, you should keep two copies in encrypted archives with different passwords with at least 5 special characters on separate USB's in separate safety deposit boxes on opposite sides of the planet.

2. Using The Master Key

You know when you access your AWS security dashboard and it tells you to get started with IAM user accounts or suffer the consequences? That. Create a new account with a new access key for each application and assign it specific priveleges. Now my applications use private keys that only enable them to do exactly what it is they need to do. Here's the documentation for securing an S3 bucket. If I had done this, my secret key would still have leaked, but my attackers would have only been capable of uploading files to the spookd.me bucket.

3. Not using git-secrets

Although a bit of a pain, this could seriously save you. The git-secrets github repository reads: "Prevents you from committing passwords and other sensitive information to a git repository." And that it does. With a simple brew install git-secrets you gain access to a commandline tool which can be used to add pre-commit hooks to any git repository to prevent accidental publication of secret keys.

The AWS Response

AWS is a wonderful platform with extensive documentation, and it turns out their customer support (at least in cases like this) is equally great. They were watching over me, and detected when my usage skyrocketed. They then got in contact, told me exactly how to fix the problem, and credited me the charges for the unauthorized usage.