An important aspect of running ASP.NET Core apps in the cloud is how you secure the secrets your app requires, things like connection strings and API keys. In this post, I show one approach to securing your application when you're running in AWS - using AWS Secrets Manager to access the secrets at runtime.

In my next post, I'll show how to improve this process by filtering out which secrets are loaded by a given application.

Protecting secrets in ASP.NET Core apps

Secrets are configuration values that are in some way sensitive and should not be public. They include things like connection strings, API Keys, and certificates. As a rule of thumb, you should never write these values in appsettings.json files or in any file that is checked-in to a source control repository. Ideally, they should be stored outside your source control working directory.

User Secrets are intended only for development-time storage of secrets, what about in a production environment?

The approach you take will depend a lot on the environment you're running in. Environment variables are often suggested as a good choice, though there are some downsides.

If you're injecting environment variables at runtime (for example, into a docker container), then you need a process to do the injecting. That typically falls on your CI/CD server, which means it (the CI/CD server) needs to have knowledge (or know how to retrieve) all the required secrets for every application.

Alternatively, you could "bake" the the secrets into your docker images themselves, so that when they're run they have all the required secrets available. That's generally a bad idea as anyone can inspect the Docker image and retrieve the secrets directly (assuming they're not previously encrypted in some way).

A better approach, and the approach generally advocated by Microsoft, is to store your secrets in a dedicated "Secrets Vault" such as Azure Key Vault. At runtime, your app requests access to Secrets from the Azure Key Vault service.

One practicality to be aware of - accessing Azure Key Vault requires a "Client Secret" to call the Azure Key Vault API. That leaves you with a chicken-and-egg problem. You store your secrets securely, but you can't access them without an API key, so how do you store that API key securely? If your app is running in Azure, I think Managed Service Identity can solve this problem, but I haven't used it myself so I'm not sure.

AWS Secrets Manager serves an essentially identical role as Azure Key Vault. It securely stores your secrets until you retrieve them at runtime. If your going to be running your ASP.NET Core app in AWS, then AWS Secrets Manager is a great option, as it allows you to finely control the permissions associated with the AWS IAM roles running your apps. It also handles the chicken and egg problem if you're running inside AWS: if the AWS role used to run your app has access to AWS Secrets Manager, you don't have to worry about storing extra API/Access keys; you control access at the AWS role/policy level instead of using API keys.

Adding a secret to AWS Secrets Manager

As with most AWS features, there are several ways to add a secret to AWS Secrets Manager. I'll show how to do so via the CLI tool and the AWS Console.

Note: To generate secrets, you'll need to have the secretsmanager:CreateSecret permission granted for your user/role in IAM

AWS Secrets Manager has a lot of different features, that I'm not going to touch on in this post. You can store database credentials, key-value pairs, plaintext strings, encrypt with custom AWS KMS keys, and rotate database credentials automatically. I'm going to store a simple plain text string (the secret value) and use the name of the secret as the key.

The secret I'm going to store is a connection string, and I want to store it so that the final key added to the ASP.NET Core configuration is ConnectionStrings:MyTestApp i.e. it's the MyTestApp key, in the ConnectionStrings section. If we were storing the secret directly in JSON, it would look something like this:

For legacy reasons, I'm actually going to store the : in the secret name as __, and replace it inside the app when the secret is loaded. This is consistent with the behaviour for environment variables in ASP.NET Core, and I think it's a nice convention to preserve. So the secret name stored in AWS will be ConnectionStrings__MyTestApp.

An important point to note is that you'll need to configure the AWS IAM role used to run your ASP.NET Core application with the necessary permissions to List and Fetch stored secrets, plus to decrypt secrets with KMS. An example IAM Policy snippet might look something like the following (but be sure to check for your own environment - this allows the role to read any secret in Secrets Manager)

This will take you to the "Store a new Secret" wizard. The first step is to choose the type of secret, and set its value. We'll be using the "Other type of secret" and will store the plaintext value. We'll leave the encryption as the default for now.

Click Next, and on the next page enter the name ConnectionStrings__MyTestApp and a description for the secret.

Click Next to view the rotation option. We'll leave rotation disabled. Click Next again and you're presented with a review of your secret, and sample code for accessing it. Don't worry about this - we'll be using the AWS .NET SDK to abstract away all this complexity. Finally, click Store to store your secret in AWS Secrets Manager.

After storing the secret, you can view the details of the secret from the Secrets Manager home page, and manually retrieve the secret value if necessary.

Loading secrets from AWS Secrets Manager with ASP.NET Core

Now we've got a secret stored, it's on to the interesting part - loading it in our ASP.NET Core application. To do this I used a little NuGet package called Kralizek.Extensions.Configuration.AWSSecretsManager. The library is open source on GitHub - it doesn't have many stars, but its actual code foot print is very small, and it does everything I need so I recommend it.

The README for the library includes great instructions for how to add configuration using AWS Secrets Manager to your application. The simplest use case is to add the AWS Secrets to the standard CreateDefaultBuilder call in Program.cs by using ConfigureAppConfiguration():

This adds AWS Secrets loading after all other configuration sources (e.g. appsettings.json, environment variables, command line arguments). It also assumes that AWS credentials are available by default to your application, using the usual AWS SDK mechanisms. If you're running your app inside AWS, then this will likely work out of the box. If not, there are other ways to obtain the necessary credentials, but I won't go into those here, as they circle back to the chicken and egg issue!

This is almost all we need for our application, but there's a couple of tweaks I want to make.

Don't add AWS secrets when developing locally

First, I don't want to use AWS SecretsManager when developing applications locally - .NET User Secrets are a better solution for that problem. It's possible that _my_ connection string for a local development database will be different to someone else's connection string when they're working locally on the app. AWS Secrets Manager configuration would be shared across all users, and would overwrite any local values we'd set. On top of that, it means I don't have to have AWS credentials configured for my local environment to run the application.

Ignoring the AWS Secrets Manager when developing locally is easy to achieve by using Hosting Environments. We can conditionally check if we're in the Development environment and only call AddSecretsManager() if it's appropriate:

Customising the generated key

There's one more piece of configuration to add. When I created the AWS secret, I said that I would store the secret using the __ token to separate configuration sections. The .NET Core configuration system requires you use : to separate sections, so we need to change the secret name when it's loaded. Luckily, Kralizek.Extensions.Configuration.AWSSecretsManager provides an options object to do just that.

Our key generation function isn't complicated, we are just replacing instances of "__" with ":":

Note that you have access to the whole SecretListEntry object here, so you can do more complex transformations if necessary. This object includes all the metadata about the secret (LastChangedDate and Tags for example), but not the value itself.

Add the GenerateKey call to the AwsSecretsManager() call by configuring the SecretsManagerConfigurationProviderOptions object. As the GenerateKey is so simple, I've inlined it here:

And that's it - you now have secrets being stored securely in Secrets Manager, and loaded at runtime based on the AWS IAM role your app is running under! This is great for getting started, but you'll generally want a more sophisticated setup than this provides, especially if you're storing secrets for multiple apps or for multiple environments. In the next post I'll show how you can achieve that with just a few tweaks to this basic configuration.

Summary

In this post I showed how to create a secret in AWS Secrets Manager using the AWS CLI or the web Console. I showed how to configure an ASP.NET Core application to load the secrets at runtime using the Kralizek.Extensions.Configuration.AWSSecretsManager NuGet package. One big advantage to using AWS Secrets Manager if you're already running in AWS is that you don't need to configure additional API keys. That avoids the "chicken and egg" problem you can run into if trying to access secure resources from outside the ecosystem.