GitOps Kubernetes Rolling Update when ConfigMaps and Secrets Change

The Kubernetes ConfigMap resource is used to mount configuration files into pods. The Kubernetes Secret resource is used to mount secret files into pods. Both of these resources are commonly used when deploying a GitOps Configuration as Code workflow.

ConfigMap and Secret files inside of containers are updated automatically when the underlying ConfigMap or Secret is updated. If an application reads a ConfigMap or Secret value on startup, it may have a stale configuration after a ConfigMap or Secret is updated.

One approach for dealing with this is to add application logic to watch ConfigMap and Secret files for changes, and reconfigure the application on the fly. This can lead to complicated logic since objects using the old configuration will need to be detected and recreated.

Another approach is to trigger a rolling update of the Deployment when it’s dependent ConfigMaps and Secrets are updated. This blog post describes a solution that creates ConfigMaps and Secrets alongside a Deployment, and uses a hash of these resources to automatically trigger a rolling update if they have changed.

Configuration as Code with GitOps

GitOps is a method for storing Configuration as Code in a source control repository, and automatically updating deployed state when the repository changes. Essentially, Git becomes the source of truth for your deployment, which ensures that your currently deployed system is documented in source control and makes disaster recovery easier.

When dealing with Configuration as Code, it is nice to be able to stage the configuration for deployment just as it will appear inside of the container. Our sample will stage configuration files and secret files, and use a deployment script to translate these to Kubernetes ConfigMaps and Secrets.

Stage ConfigMap and Secret Files

Let’s consider an nginx container that defines 2 hosts - a frontend website that is unrestricted, and an admin website that is protected by HTTP basic auth.

We will stage 2 files for a ConfigMap that will hold the website configuration -frontend.conf and admin.conf. We will stage 1 file for a Secret that will hold the HTTP basic auth credentials - htpasswd.

Copy the following contents to config/protected/htpasswd to create a user with the username admin and the password password:

admin:$apr1$puyQaI/a$lphikhByCbacJKD9taAxN1

Stage the Deployment and Service Specification

The Deployment mounts the frontend.conf and admin.conf from the ConfigMap to /etc/nginx/conf.d. It mounts the htpasswd from the Secret to /etc/nginx/protected. It exposes websites as NodePort services with the frontend website on port 31228 and the admin website on port 31229.

In order to trigger a rolling update if any of the configuration items change, we have added an environment variable CONFIG_HASH. This is set to a variable - we will make a deployment script that performs variable substitution to insert the hash later.

Create a file called nginx-config-example.yml to create the Deployment and 2 Services for the websites:

Create the Deployment Script

Hash all of the configuration-related files and store to the CONFIG_HASH variable

Apply the Deployment and Services in nginx-config-example.yml

Kubernetes has helper functions to create ConfigMaps and Secrets from files, however these functions only exist in kubectl create. Since ConfigMaps or Secrets may already exist, we will use the --dry-run option and pipe the resulting configuration to kubectl apply.

In order to generate the CONFIG_HASH variable, we will use the find command to find all configuration files, sort them so that they are in a consistent order, and md5sum the contents.

Create a file called nginx-config-example.sh with the following contents:

Run the Deployment Script

Run the script ./nginx-config-example.sh. You should now see resources when you run the following:

# should print a "nginx-config-example" deployment
kubectl get deployment
# should print two "nginx-config-example-xxxxx" pods
kubectl get pods
# should print two NodePort services
kubectl get services

If the nginx-config-example-xxxxx pod has errors starting, run the following commands to debug:

Run the script ./nginx-config-example.sh again. You should see 2 new containers:

# should print two new "nginx-config-example-xxxxx" pods
kubectl get pods

Refresh the frontend website at http://Kubernetes-Worker-IP:31228. You should see the new message on the frontend website.

All of the sample code in this guide can be found at the GitHub repository boxboat/kube-configmap-secret-update. Hopefully you will find that this is a helpful pattern for handling ConfigMap and Secret updates when using a GitOps Configuration as Code workflow.

BoxBoat Accelerator

Learn how to best introduce Docker into your organization. Leave your name and email, and we'll get right back to you.