Kubernetes: Why won’t that deployment roll?

Kubernetes Deployments provide a declarative way of managing replica sets and pods. A deployment specifies how many pods to run as part of a replica set, where to place pods, how to scale them and how to manage their availability.

Deployments are also used to perform rolling updates to a service. They can support a number of different update strategies such as blue/green and canary deployments.

The examples provided in the Kubernetes documentation explain how to trigger a rolling update by changing a docker image. For example, suppose you edit the Docker image tag by using the kubectl edit deployment/my-app command, changing the image tag from acme/my-app:v0.2.3 to acme/my-app:v0.3.0.

When Kubernetes sees that the image tag has changed, a rolling update is triggered according to the declared strategy. This results in the pods with the 0.2.3 image being taken down and replaced with pods running the 0.3.0 image. This is done in a rolling fashion so that the service remains available during the transition.

If your application is packaged up as Helm chart, the helm upgrade command can be used to trigger a rollout:

helm upgrade my-release my-chart

Under the covers, Helm is just applying any updates in your chart to the deployment and sending them to Kubernetes. You can achieve the same thing yourself by applying updates using kubectl.

Now let’s suppose that you want to trigger a rolling update even if the docker image has not changed. A likely scenario is that you want to apply an update to your application’s configuration.

As an example, the ForgeRock Identity Gateway (IG) Helm chart pulls its configuration from a git repository. If the configuration changes in git, we’d like to roll out a new deployment.

My first attempt at this was to perform a helm upgrade on the release, updating the ConfigMap in the chart with the new git branch for the release. Our Helm chart uses the ConfigMap to set an environment variable to the git branch (or commit) that we want to checkout:

kind: ConfigMap
data:
GIT_CHECKOUT_BRANCH: test

After editing the ConfigMap, and doing a helm upgrade, nothing terribly exciting happened. My deployment did not “roll” as expected with the new git configuration.

As it turns out, Kubernetes needs to see a change in the pod’s spec.template before triggers a new deployment. Changing the image tag is one way to do that, but any change to the template will work. As I discovered, changes to a ConfigMap *do not* trigger deployment updates.

The solution here is to move the git branch variable out of the ConfigMap and into to the pod’s spec.template in the deployment object:

When the IG helm chart is updated (we supply a new value to the template variable above), the template’s spec changes, and Kubernetes will roll out the new deployment.

Here is what it looks like when we put it all together (you can check out the full IG chart here).

# install IG using helm. The git branch defaults to "master"
$ helm install openig
NAME: hissing-yak
$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
openig 1 1 1 1 4m
# Take a look at our deployment history. So far, there is only one revision:
$ kubectl rollout history deployment
deployments "openig"
REVISION CHANGE-CAUSE
1
# Let's deploy a different configuration. We can use a commit hash here instead
# of a branch name:
$ helm upgrade --set global.git.branch=d0562faec4b1621ad53b852aa5ee61f1072575cc
hissing-yak openig
# Have a look at our deployment. We now see a new revision
$ kubectl rollout history deploy
deployments "openig"
REVISION CHANGE-CAUSE
1
2
# Look at the deployment history. You should see an event where the original
# replicaset is scaled down and a new set is scaled up:
kubectl describe deployment
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
---> Time=19M ago. This is the original RS
Normal ScalingReplicaSet 19m deployment-controller Scaled up replica set openig-6f6575cdfd to 1
------> Time=3m ago. The new RS being brought up
Normal ScalingReplicaSet 3m deployment-controller Scaled up replica set openig-7b5788659c to 1
-----> Time=3m ago. The old RS being scaled down
Normal ScalingReplicaSet 3m deployment-controller Scaled down replica set openig-6f6575cdfd to 0

One of the neat things we can do with deployments is roll back to a previous revision. For example, if we have an error in our configuration, and want to restore the previous release:

$ kubectl rollout undo deployment/openig
deployment "openig" rolled back
$ kubectl describe deployment:
Normal DeploymentRollback 1m deployment-controller Rolled back deployment "openig" to revision 1
# We can see the old pod is being terminated and the new one has started:
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
openig-6f6575cdfd-tvmgj 1/1 Running 0 28s
openig-7b5788659c-h7kcb 0/1 Terminating 0 8m