What is a Kubernetes Controller?

Before going into operators, we should have a look at Kubernetes controllers. Controllers are one of the basic building blocks of Kubernetes. There are few dozen built-in controllers, all of which run on the Kubernetes master in a Pod named kube-controller-manager in the kube-system namespace.

One of these built-in controllers is the replica set controller. Its job is to take care of replica sets. The following is a typical example of a replica set resource:

If this looks unfamiliar, it’s because you usually do not create replica sets explicitly. You create a Deployment, and the built-in deployment controller creates a ReplicaSet for you. Deployments do more than just creating replica sets. For example, deployments implement rolling updates. Replica sets have a single purpose: They specify the number of replicas, i.e. the desired number of Pods running.

The replica set controller makes sure that the number of Pods matches the number of replicas defined in the replica set. If a Pod is deleted, the replica set controller starts a new one. If the replica set is modified, the replica set controller adds or deletes Pods to get back to the desired number of replicas.

What is an Operator?

The replica set controller example above illustrates the most important features implemented by controllers:

Watching the primary resource, which is the ReplicaSet in this example.

Watching the secondary resource, which is the Pod in this example.

Interacting with the API server: Whenever a ReplicaSet is created or updated, or whenever a Pod is created or deleted, the replica set controller talks to the API server and creates or deletes Pods until the desired state is restored.

These concepts are implemented in a generic, extensible way in Kubernetes. You can create your own custom resources, and write your own custom controllers watching these resources and interacting with the API server.

An operator is a custom controller with a focus on operating a specific application.

A good example of such an application is a distributed key/value store. Kubernetes provides deployments (or replica sets) to scale up the number of Pods. However, for a particular key/value store additional steps might be necessary when Pods are added, like re-balancing the data. In order to implement this, you might create your own custom resource (similar to ReplicaSet, but with application-specific configuration) and you may implement your own controller (like the replica set controller, but taking care of re-balancing when Pods are added or deleted). This custom controller would be the operator for the key/value store.

Development Tools for Creating Operators

From a high level perspective, operators are applications talking to the Kubernetes API server via HTTPS. They can be written in any programming language, and the can be deployed inside the cluster or outside the cluster. However, it’s most common and most convenient to bundle them into a Docker image and run them inside the cluster.

As operators implement functionality similar to built-in controllers, it is a natural choice to re-use the same libraries that Kubernetes uses internally. This is the origin of the CoreOS operator SDK. It provides tools to generate the necessary boilerplate code, and uses Kubernetes internal libraries as dependencies. As Kubernetes is written in Go, operators are written in Go as well.

For Java developers, Quarkus and the upcoming Kubernetes client extension provides a promising alternative to the Go ecosystem. Java applications built with Quarkus can be compiled into native executables similar to Go binaries. This allows for small Docker images with fast startup times. Moreover, the upcoming Kubernetes client extension implements much of the functionality needed to watch resources and to interact with the API server.

The next Blog post in this series will show how to set up a Quarkus project with the Kubernetes client extension. Part 3 will show how to implement typical controller functionality, like watching for updates of custom resources.

Garbage Collection

As a side note, I would like to introduce owner references and garbage collection. In many cases, Kubernetes’ built-in garbage collection is sufficient to clean up resources created by operators, so operators don’t need to implement explicit clean-up.

In Kubernetes, every resource can be annotated with an owner reference. For example, when the replica set controller creates a Pod, it sets the ReplicaSet as the owner reference for that Pod:

When the replica set is deleted, the controller does not need to do anything. Kubernetes will implicitly delete all resources having the replica set as the owner reference, thus terminating the Pods associated with that replica set.

Operators should adapt this mechanism and use the custom resource as owner reference for all generated resources. That way, whenever the user deletes a custom resource, Kubernetes will take care to remove all associated resources. This works even if the operator is no longer running.

Summary

This blog post introduced operators. An operator is a custom controller with a focus on operating a specific application.

About Instana: As the leading provider of Automatic Application Performance Monitoring (APM) solutions for microservices, Instana has developed the automatic monitoring and AI-based analysis DevOps needs to manage the performance of modern applications. Instana is the only APM solution that automatically discovers, maps and visualizes microservice applications without continuous additional engineering. Customers using Instana achieve operational excellence and deliver better software faster. Visit https://instana.com to learn more.