Tags

Checking the health of microservices on Kubernetes

This repository contains the guide documentation source. To view the guide in published form, view it on the Open Liberty website.

Learn how to check the health of microservices on Kubernetes by setting up readiness probes to inspect MicroProfile Health Check endpoints.

What you’ll learn

You will learn how to create a health check endpoint for your microservices. Then, you will configure Kubernetes to use this endpoint to keep your microservices running smoothly.

MicroProfile Health allows services to report their health, and it publishes the overall health status to a defined endpoint. A service reports UP if it is available and reports DOWN if it is unavailable. MicroProfile Health reports an individual service status at the endpoint and indicates the overall status as UP if all the services are UP. A service orchestrator can then use the health statuses to make decisions.

Kubernetes provides liveness and readiness probes that are used to check the health of your containers, you will work with readiness probes. These probes can check certain files in your containers, check a TCP socket, or make HTTP requests. MicroProfile Health exposes a health endpoint on your microservices. Kubernetes polls the endpoint as specified by the probes to react appropriately to any change in the microservice’s status. Read the Adding health reports to microservices guide to learn more about MicroProfile Health.

The two microservices you will work with are called name and ping. The name microservice displays a brief greeting and the name of the container that it runs in. The ping microservice pings the Kubernetes Service that encapsulates the pod running the name microservice. The ping microservice demonstrates how communication can be established between pods inside a cluster.

Prerequisites

Before you begin, have the following tools installed:

First, you will need a containerization software for building containers. Kubernetes supports a variety of container types. You will use Docker in this guide. For installation instructions, refer to the official Docker documentation.

Windows | Mac

Use Docker Desktop, where a local Kubernetes environment is pre-installed and enabled. If you do not see the Kubernetes tab then you have an older version of Docker Desktop; upgrade to the latest version.

After following one of the sets of instructions, ensure that Kubernetes (not Swarm) is selected as the orchestrator in Docker Preferences.

Linux

You will use Minikube as a single-node Kubernetes cluster that runs locally in a virtual machine. For Minikube installation instructions see the minikube installation instructions. Make sure to read the Requirements section as different operating systems require different prerequisites to get Minikube running.

Getting started

The fastest way to work through this guide is to clone the Git repository and use the projects that are provided inside:

The start directory contains the starting project that you will build upon.

The finish directory contains the finished project that you will build.

Starting and preparing your cluster for deployment

Start your Kubernetes cluster.

Windows | Mac

Start your Docker Desktop environment.

Linux

Run the following command from a command line:

minikube start

Next, validate that you have a healthy Kubernetes environment by running the following command from the command line.

kubectl get nodes

This command should return a Ready status for the master node.

Windows | Mac

You do not need to do any other step.

Linux

Run the following command to configure the Docker CLI to use Minikube’s Docker daemon. After you run this command, you will be able to interact with Minikube’s Docker daemon and build new images directly to it from your host machine:

eval $(minikube docker-env)

Adding a health check to the ping microservice

Navigate to start directory to begin.

The ping microservice should only be healthy when name is available. To add this check to the /health endpoint, you will create a class implementing the HealthCheck interface.

Create the PingHealth class in ping/src/main/java/io/openliberty/guides/ping/PingHealth.java:

This health check verifies that the name microservice is available at http://name-service:9080/api. The name-service host name is only accessible from inside the cluster, you can’t access it yourself. If it’s available, then it returns an UP status. Similarly, if it’s unavailable then it returns a DOWN status. When the status is DOWN the microservice is considered to be unhealthy.

The health check for the name microservice has already been implemented. It has been setup to become unhealthy for 60 seconds when a specific endpoint is called. This endpoint has been provided for you to observe the results of an unhealthy pod and how Kubernetes reacts.

Configuring readiness probes

You will configure Kubernetes readiness probes. Readiness probes are responsible for determining that your application is ready to accept requests. If it’s not ready, traffic won’t be routed to the container.

The readiness probes are configured for the containers running the name and ping microservices.

The readiness probes are configured to poll the /health endpoint. The readiness probe determines the READY status of the container as seen in the kubectl get pods output. The initialDelaySeconds field defines how long the probe should wait before it starts to poll so the probe does not start making requests before the server has started. The failureThreshold option defines how many times the probe should fail before the state should be changed from ready to not ready. The periodSeconds option defines how often the probe should poll the given endpoint.

Deploying the microservices

To build these microservices, navigate to the start directory and run the following command.

mvn package

When the build succeeds, run the following command to deploy the necessary Kubernetes resources to serve the applications.

kubectl apply -f kubernetes.yaml

Use the following command to view the status of the pods. There will be two name pods and one ping pod, later you’ll observe their behaviour as the name pods become unhealthy.

Wait until the pods are ready. After the pods are ready, you will make requests to your services.

Windows | Mac

The default host name for Docker Desktop is localhost.

Linux

The default host name for minikube is 192.168.99.100. Otherwise it can be found using the minikube ip command.

Navigate to http://[hostname]:31000/api/name and observe a response similar to Hello! I’m container name-deployment-5f868854bf-2rhdq. Replace [hostname] with the IP address or host name of your Kubernetes cluster. The readiness probe ensures the READY state won’t be 1/1 until the container is available to accept requests. Without a readiness probe, you may notice an unsuccessful response from the server. This scenario can occur when the container has started, but the application server hasn’t fully initialized. With the readiness probe, you can be certain the pod will only accept traffic when the microservice has fully started.

Similarly, navigate to http://[hostname]:32000/api/ping/name-service and observe a response with the content pong.

Changing the ready state of the name microservice

An endpoint has been provided under the name microservice to set it to an unhealthy state in the health check. The unhealthy state will cause the readiness probe to fail. Use the curl command to invoke this endpoint by making a POST request to http://[hostname]:31000/api/name/unhealthy — if curl is unavailable then use a tool such as Postman.

You will notice that one of the two name pods is no longer in the ready state. Navigate to http://[hostname]:31000/api/name. Observe that your request will still be successful because you have two replicas and one is still healthy.

Observing the effects on the ping microservice

Wait until the name pod is ready again. Make two POST requests to http://[hostname]:31000/api/name/unhealthy. If you see the same pod name twice, make the request again until you see that the second pod has been made unhealthy. You may see the same pod twice because there’s a delay between a pod becoming unhealthy and the readiness probe noticing it. Therefore, traffic may still be routed to the unhealthy service for approximately 5 seconds. Continue to observe the output of kubectl get pods. You will see both pods are no longer ready. During this process, the readiness probe for the ping microservice will also fail. Observe it’s no longer in the ready state either.

First, both name pods will no longer be ready because the readiness probe failed.

Tearing down the environment

To remove all of the resources created during this guide, run the following command to delete all of the resources that you created.

kubectl delete -f kubernetes.yaml

Windows | Mac

Nothing more needs to be done for Docker Desktop.

Linux

Perform the following steps to return your environment to a clean state.

Point the Docker daemon back to your local machine:

eval $(minikube docker-env -u)

Stop your Minikube cluster:

minikube stop

Delete your cluster:

minikube delete

Great work! You’re done!

You have used MicroProfile Health to create an endpoint that reports on your microservice’s status. Then, you observed how Kubernetes uses the /health endpoint to keep your microservices running smoothly.