Admin

Note: Starting with v0.12.0, Prometheus Operator requires use of Kubernetes v1.7.x and up.

Cluster Monitoring

This guide is intended to give an introduction to all the parts required to start monitoring a Kubernetes cluster with Prometheus using the Prometheus Operator.

This guide assumes you have a basic understanding of how to use the functionality the Prometheus Operator implements. If you haven't yet, we recommend reading through the getting started guide as well as the alerting guide.

Metric Sources

Monitoring a Kubernetes cluster with Prometheus is a natural choice as Kubernetes components themselves are instrumented with Prometheus metrics, therefore those components simply have to be discovered by Prometheus and most of the cluster is monitored.

Metrics that are rather about cluster state than a single component's metrics is exposed by the add-on component kube-state-metrics.

Additionally, to have an overview of cluster nodes' resources the Prometheus node_exporter is used. The node_exporter allows monitoring a node's resources: CPU, memory and disk utilization and more.

Once you complete this guide you will monitor the following:

cluster state via kube-state-metrics

nodes via the node_exporter

kubelets

apiserver

kube-scheduler

kube-controller-manager

Preparing Kubernetes Components

The manifests used here use the Prometheus Operator, which manages Prometheus servers and their configuration in a cluster. Prometheus discovers targets through Endpoints objects, which means all targets that are running as Pods in the Kubernetes cluster are easily monitored. Many Kubernetes components can be self-hosted today. The kubelet, however, is not. Therefore the Prometheus Operator implements a functionality to synchronize the kubelets into an Endpoints object. To make use of that functionality the --kubelet-service argument must be passed to the Prometheus Operator when running it.

Once started it ensures that all internal IPs of the nodes in the cluster are synchronized into the specified Endpoints object. In this case the object is called kubelet and is located in the kube-system namespace.

By default every Kubernetes cluster has a Service for easy access to the API server. This is the Service called kubernetes in the default namespace. A Service object automatically synchronizes an Endpoints object with the targets it selects. Therefore there is nothing, extra to do for Prometheus to be able to discover the API server.

Aside from the kubelet and the API server the other Kubernetes components all run on top of Kubernetes itself. To discover Kubernetes components that run in a Pod, they simply have to be added to a Service.

Exporters

Unrelated to Kubernetes itself, but still important is to gather various metrics about the actual nodes. Typical metrics are CPU, memory, disk and network utilization, all of these metrics can be gathered using the node_exporter.

And last but not least, kube-state-metrics which collects information about Kubernetes objects themselves as they are accessible from the API. Find more information on what kind of metrics kube-state-metrics exposes in its repository.

Setup Monitoring

Once all the steps in the previous section have been taken there should be Endpoints objects containing the IPs of all of the above mentioned Kubernetes components. Now to setup the actual Prometheus and Alertmanager clusters. This manifest assumes that the Alertmanager cluster will be deployed in the monitoring namespace.

apiVersion:monitoring.coreos.com/v1kind:Prometheusmetadata:name:k8slabels:prometheus:k8sspec:replicas:2version:v1.7.1serviceAccountName:prometheus-k8sserviceMonitorSelector:matchExpressions:-{key:k8s-app,operator:Exists}ruleSelector:matchLabels:role:prometheus-rulefilesprometheus:k8sresources:requests:# 2Gi is default, but won't schedule if you don't have a node with >2Gi# memory. Modify based on your target and time-series count for# production use. This value is mainly meant for demonstration/testing# purposes.memory:400Mialerting:alertmanagers:-namespace:monitoringname:alertmanager-mainport:web

The expression to match for selecting ServiceMonitors here is that they must have a label which has a key called k8s-app. If you look closely at all the Service objects described above they all have a label called k8s-app and their component name this allows to conveniently select them with ServiceMonitors.

Read more in the alerting guide on how to configure the Alertmanager as it will not spin up unless it has a valid configuration mounted through a Secret. Note that the Secret has to be in the same namespace as the Alertmanager resource as well as have the name alertmanager-<name-of-alertmanager-object and the key of the configuration is alertmanager.yaml.

Outlook

Once finished with this guide you have an entire monitoring pipeline for Kubernetes. To now access the web UIs they need to be exposed by the Kubernetes cluster, read through the exposing Prometheus and Alertmanager guide to find out how.

To help get started more quickly with monitoring Kubernetes clusters, kube-prometheus was created. It is a collection of manifests including dashboards and alerting rules that can easily be deployed. It utilizes the Prometheus Operator and all the manifests demonstrated in this guide.