The Prometheus Operator introduces third party resources in Kubernetes to declare the desired state of a Prometheus and Alertmanager cluster as well as the Prometheus configuration. The resources it introduces are:

Prometheus

Alertmanager

ServiceMonitor

Important for this guide are the Prometheus and ServiceMonitor resources. Have a look at the alerting guide for more information about the Alertmanager resource or the design doc for an overview of all resources introduced by the Prometheus Operator.

The Prometheus resource declaratively describes the desired state of a Prometheus deployment, while a ServiceMonitor describes the set of targets to be monitored by Prometheus.

The Prometheus resource includes a selection of ServiceMonitors to be used, this field is called the serviceMonitorSelector.

First, deploy three instances of a simple example application, which listens and exposes metrics on port 8080.

The ServiceMonitor has a label selector to select Services and the underlying Endpoints objects. The Service object for the example application selects the Pods by the app label having the example-app value. In addition to that the Service object specifies the port the metrics are exposed on.

Important: If you have RBAC authorization activated you need to create RBAC rules for both Prometheus and Prometheus Operator. We already created a ClusterRole and a ClusterRoleBinding for the Prometheus Operator in the first step. The same has to be done for the Prometheus Pods:

Finally, a Prometheus object defines the serviceMonitorSelector to specify which ServiceMonitors should be included. Above the label team: frontend was specified, so that's what the Prometheus object selects by.

Once this Service is created the Prometheus web UI is available under the node's IP address on port 30900. The targets page in the web UI now shows that the instances of the example application have successfully been discovered.