How To Install Prometheus using Docker on CentOS 7

Introduction

Prometheus is an open source monitoring system and time series database. It addresses many aspects of monitoring such as the generation and collection of metrics, graphing the resulting data on dashboards, and alerting on anomalies. To achieve this, it offers a variety of components that are run separately but used in combination.

Note: This article has been tested on a 1 CPU/512 MB RAM CentOS 7.1 Droplet. The Prometheus configuration should be adjusted for different amounts of available resources as stated in Step 2 of this article.

Step 1 — Installing Prometheus

This section will explain how to install the main Prometheus server using Docker. Make sure to install Docker per the Prerequisites section before continuing. The Prometheus server is the central piece of the Prometheus ecosystem and is responsible for collecting and storing metrics as well as processing expression queries and generating alerts.

Docker container images for all Prometheus components are hosted under the prom organization on Docker Hub. Running the prom/prometheus Docker image without any further options starts the Prometheus server with an example configuration file located at /etc/prometheus/prometheus.yml inside the container. It also uses a Docker data volume mounted at /prometheus inside the container to store collected metrics data. This data volume directory is actually a directory on the host which Docker auto-creates when the container is first started. The data inside it is persisted between restarts of the same container.

There are multiple ways for overriding the default configuration file. For example, a custom configuration file may be passed into the container from the host filesystem as a Docker data volume, or you could choose to build a derived Docker container with your own configuration file baked into the container image. In this tutorial, we will choose to pass in a configuration file from the host system.

There are different patterns for organizing the storage of metrics as well. In this tutorial, we will use the Docker image's default behavior of using a Docker data volume to store the metrics. You may alternatively consider creating a data volume container if that suits your needs better.

This example configuration makes Prometheus scrape metrics from itself (since Prometheus also exposes metrics about itself in a Prometheus-compatible format) as well as from a Node Exporter, which we will set up later. While Prometheus can connect to itself within its container using the localhost host name, it will need to scrape the Node Exporter using your server's external IP since the Node Exporter will run in a separate container with a separate networking namespace.

Start the Prometheus Docker container with the external configuration file:

The -v [...] option mounts the prometheus.yml configuration file from the host filesystem into the location within the container where Prometheus expects it (/etc/prometheus/prometheus.yml).

The -config.file option is set accordingly to the location of the Prometheus configuration file within in the container.

The -storage.local.path option configures the metrics storage location within the container.

Finally, the -storage.local.memory-chunks option adjusts Prometheus's memory usage to the host system's very small amount of RAM (only 512MB) and small number of stored time series in this tutorial (just under 1000). It instructs Prometheus to keep only 10000 sample chunks in memory (roughly 10 chunks per series), instead of the default of 1048576. This is a value you will definitely need to tune when running Prometheus on a machine with more RAM and when storing more time series. Refer to Prometheus's storage documentation for more details around this.

You can list all running Docker containers using the following command:

docker ps

For example, you will see something similar to the following for the Prometheus Docker container:

In this example, the metrics are stored in /var/lib/docker/volumes/821b0abc470a9c758ff35ed5cff69077423a629566082a605a01d8207d57cd6c/_data on the host system. This directory was automatically created by Docker when first starting the Prometheus container. It is mapped into the /prometheus directory in the container. Data in this directory is persisted across restarts of the same container. If you prefer, you may also mount an existing host directory for storing metrics data. See How To Work with Docker Data Volumes on Ubuntu 14.04 for how to achieve this. This article is written for Ubuntu 14.04, but the docker commands and configuration files are the same on CentOS 7.

You should now be able to reach your Prometheus server at http://your_server_ip:9090/. Verify that it is collecting metrics about itself by heading to http://your_server_ip:9090/status and locating the http://localhost:9090/metrics endpoint for the prometheus job in the Targets section. The State column for this target should show the the target's state as HEALTHY. In contrast, the http://localhost:9100/metrics (Node Exporter) endpoint should still show up as UNHEALTHY since the Node Exporter has not yet been started and thus cannot be scraped:

To summarize, you now have Prometheus running as a Docker container using the custom Prometheus configuration file ~/prometheus.yml, which is located on the host filesystem. The metrics storage is located in the /prometheus directory in the container, which is backed on the host system by the path shown by the docker inspect command explained in this section.

Step 2 — Setting up Node Exporter

In this section, we will install the Prometheus Node Exporter. The Node Exporter is a server that exposes Prometheus metrics about the host machine (node) it is running on. This includes metrics about the machine's filesystems, networking devices, processor usage, memory usage, and more.

Note that running the Node Exporter on Docker poses some challenges since its entire purpose is to expose metrics about the host it is running on. If we run it on Docker without further options, Docker's namespacing of resources such as the filesystem and the network devices will cause it to only export metrics about the container's environment, which will differ from the host's environment. Thus it is usually recommended to run the Node Exporter directly on the host system outside of Docker. However, if you have a requirement to manage all your processes using Docker, we will describe a workaround which provides a reasonable approximation for exporting host metrics from within Docker.

The following Docker and Node Exporter flags are used to provide a reasonable approximation for the host metrics:

On Linux, the Node Exporter gathers most of its metrics from the /proc and /sys filesystems. These filesystems are mounted from the host into the container underneath a /host directory, using Docker's -v flag.

Via the Node Exporter's -collector.procfs and -collector.sysfs flags, we instruct the Node Exporter to look for the /proc and /sys filesystems in a non-standard location.

To report host filesystem metrics, we also mount the entire root (/) filesystem into the container (at /rootfs), again using Docker's -v flag.

Use Node Exporter's -collector.filesystem.ignored-mount-points flag to ignore any other filesystems within the container that do not belong to the host system. This option takes a regular expression of mount points to exclude from the reported metrics.

Using the --net=host Docker flag, we place the container into the same network stack as the host, so that reading from files such as /proc/net/dev will yield the same results as on the host (reading from the /proc filesystem mounted in from the host is not sufficient).

Note that some metrics will still differ in comparison to a Node Exporter running directly on the host. Specifically, metrics reported about filesystem usage will have a /rootfs prefix in the value of their mountpoint labels, due to us mounting in the root filesystem under this prefix in the container. There is also no guarantee that the described workarounds will be sufficient for any future Node Exporter features, so run Node Exporter on Docker at your own discretion.

The Prometheus server should now automatically start scraping the Node Exporter. Head to your Prometheus server's status page at http://your_server_ip:9090/status and verify that the http://your_server_ip:9100/metrics target for the node job is now showing a HEALTHY state:

Step 3 — Setting up Grafana

Finally, we will set up Grafana. Grafana is a graphical dashboard builder that supports Prometheus as a backend to query for data to graph.

Grafana stores its dashboard metadata (such as which dashboards exist and what graphs they should show) in a configurable SQL-based database. Grafana supports using a local file-backed SQLite3 database as well as external database servers such as MySQL or PostgreSQL for this.

In this tutorial, we will use a SQLite3 database backed by a Docker data volume. See How and When to Use Sqlite for more information.

Launch Grafana as a Docker container with an administrator password (admin_password) of your choosing:

This will download the Grafana Docker image from the Docker Hub and create a new Docker volume placed at ~/grafana_db on the host system and at /var/lib/grafana in the container filesystem. In the container, Grafana will then automatically create and initialize its SQLite3 database at /var/lib/grafana/grafana.db.

The -e flag allows passing environment variables to the process launched inside the Docker container. Here, we use it to set the GF_SECURITY_ADMIN_PASSWORD environment variable to the desired dashboard administrator password, overriding the default password of admin. Environment variables may also be used to override any other Grafana configuration settings. See Using environment variables for more details.

To verify that Grafana is running correctly, head to http://your_server_ip:3000/. The administrator username is admin and the password is the one you chose when starting the Docker container previously.

Conclusion

Congratulations! You have set up a Prometheus server, a Node Exporter, and Grafana — all using Docker. Even though these are currently all running on the same machine, this is only for demonstration purposes. In production setups, one would usually run the Node Exporter on every monitored machine, multiple Prometheus servers (as needed by the organization), as well as a single Grafana server to graph the data from these servers.

To learn more about using Prometheus in general, refer to its documentation.