GitLab Prometheus

Notes:

Prometheus and the various exporters listed in this page are bundled in the
Omnibus GitLab package. Check each exporter's documentation for the timeline
they got added. For installations from source you will have to install them
yourself. Over subsequent releases additional GitLab metrics will be captured.

Prometheus services are on by default with GitLab 9.0.

Prometheus and its exporters do not authenticate users, and will be available
to anyone who can access them.

Prometheus is a powerful time-series monitoring service, providing a flexible
platform for monitoring GitLab and other software products.
GitLab provides out of the box monitoring with Prometheus, providing easy
access to high quality time-series monitoring of GitLab services.

Configuring Prometheus

Note:
For installations from source you'll have to install and configure it yourself.

Prometheus and it's exporters are on by default, starting with GitLab 9.0.
Prometheus will run as the gitlab-prometheus user and listen on
http://localhost:9090. Each exporter will be automatically be set up as a
monitoring target for Prometheus, unless individually disabled.

To disable Prometheus and all of its exporters, as well as any added in the future:

Edit /etc/gitlab/gitlab.rb

Add or find and uncomment the following line, making sure it's set to false:

Changing the port Prometheus listens on

Note:
The following change was added in GitLab Omnibus 8.17. Although possible,
it's not recommended to change the default address and port Prometheus listens
on as this might affect or conflict with other services running on the GitLab
server. Proceed at your own risk.

To change the address/port that Prometheus listens on:

Edit /etc/gitlab/gitlab.rb

Add or find and uncomment the following line:

prometheus['listen_address']='localhost:9090'

Replace localhost:9090 with the address/port you want Prometheus to
listen on.

Viewing performance metrics

You can visit http://localhost:9090 for the dashboard that Prometheus offers by default.

Note:
If SSL has been enabled on your GitLab instance, you may not be able to access
Prometheus on the same browser as GitLab due to HSTS. We plan to
provide access via GitLab, but in the interim there are
some workarounds: using a separate browser for Prometheus, resetting HSTS, or
having Nginx proxy it.

The performance data collected by Prometheus can be viewed directly in the
Prometheus console or through a compatible dashboard tool.
The Prometheus interface provides a flexible query language to work
with the collected data where you can visualize their output.
For a more fully featured dashboard, Grafana can be used and has
official support for Prometheus.

Configuring Prometheus to monitor Kubernetes

Introduced in GitLab 9.0.
Pod monitoring introduced in GitLab 9.4.

If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and annotated Pods in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them.

GitLab Prometheus metrics

Introduced as an experimental feature in GitLab 9.3.

GitLab monitors its own internal service metrics, and makes them available at the /-/metrics endpoint. Unlike other exporters, this endpoint requires authentication as it is available on the same URL and port as user traffic.

Prometheus exporters

There are a number of libraries and servers which help in exporting existing
metrics from third-party systems as Prometheus metrics. This is useful for cases
where it is not feasible to instrument a given system with Prometheus metrics
directly (for example, HAProxy or Linux system stats). You can read more in the
Prometheus exporters and integrations upstream documentation.

While you can use any exporter you like with your GitLab installation, the
following ones documented here are bundled in the Omnibus GitLab packages
making it easy to configure and use.

Node exporter

The node exporter allows you to measure various machine resources such as
memory, disk and CPU utilization.