Prometheus metrics

Introduction

Prometheus is an open-source systems monitoring and
alerting toolkit originally built at SoundCloud. Since its inception in 2012,
many companies and organizations have adopted Prometheus, and the project has
a very active developer and user community. It is now a standalone open source
project and maintained independently of any company.

Mongoose OS is a purpose-built secure Operating
System for commercial connected devices. It focuses on stable and secure
functioning of multiple connected devices in production and post-sale stages.
Key features include secure communication (TLS), over-the-air updates (OTA)
and remote device management. These features are usually missing from SDK and
their correct implementation would be a complex and resource consuming task.
Neglecting them may result in compromised device security and negative brand
perception of your products.

Structure

prometheus-metrics is a library component that can be added to the app's
mos.yml file without any configuration needed out of the box, and it pulls
in the http-server module. The library opens a /metrics endpoint which
exposes the operating system and library vitalsigns to Prometheus.

By adding the library to the build manifest in mos.yml, a compiler define
MGOS_HAVE_PROMETHEUS_METRICS is set, which other libraries can use to
create metrics and update them. This is non intrusive because if the
library is not used, no additional code is compiled in Mongoose OS and its
libraries.

Implementation

Base MGOS Metrics

All Mongoose vitals (memory, WiFi/Ethernet, CPU, scheduling) are exposed
using the mgos_ prefix.

Library Specific Metrics

Library owners gate the code that creates, updates and exposes the metrics
by the define MGOS_HAVE_PROMETHEUS_METRICS. Metrics should be defined as
static variables to stay private to the implementation. Then, a callback
function is installed, and prometheus-metrics will loop over all
registered callbacks to allow them to add their metrics to the output.

POSTing to Pushgateway

Prometheus offers an intermediate receiver called a Pushgateway, see their
codebase for details. Some users
may not wish to have their Mongoose IoT device listen on the network for HTTP
connections to the /metrics endpoint, for security reasons. As an
alternative, the library can be configured to close its serving endpoint, and
push its metrics upstream instead. In mos.yml:

mgos_prometheus_metrics_add_handler

Registers a function handler, which will be called each time Prometheus
scrapes our HTTP /metrics endpoint. Libraries and application code can
register any number of handlers, which will be called one after another.
Example:

mgos_prometheus_metrics_push

Perform an HTTP POST request against the Prometheus Pushgateway specified in
the flag prometheus.pushgateway in mos.yml, using 'job' and 'instance'.
The job argument is mandatory, but instance can be passed NULL.