Reliable Insights

A handy feature of the Alertmanager is that almost all notification fields are templatable. This can be used to route emails based on labels.

If you are sending email notifications directly to customers, it'd be useful not to have to create a route and receiver per customer in the Alertmanager. This can be useful whether you are a hosting company providing alerting on the machines you rent out, or the provider of a shared service within a company.

I'm going to presume you already have a working email notification setup, such as was shown previously with Gmail.

We start at Prometheus. We'll need to get the target email address into an alert label. Let's say you have constructed time series looking like:

Due to email_to being in the group_by, we can access it via .GroupLabels. By contrast as alerts with many different Alertname and instance labels could be in this one notification, we have to iterate over them to list t hem all.

This technique works with other receivers too, so you could use this to send to customer's Pagerduty services or Slack channels for example.