How do I reduce alert flapping / noise?

We often discuss alerting with our clients and a frequent issue or pain point is alert fatigue, or when alerts ‘flap’ (rapidly switching from an ‘ok’ to an ‘alert’ status). Your individual Datadog alerts with groups will have notification rollups on by default, but there is functionality within Datadog that often leads to less noisy, more meaningful alerts.

Re-Evaluate the Alert Threshold Value

The easiest way to reduce flapping when the alert <-> ok or state changes are frequent could be to increase/decrease the threshold condition.

Utilize the ‘At all times’ threshold

This triggers the alert only when all data points for the metric in the timeframe violate the threshold

The most recent addition to Datadog’s alerting capabilities, composite alerts will allow you to combine two or more previously created alerts.For example: if CPU is high AND disk is high on a host, trigger the alert.