What is Sensu Silencing?

As check results are processed by a Sensu server, the server executes event
handlers to send alerts to personnel or otherwise relay event data to external
services. Although event handlers can be directly configured with filters
to improve overall signal-to-noise ratio, there are many scenarios in which
operators receiving notifications from Sensu require an on-demand means to
suppress alerts. Sensu’s built-in silencing provides the means to suppress
execution of event handlers on an ad-hoc basis. By using a dashboard or other
tool to interact with the /silenced API, operators can mute notifications
on-the-fly.

The /silenced API manipulates silencing entries in the Sensu data store. These
entries describe a combination of check name and subscription. When the check
name and/or subscription described in a silencing entry match an event,
the handler will not be executed and an informational message will be logged.
Individual handlers may opt-out of silencing by configuring the
handle_silenced attribute.

NOTE: Silencing described in this reference document is implemented in Sensu
version 0.26 or later and Sensu Enterprise 2.0 or later.

When to use silencing

Silencing is used to prevent handlers from being triggered based on the check
name present in a check result or the subscriptions associated with the client
that published the check result. This can be desirable in many scenarios, giving
operators the ability to quiet incoming alerts while coordinating their response.

How does silencing work?

Silencing entries are created on an ad-hoc basis via the /silenced API
endpoint. When silencing entries are successfully created via the API, they
are assigned an ID in the format $SUBSCRIPTION:$CHECK, where $SUBSCRIPTION
is the name of a Sensu client subscription and $CHECK is the name of a Sensu
check. Silencing entries can be used to silence checks on specific clients by
by taking advantage of per-client subscriptions added in Sensu 0.26, e.g.
client:$CLIENT_NAME.

These silencing entries are persisted to the silenced registry in the Sensu
data store. When the Sensu server processes subsequent check results, it
consults this registry to determine whether or not a matching silencing entry
exists. If one or more matching entries exist in the registry, the event context
for the check result is updated to indicate that the event is silenced and include the
ID of the entries which the check result matched.

When creating a silencing entry, a combination of check and subscription can
be specified, but only one or the other is strictly required.

For example, when a silencing entry is created specifying only a check, its ID
will contain an asterisk (or wildcard) in the $SUBSCRIPTION position. This
indicates that any event with a matching check name will be marked as silenced,
regardless of the originating client’s subscriptions.

Conversely, a silencing entry which specifies only a subscription will have an
ID with an asterisk in the $CHECK position. This indicates that any event where
the originating client’s subscriptions match the subscription specified in the
entry will be marked as silenced, regardless of the check name.

NOTE: Starting with version 0.26, Sensu clients automatically add a
subscription containing their client name prefixed with the string client:.
For example, client i-424242 will automatically add subscription
client:i-424242. Silencing checks at the individual client level requires
clients to run Sensu 0.26+, or be manually configured with a client:
subscription.

Silencing entry specification

Silencing entries are composed as a JSON document containing at least one of the
required subscription or check attributes, and additional optional
attributes as desired. Silencing entries are created, updated, and deleted by
submitting JSON documents to endpoints on the /silenced API via HTTP POST
as shown in the examples below.

Silencing entry attributes

check

description

Name of check which the entry should match on

required

true, unless subscription is provided

type

String

default

null

example

{"check":"haproxy_status"}

example

{"check":"haproxy_status","subscription":"load_balancer"}

subscription

description

Name of subscription which the entry should match on

required

true, unless check is provided

type

String

default

null

example

{"subscription":"client:i-424242"}

example

{"subscription":"client:i-424242","check":"haproxy_status"}

id

description

Read-only attribute generated from the intersection of subscription name and check name.

PRO TIP: Due to performance considerations, silencing entries cannot match
events using regular expressions. Instead, use client subscriptions
to create easily-silenceable groups of clients or checks. For example, create a
site-specific subscription to silence all events when performing maintenance
at that site.

Now, imagine that we’d like to make this entry expire in 3600 seconds:

The optional expire_on_resolve attribute used here indicates that when the
server processes a matching check from the specified client with status OK,
this silencing entry will automatically be removed from the registry.

When used in combination with other attributes (e.g. creator and reason),
this provides Sensu operators with way to acknowledge that they have received an
alert, suppressing additional notifications, and automatically clearing the
silencing entry when the check status returns to normal.

Silence all checks on clients with a specific subscription

Assume a client subscription “appserver” which we wish to silence completely.
Just as with our example of silencing all checks on a specific client, we’ll
create a silencing entry specifying only the applicable subscription:

Silencing can be applied to clients not yet registered with the system by
targeting subscriptions instead of client names.

Silencing entries can be automatically removed once the corresponding check
returns OK

Lower overhead - does not require forking a handler process to access the API

Migrating from stash-based silencing

For most operators, a browser-based dashboard like Uchiwa or the Sensu
Enterprise Dashboard is the primary interface for silencing notifications on an
ad-hoc basis. As of Uchiwa 0.18 and Sensu Enterprise Console 2.0, these
dashboards now use the /silenced API in lieu of the /stashes API.

Even after upgrading both Sensu and the dashboard to take advantage of the /silenced
API, handler plugins will continue to query the /stashes API and honor stashes
under the silence path.

As a result, we recommend the following steps as part of any migration effort:

Sensu Client should be upgraded so that clients will add client:$CLIENT_NAME
to their subscriptions.
NOTE: The client:$CLIENT_NAME subscription is required for native silencing
to work at the individual client level.

Sensu API and Server should be updated prior to upgrading Uchiwa or Sensu
Enterprise Console.

Any existing entries under /stashes/silence should be recreated via the
/silenced API prior to upgrading Uchiwa or Sensu Enterprise Console.

Any custom tooling which uses the /stashes/silence pattern should be updated
to use the new /silenced API.

All entries in /stashes/silence be deleted via the /stashes API before
upgrading Uchiwa 0.18+ or Sensu Enterprise Console 2.0+.

About Sensu

The Sensu monitoring event pipeline empowers businesses to automate their monitoring workflows and gain deep visibility into their multi-cloud infrastructure, from Kubernetes to bare metal. Companies like Sony, Box.com, and Activision rely on Sensu to help deliver value faster, at scale.