What is a Sensu client?

Sensu clients are monitoring agents, which are installed and run on every
system (e.g. server, container, etc) that needs to be monitored. The client is
responsible for registering the system with Sensu, sending client keepalive
messages (the Sensu heartbeat mechanism), and executing monitoring checks. Each
client is a member of one or more subscriptions – a list of roles
and/or responsibilities assigned to the system (e.g. a webserver, database,
etc). Sensu clients will “subscribe” to (or watch for) check requests
published by the Sensu server (via the Sensu Transport), execute the
corresponding requests locally, and publish the results of the check back to the
transport (to be processed by a Sensu server).

Client keepalives

What is a client keepalive?

Sensu Client keepalives are the heartbeat mechanism used by Sensu to ensure
that all registered Sensu clients are still operational and able to reach the
Sensu Transport. Sensu clients publish keepalive messages containing client
configuration data to the Sensu transport every 20 seconds. If a Sensu client
fails to send keepalive messages over a period of 120 seconds (by default), the
Sensu server (or Sensu Enterprise) will create a keepalive event.
Keepalives can be used to identify unhealthy systems and network partitions
(among other things), and keepalive events can trigger email notifications and
other useful actions.

Client registration & the client registry

In practice, client registrations happens when a Sensu server processes a client
keepalive message for a client that is not already registered in the Sensu
client registry (based on the configured client name or source attribute).
This client registry is stored in the Sensu data store, and is accessible
via the Sensu Clients API.

All Sensu client data provided in client keepalive messages gets stored in the
client registry, which data is used to add context to Sensu Events and
to detect Sensu clients in an unhealthy state.

WARNING: registration events are not stored in the event registry (in the Sensu
data store), so they are not accessible via the Sensu API; however, all
registration events are logged in the Sensu server log.

Proxy clients

Sensu proxy clients (formerly known as “Just-in-time” or “JIT” clients) are
dynamically created clients, added to the client registry if a client does not
already exist for a check result.

Proxy client registration differs from keepalive-based registration
because the registration event happens while processing a check result (not a
keepalive message). If a check result includes a source attribute, a proxy
client will be created for the source, and the check result will be stored
under the newly created client. Sensu proxy clients allow Sensu to monitor
external resources (e.g. on systems and/or devices where a sensu-client cannot
be installed, such a network switches), using the defined check source to
create a proxy clients for the external resource. Once created, proxy clients
work much in the same way as any other Sensu client; e.g. they are used to store
check execution history and provide context within event data.

NOTE: keepalive monitoring is not supported for proxy clients, as they are
inherently unable to run a sensu-client.

Proxy client example

Proxy clients are created when a check result includes a source attribute, as
follows:

How are keepalive events created?

Sensu servers (including Sensu Enterprise) monitor the Sensu client registry for
stale client data, detecting clients that have failed to send client keepalive
messages for more than 120 seconds (by default). When a Sensu server
detects that a client hasn’t sent a keepalive message within the configured
threshold, the Sensu server (or Sensu Enterprise) will create an event (this
is different from how events are created for monitoring checks; see “How are
Sensu events created?”).

Client keepalive configuration

Sensu client keepalives are published to the Sensu transport every 20 seconds.
Client keepalive behavior can be configured per Sensu client, allowing each
Sensu client to set its own alert thresholds and keepalive event handlers. By
default, client data is considered stale if a keepalive hasn’t be received in
120 seconds (WARNING). By default, keepalive events will be sent to the Sensu
handler named keepalive if defined, or the default handler will be used.

Client signature

Sensu client definitions may specify a unique string identifier as their
signature, which is included in each keepalive message.

Once a client advertises a signature via keepalives, the server will expect that
signature to be present in any keepalives or check results originating from the
client. Any keepalives or check results which do not contain a matching
signature will be dropped with an “invalid client signature” warning in the log.

A client can begin to use a signature if one was not previously configured, but
removing a client signature requires deleting the client.

Providing a unique client signature prevents other clients from accidentally or
maliciously submitting keepalives or check results using the same client name.

Client subscriptions

What is a client subscription?

Each Sensu client has a defined set of subscriptions, a list of roles and/or
responsibilities assigned to the system (e.g. a webserver, database, etc). These
subscriptions determine which monitoring checks are executed by the client.
Client subscriptions allow Sensu to request check executions on a group of
systems at a time, instead of a traditional 1:1 mapping of configured hosts to
monitoring checks. Sensu checks target Sensu client subscriptions, using the
check definition attribute subscribers.

Round-robin client subscriptions

Round-robin client subscriptions allow checks to be executed on a single client
within a subscription, in a round-robin fashion. To create a round-robin client
subscription, prepend the subscription name with roundrobin:, e.g.
roundrobin:elasticsearch. Any check that targets the
roundrobin:elasticsearch subscription will have its check requests sent to
clients in a round-robin fashion, meaning only one member (client) in the
subscription will execute a roundrobin check each time it is published.

The following is a Sensu client definition that includes a round-robin
subscription.

Client subscription configuration

In addition to the subscriptions defined in the client configuration, Sensu
clients also subscribe automatically to a subscription matching their client
name. For example, a client named i-424242 will subscribe to check requests
for the subscription client:i-424242. This makes it possible to generate
ad-hoc check requests targeting specific clients via the /request API.

HTTP Socket

The HTTP socket, just like the TCP and UDP sockets, accepts check results, but it
requires a well-formed HTTP request and exposes other functionality that is not
possible with the raw TCP/UDP sockets. In exchange for a bit more complexity, the
HTTP socket interface has the advantage of being more expressive than a raw TCP/UDP
socket, both in the requests that it accepts and how it responds, and so exposes
more functionality. The following endpoints are available for the HTTP socket:

GET /info

This endpoint returns 200 OK with some basic Sensu information like the version
and transport metrics.

POST /results
This endpoint expects an application/json body with a check result

GET /settings

This endpoint responds with 200 OK and the sensu configuration. Due to the possible
sensitive nature of the client settings (e.g passwords might be in the config) this
endpoint is protected using HTTP basic authentication and by default the information
returned is redacted (e.g common config keys like ‘password’ have their values replaced
by ‘REDACTED’). See the ‘redact’ configuration option if you need control over what is
redacted. This endpoint accepts the optional query string ?redacted=false to disable
redaction.

Creating a “dead man’s switch”

The Sensu client socket(s) in combination with check TTLs can be used to create
what’s commonly referred to as “dead man’s switches”. Outside of the software
industry, a dead man’s switch is a switch that is automatically triggered if a
human operator becomes incapacitated (source: Wikipedia). Sensu is more
interested in detecting silent failures than incapacited human operators. By
using Check TTLs, Sensu is able to set an expectation that a Sensu client will
continue to publish results for a check at a regular interval. If a Sensu client
fails to publish a check result and the check TTL expires, Sensu will create an
event to indicate the silent failure. For more information on check TTLs, please
refer to the check attributes reference documentation.

A great use case for the Sensu client socket is to create a dead man’s switch
for backup scripts, to ensure they continue to run successfully at regular
intervals. If an external source sends a Sensu check result with a check TTL to
the Sensu client socket, Sensu will expect another check result from the same
external source before the TTL expires.

The following is an example of external check result input via the Sensu client
TCP socket, using a check TTL to create a dead man’s switch for MySQL backups.
The example uses a check TTL of 25200 seconds (or 7 hours). A MySQL backup
script using the following code would be expected to continue to send a check
result at least once every 7 hours or Sensu will create an event to
indicate the silent failure.

Standalone check execution scheduler

Because the Sensu client shares the same check scheduling algorithm as the
Sensu server, it is not only possible to have consistency between subscription
checks and standalone checks — it’s also possible to maintain consistency between standalone
checks across an entire infrastructure (assuming that system clocks are
synchronized via NTP).

Client configuration

Example client definition

The following is an example Sensu client definition, a JSON configuration file
located at /etc/sensu/conf.d/client.json. This client definition provides
Sensu with information about the system on which it resides. This is a
production system, running a web server and a MySQL database. The client ‘name’
attribute is required in the definition, and must be unique.

An array of client subscriptions, a list of roles and/or responsibilities
assigned to the system (e.g. webserver). These subscriptions determine which
monitoring checks are executed by the client, as check requests are sent to
subscriptions. The subscriptions array items must be strings.

required

true

type

Array

example

"subscriptions": ["production", "webserver"]

safe_mode

description

If safe mode is enabled for the client. Safe mode requires local check
definitions in order to accept a check request and execute the check.

EXAMPLE

ATTRIBUTES

warning

description

The warning threshold (in seconds) where a Sensu client is determined to be
unhealthy, not having sent a keepalive in so many seconds.

WARNING: keepalive messages are sent at an interval of 20 seconds. Setting
a warning threshold of 20 seconds or fewer will result in false-positive
events. Also note that due to the potential for NTP synchronization issues
and/or network latency or packet loss interfering with regular delivery of
client keepalive messages, we recommend a minimum warning threshold of 40
seconds.

required

false

type

Integer

default

120

example

"warning": 60

critical

description

The critical threshold (in seconds) where a Sensu client is determined to be
unhealthy, not having sent a keepalive in so many seconds.

WARNING: keepalive messages are sent at an interval of 20 seconds. Setting
a critical threshold of 20 seconds or fewer will result in false-positive
events. Also note that due to the potential for NTP synchronization issues
and/or network latency or packet loss interfering with regular delivery of
client keepalive messages, we recommend a minimum critical threshold of 60
seconds.

required

false

type

Integer

default

180

example

"critical": 90

registration attributes

The following attributes are configured within the { "client": {
"registration": {} } }configuration scope.

EXAMPLE

ATTRIBUTES

NOTE: client deregistration attributes are used to generate check
result data for the de-registration event. Client deregistration
attributes are merged with some default check definition attributes by the
Sensu server during client deregistration, so any valid check definition
attributes – including custom check definition attributes
– may be used as deregistration attributes, with the following
exceptions (which are used to ensure the check result is valid): check name,
output, status, and issued timestamp. The following attributes are
provided as recommendations for controlling client deregistration behavior.

handler

description

The deregistration handler that should process the client deregistration
event.

required

false

type

String

default

deregistration

example

"handler": "cmdb_deregistration"

ec2 attributes

The following attributes are configured within the { "client": { "ec2": {} }
}configuration scope.

Custom attributes

Because Sensu configuration is just JSON data, it is possible to define
configuration attributes that are not part of the Sensu client specification.
Custom client definition attributes may be defined to provide context about the
Sensu client and the services that run on its system. Custom client attributes
will be included in client keepalives, and event data and can be used
by Sensu filters (e.g. only alert on events in the “production”
environment), and accessed via check token substitution.

EXAMPLE

The following is an example Sensu client definition that has custom attributes
for the environment it is running in, a mysql attribute containing
information about a local database, and a link to an operational playbook.

NOTE: Because client data is included in alerts created by Sensu, custom
attributes that only exist for the purpose of providing troubleshooting
information for operations teams can be extremely valuable.

Newsletter

Subscribe to the newsletter to get product updates about Sensu, including notifications when new releases are available. No more than one email per week, no less than one email per month. #monitoringlove.