I should probably document the purple server I hacked together in Perl and mentioned in my last post. In short it allows you to centralise notifications. Send "alerts" to it, and when they are triggered they will be routed from that central location. There is only a primitive notifier included, which sends data to the console, but there are sample stubs for sending by email/pushover, and escalation.

In brief you create alerts by sending a JSON object via HTTP-POST. These objects contain a bunch of fields, but the two most important are:

id

A human-name for the alert. e.g. "disk-space", "heartbeat", or "unread-mail".

raise

When to raise the alert. e.g. "now", "+5m", "1466006086".

When an update is received any existing alert has its values updated, which
makes heartbeat alerts trivial. Send a message with:

{ "id": "heartbeat",
"raise": "+5m",
.. }

The existing alert will be updated each time such a new event is submitted,
which means that the time at which that alert will raise will be pushed back
by five minutes. If you send this every 60 seconds then you'll get informed
of an outage five minutes after your server explodes (because the "+5m"
will have been turned into an absolute time, and that time will eventually
become in the past - triggering a notification).

Alerts are keyed on the source IP which sent the submission and
the id field, meaning you can send the same update from multiple
hosts without causing any problems.

Notifications can be viewed in a reasonably pretty Web UI, so you can
clear raised-alerts, see the pending ones, and suppress further notifications
on something that has been raised. (By default notifications are issued
every sixty seconds, until the alert is cleared. There is support for
only raising an alert once, which is useful for services you might deliver
events via, such as pushover which will repeat themselves.)

Anyway this is a fun project, which is a significantly simplified and
less scalable version of a project which is open-sourced already and
used at Bytemark.