Contents

Overview

OpenNMS uses notifications to make users aware of an event. Common notification methods are email and paging, but notification mechanisms also exist for

XMPP (Jabber, an instant messaging protocol),

arbitrary external programs

SNMP traps can be sent, and

arbitrary HTTP GETs/POSTs can be made to a web site.

Custom scripts in a BSF language (groovy, jython, etc) can be executed within the JVM (to avoid fork bombs)

A notification can be sent to users, groups, or roles configured in OpenNMS, as well as to arbitrary email addresses, if needed. A delay can be introduced before sending a notification, and one or more escalations can be added in case a notification isn't acknowledged within a configurable period of time. The notifications themselves contain a text message and oftentimes a subject (depending on the notification method) that is built with text. The text of the message and/or subject can be configured to include details from the triggering event, such as the name of the node, ip address, service, error message, etc.

Configuration files

Notifications are handled through the notification daemon, "notifd." This daemon runs by default, and is managed through three configuration files:

destinationPaths.xml

Configures destination paths which specify who gets notified and any escalations.

notifd-configuration.xml

Configures global properties for the notification daemon such as the details for the processing queue and automatic acknowledgments (mapping "down" events with "up" events that automatically acknowledge a notification, causing no more escalation to be performed).

Configures notification methods, such as email, paging, XMPP, SNMP traps, etc.. Although it is named notification "commands", it can not only execute external commands, but also Java classes that can perform a notification action. The Java notification methods are usually preferred as they have higher performance and,more importantly lower overhead than calling an external program. Most notification methods are implemented this way. A standard interface exists, org.opennms.netmgt.notifd.NotificationStrategy, that can be used to implement custom Java notification methods. Of course, calling command-line programs and shell scripts is also allowed.

notifications.xml

Configures actual notifications.

Operation

Upon startup, notifd builds a list of event UEIs that it should listen for based on the notifications configured in notifications.xml and subscribes with the OpenNMS event daemon, eventd, to receive these events.

When an event is received, a few evaluations are performed:

Are notifications turned on (the "status" attribute on the "notifd-configuration" element in notifd-configuration.xml)? If not, the event is discarded and a notification is not performed.

Does the UEI in the event match a UEI configured in an enabled notification and does the rule in the notification match the event (see Rule Matching below)? If not, the event is discarded and notification is not performed. Note: the special UEI string "MATCH-ANY-UEI" can be used to match all event UEIs (the rule still needs to match, as well).

If the notification has a <varbind> configured with a name and value, it is used for a case sensitive match against the beginning an event parameter of the same name.

If the above evaluations pass, one or more notifications are sent. If the "match-all" attribute on the "notifd-configuration" element is set to "true", every matching notification will be executed by walking its destination path, otherwise only the first matching notification will be executed.

Destination Paths

In OpenNMS, a destination path specifies the "who", "when", and "how" of the notification. It specifies the targets of a notification (that's: its recipient, the notification method, and an optional notification interval), and any escalations (which are simply delayed targets). The destination path is separated from individual events as the same information is often used for multiple notifications, so it minimizes duplication and encourages re-use.

When an event is received that matches the UEI and rule in an enabled notification, OpenNMS "walks" the destination path for that notification (or notifications if there are multiple and "match-all" is set to "true"). We say that the destination path is "walked" because it is often a series of actions performed over time and not necessarily just a single action (although it can be). The destination path continues to be walked until all notifications and escalations have been sent or the notification is acknowledged (automatically or by manual intervention).

Once the destination path is started, the initial delay is waited (default: zero seconds) before sending the first notification to its targets. It then waits for the delay for each escalation (if any) and sends the escalations in sequence (the delay for each escalation is the delay from the previous target/escalation). If a target -or an escalation- has a notification interval, and it relates to a group of users or a role, users within this group/role will receiv notification one-by-one at the specified interval until the notification is acknowledged, or all users in the group/role have been notified.

Elements of a notification

Name

A unique name to identify this notification definition. Used to identify the definition in the web UI and in log messages. This is stored in the "name" attribute of the "notification" element in the XML configuration file.

Event

The UEI that causes this event to fire. This is stored in the "uei" element in the XML configuration file.

Description

A description of the event, but is not terribly visible (it is only visible in the web UI a few pages into the edit wizard and in the XML configuration file). Stored in the "description" element in the XML configuration file.

Rule

A filter that must match for the notification to be sent. Oftentime this is an IP address and/or service match. Stored in the "rule" element in the XML configuration file.

Destination Path

The "destination path" to which this notification will be sent if the event is received and the rule matches. See below for details on destination paths. The name of the destination path is stored in the "destinationPath" attribute in the XML configuration file and must match the name of a configuration destination path in destinationPaths.xml.

Subject

The subject of the notification, in particular, the subject of email messages generated by this notification. Event substitutions in the form of %key% can be used to insert details from the event into the text message. Stored in the "subject" element in the XML configuration file.

Text message

The text message of the notification. Just like the subject, event substitutions in the form of %key% can be used to insert details from the event into the text message. Stored in the "text-message" element in the XML configuration file.

On/off

Whether this notification is enabled or not. Stored in the "status" attribute of the "notification" element in the XML configuration file.

Acknowledgment

notifd continues walking the destination path for a notification until the notification is acknowledged. A notification can be acknowledged by a user through the web interface or automatically via a matching "clearing" event. Once the notification is acknowledged no more users, groups, etc. will be notified for that notification.

Automatic Acknowledgment

Many events that represent an outage of some sort also have a matching "clearing" outage which is sent when the original problem is resolved. An example is a "nodeDown" event and a matching "nodeUp" event. OpenNMS has the idea of an acknowledging event that will auto-acknowledge the original event.

Just like a normal acknowledgment, an automatic acknowledgment will stop the destination path for being walked for the original notification. It will also create a new notification to tell users that the original issue is resolved.

Here is an example configuration for the nodeUp/nodeDown event pair in notifd-configuration.xml:

Note the "match" element. This specifies what data in the clearing event needs to match an original event for the automatic acknowledgment to be applied. Unfortunately, it is not possible to match event parameters here. If you need to do that, Auto-acknowledge and match event parameters might be worth a look.

Rule Matching

The rule in the notification is matched against data in the event if the event contains a valid node ID and either an interface or service. If the event does not contain an interface or if the interface is "0.0.0.0", only the node is matched against the rule. Otherwise, the interface is matched, and if the event contains a service, it is matched as well.

Prosper! Notice that the phoneCall command is assigned to the OnCall Role. Keen users will also notice that user "sortova" is on call all month (grin) and that the group will only be notified during duty hours. If the problem happens after duty hours and the notification has not been acknowledged (or the problem resolved) by the time the duty schedule for the group begins, again, the users of the group will be notified via email.