Note While every Wavefront user can view alert targets, you must have Alert Management permission to create and manage alert targets. If you do not have permission, the UI menu selections, buttons, and links you use to perform management tasks are not visible.

About Alert Targets

Alert targets specify when and how to send notifications in response to alert activity. You implicitly use Wavefront’s built-in alert targets when you specify an email address or a PagerDuty key in the target list of an alert. These simple alert targets:

Cause notifications to be sent whenever the alert is firing, updated, resolved, snoozed or in a maintenance window.

Provide internal templates that compose and format the notification contents. These internal templates are maintained by Wavefront, and may change from release to release.

You can create custom alert targets when you need more flexibility about where you want notifications to be sent, what kind of information you want in them, how you want them to be formatted, and which alert events should trigger them. For example, you could use a custom alert target to:

Expand (or limit) the set of triggering events for notifications.

Configure different contents for notifications triggered by different events.

Associate a short name with a long list of email addresses or a lengthy PagerDuty key.

Note: You always use a custom alert target for sending notifications to a webhook-based messaging platform such as Slack, HipChat, or VictorOps.

Viewing Custom Alert Targets

To view alert targets, select Browse > Alert Targets.

Creating a Custom Alert Target

The process for creating an alert target is similar for the different types of targets. Setting the Type causes a few different fields to be displayed.

Select Browse > Alert Targets.

Click the Create Alert Target button.

Fill in the properties that are common to all alert targets.

Property

Description

Name

Name of the alert target. Pick a name that is simple and that makes it easy to identify the alert target's purpose.

Alert Firing - Trigger when the alert transitions from checking to firing.

Alert Status Updated - Trigger when at least one time series changes category while the alert continues firing. For example, an individual time series could start to fail (satisfy the alert condition during the Alert fires time window) or could recover (stop satisfying the alert condition during the Alert resolves time window).

Alert Resolved - Trigger when the alert resolves.

Alert Affected by Maintenance Window - Trigger when the alert is affected by a maintenance window.

Alert Snoozed - Trigger when the alert is snoozed.

Alert Has No Data - Trigger when the time series associated with the alert have all stopped reporting data.

Alert Has No Data Resolved - Trigger when at least one time series associated with the alert has started reporting data, while all other time series are still reporting no data.

Alert Entered Maintenance From No Data - Trigger when none of the alert's time series are reporting data, and the alert is affected by a maintenance window.

From the Type pull-down menu, select the alert target type:

Type

Description

Webhook

Alert target for sending notifications to messaging platforms such as Slack, VictorOps, or HipChat. This alert target defines the HTTP callback (POST request and URL) that is triggered when an alert changes state.

Email

Alert target for sending notifications to email systems. This alert target specifies the attributes of the email messages to be sent when an alert changes state.

PagerDuty

Alert target for sending notifications to PagerDuty. This alert target specifies the PagerDuty key and a POST body to use when an alert changes state.

Name and value of one or more HTTP headers to pass in the POST request.

Body Template

Template describing the contents of the alert notification. Click Template and select the template that corresponds your messaging platform: Slack, VictorOps, or HipChat. Or, select Generic Webhook to see all of the available content options combined in a single template.

For type Email, set these properties:

Email Property

Description

HTML Format

Specifies whether the email platform should interpret the message body as HTML or plain text. When checked (the default), messages are interpreted as HTML. It is your responsibility to coordinate this setting with the chosen Body Template option.

Email Address List

One or more valid email addresses, separated by commas.

Email Subject

Subject of all emails from this alert target.

Body Template

Template describing the contents of the alert notification. Click Template and select the template that corresponds to your email formatting preference: HTML Email or Plain Text. It is your responsibility to coordinate this option with the HTML Format setting.

For type PagerDuty, set these properties:

PagerDuty Property

Description

PagerDuty Key

API integration key for the PagerDuty application. Follow the setup steps in PagerDuty Integration to obtain the key.

Adding a Custom Alert Target to a Wavefront Alert

In the Alert Target field, start typing. A dropdown list appears that contains all available Wavefront alert targets that can be integrated to your alert.

Select the alert target that you want to add, and click Save.

Editing a Custom Alert Target

You can change a custom alert target at any time.

To edit a alert, click the alert target name in the Alert Targets browser or click the three dots to the left of the alert target and select Edit.

Deleting Custom Alert Targets

You can delete one or more custom alert targets by checking the checkboxes next to the alert targets and clicking the Trash icon at the top of the Alert Targets page. The trash icon is grayed out if you don’t have permission to delete any of the selected alert targets.

To delete one alert target, use the trash icon or click the three dots to the left of the alert target and select Delete.

Finding an Alert Target ID

Each custom alert target has a unique ID that the system generates when you first create the alert target. To find the ID:

Click Browse > Alert Targets.

In the Name column, note the ID of the alert target under the description.

Note Wavefront does not expose response codes from the simpler alert targets (Email and PagerDuty).

The response codes indicate if a webhook call was successful and if the webhook generated a notification. You can query these metrics to determine if any webhooks are generating a problem response code. The metrics have the point tag name = <webhook_name> so you can determine all the response codes for a particular webhook alert target:

ts(~alert.webhooks.*.*, name=<webhook_name>)

If the response code of the webhook is anything other than 2xx, Wavefront creates an event with the name <webhook_id>.<webhook_name>.<response_code>.