Welcome to the xMatters community!

We created this site for our customers and partners and we encourage you to explore, engage, and learn. To ensure that this community is fun and helpful, professional and respectful participation is expected... and troll-like behavior won't be tolerated.

The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.

Check out our built-in integration

You can now install a built-in Datadog integration right from within xMatters, using the Integration Directory (Developer tab > Integrations). Built-in integrations simplify the integration process. They're pre-configured for your xMatters: you don't need to download and import the communication plan, or follow the configuration steps described below.

Introduction

How it works

Datadog is a cloud monitoring solution that brings metrics from all of your apps, tools, and services into one place. By coupling Datadog with the power of xMatters, this integration:

Quickly identifies and notifies the on-call resource on a variety of devices

Allows for voice, SMS, and push messages to team members

Allows users to Mute and Claim alerts from Datadog Monitors directly from their device

Allows users to view xMatters shift data from within the Datadog interface

The integration uses custom webhooks in Datadog that can be @-mentioned in event or monitor alert messages, or even directly in a comment within the event stream. Each webhook sends the same payload to a different xMatters integration, which process the payload to create events in xMatters, or send comments back to the original Datadog event.

Features and updates

This version of the xMatters On-Demand and Datadog integration uses the following custom webhooks created within Datadog.

Once the event is created within xMatters, status updates and delivery notifications can be sent back to the source event with Datadog via outbound integrations.

Engage with xMatters

The Engage with xMatters feature allows users to provide updates, request help, or start a conference bridge with colleagues from other teams to collaborate on events, event stream posts or snapshots within Datadog.

Who's On Call

The Who's On Call feature displays current xMatters shift information within the Datadog event stream - no need to open a new browser window or login to xMatters.

For example, a DevOps team member wants to place a direct call to someone on the database team to confirm a solution. Instead of looking up the team in a company directory or calling a data center and hoping someone picks up, the DevOps person can use the Who's On Call webhook to see who is the right primary on-call person in the database group to contact.

Check out our built-in integration

You can install a "built-in" version of this integration using the Integration Directory (Developer tab > Integrations). Built-in integrations are pre-configured for your xMatters: you don't need to download and import the communication plan, or follow the directions to configure xMatters as described below.

To continue setting up this packaged integration, use the following steps.

Prerequisites

There are a couple of tasks you can do ahead of time to make the configuration process a little easier.

Download the integration package

To begin, download the communication plan attached to this article; it contains everything needed for this integration.

Note: Don't extract the contents of the .zip file; you'll import the file directly into xMatters.

Create API and application keys

For xMatters to authenticate with the Datadog API, you need to retrieve your API key, and create an application key for xMatters. You need these keys to set up communication between xMatters and Datadog.

To access the keys:

Log into Datadog.

Click Integrations > APIs.

You can find your API Key in the Key column of the API Keys list.

To create a new Application Key for xMatters, type a name for the key (for example, "xMatters") in the New application key field, and click Create Application Key.

You can now copy the application key from the Hash column in the Application Keys list.

Configure xMatters

To begin setting up your integration, configure the xMatters components.

Create an integration user

This integration requires a user who can authenticate REST web service calls when injecting events.

This user needs to be able to work with events, but does not need to update administrative settings. While you can use the default Company Supervisor role to authenticate REST calls, the best method is to create a user specifically for this integration with a dedicated role that includes the permissions and capabilities.

Note: If you're installing this integration into an xMatters trial instance, you don't need to create a new user. Instead, locate the "Integration User" sample user that was automatically configured with the REST Web Service User role when your instance was created and assign them a new password. You can then skip ahead to the next section.

To create an integration user:

Log in to the target xMatters system.

On the Users tab, click Add.

Enter the appropriate information for your new user. Because this user affects how messages appear for recipients and how events are displayed in the reports and Communication Center, you may want to identify the user as specific to Datadog; for example:

First Name: Datadog

Last Name: Integration

User ID: datadog

In the Roles area, add the REST Web Service User, Group Supervisor, People Supervisor and Support User roles.

Make a note of the user credentials and details – you need them when configuring other parts of this integration.

Click Add.

Create users and groups to receive notifications

The integration notifies groups and users when events are created in xMatters.

To map xMatters users to Datadog, set the "Work Email" device for each user in xMattters to the same email as their handle within Datadog.

Engage with xMatters

You can test the Engage with xMatters feature using the Event Stream or a Snapshot Comment.

To engage with xMatters:

Click Events.

In the Leave a status update field for an event, use the following format to initiate the webhook (see the Notes below for more syntax information):

<message_text> @webhook-xmatters_engage <recipient>

Click Post to create the post in Datadog and trigger the event in xMatters.

If enabled, outbound integrations for status updates, delivery notifications and response updates create comments and other updates on the Datadog post.

Notes

Engage with xMatters posts have some important required formatting:

All text preceding the @-mention is used as the message when sending notifications to the recipients.

All text after the @-mention must identify the recipients. For example, in the screenshot above, the "Database" and "Hardware" xMatters groups are targeted for notification.

You can use the same format to initiate the xmatters_conference webhook.

If you want to specify an external conference bridge for an xmatters_conference webhook, add -bridge <bridge_name>, where <bridge_name> is a valid conference bridge already configured in xMatters. For example:

Who's On Call

When using the Who's On Call feature, you can use two different post types.

To test the Who's On Call feature:

Click Events.

In the Leave a status update field for an event, use the following format to initiate the webhook (see the Notes below for more syntax information):

@webhook-xmatters_on_call <group>

Click Post.

xMatters displays the Who's On Call results.

Notes

Any text after the @-mention determines the type of Who's On Call you want to make.

To get a group listing, leave the text after the @-mention empty or add -page x where x is a positive integer. The xMatters integration replies with the page of groups for the value of x or, if empty, the first page of groups:

To get on call shift details for a specific group, enter the name of a valid xMatters group after the @-mention. The xMatters integration replies with all active shifts for that group and the team members who are on call:

Troubleshooting

There are several places you can inspect when troubleshooting why an event is not sent to xMatters.

Inbound to xMatters

First, in Datadog, any errors in connecting to the URL encountered by the webhook will create new events in the Datadog Event Stream. For example:

The failing webhook is highlighted in the payload, indicating that you are not authorized to access the @webhook-xmatters_engage feature.

The next place to look is the Integration Builder Activity Stream. While on the Integration Builder tab, expand the Inbound Integrations section, click the gear icon beside the intended integration service, and then click Activity Stream.

The Activity Stream contains the incoming (and for outbound integrations, the outgoing) request, any logging statements as well as the final event creation messages.

Outbound from xMatters

For the two-way integration from xMatters to Datadog, the primary source of logging is the Integration Builder Activity Stream for that particular integration.

For example, if you are troubleshooting why a ticket was not assigned to a user after they replied with "Assign to me", check the Activity Steam for the Datadog Alert Outbound Response outbound integration:

Extend your integration: using basic authentication

When the communication plan is imported into xMatters, the Integration Builder services are set to use URL authentication. You can choose to use basic authentication instead so the endpoints cannot be triggered unless the calling system authenticates with a user name and password – the integration user, in this case. (As an added bonus, using basic authentication means you don't have to log out and log back in to configure the inbound integrations!)

Enable basic authentication

First you need to enable basic authentication for each inbound integration (except the Create Datadog Event integration).

To set up your inbound integrations for basic authentication:

In the Integration Builder, expand the list of inbound integrations.

Click the name of an integration to view its details.

Under the Select authentication method step, select Basic from the drop-down list.

Click Update Integration.

Scroll down to the bottom of the page, and click Copy URL beside the field (you'll need this later to configure DataDog):

Encode the authentication header value

All webhooks within Datadog need a custom authentication header that identifies the generated integration credentials to access the xMatters inbound integration.