This guide is quick and simple but a little outdated. For example it says Sensu will not resolve
the pagerduty incident once triggered, that is not longer true (with the code
they are using)

Installing redphone gem

There is a gem called redphone its github page is at https://github.com/portertech/redphone
[2]. It is used to support pagerduty,
pingdom, loggly, and statusPage. All the
API request are done over SSL.

To install this gem run

> sudo gem install redphone

To list your installed gems run this command

> gem list

Before I get too far…

Before I get too far into this I want to use the redphone
gem and create a simple ruby script trigger a pagerduty alert.

Pagerduty Sensu Handler

Looking over this page there are several different types of
handlers, pipe, TCP, UDP, AMQP, and
Sets.

For my near term purposes I think I only really need pipe
and sets.

Pipe handlers execute a script and pass the event in via
STDIN.

Sets are used for grouping handlers. It’s a way to send the same event to several
handlers at the same time. For example
if you want an event to two different Pipe handlers, one which sends a message
to HipChat and one that sends an email, you can use a set handler. I am not going to use a Set handler in this
document, but I thought it worth mentioning.

I found this post https://github.com/sensu/sensu/issues/613
[5] which mentions that if you are using the sensu-handler this is the expected
behavior. The default is to only trigger
once every 30 minutes (after the initial trigger…. The initial trigger delay
does not count).

I ran a little test.
I triggered a pagerduty alert and acknowledged it. Then let my sensu alarm run for 30
minutes. At the 30 minute mark the
pagerduty handler triggered again. All
it did was add a "Triggered" event to the current open triggered
alert. So no new alert was created,
which is exactly the behavior I was looking for.

If I leave the service alert in an acknowledged state and
fix the Sensu check (by creating the file again). Wait for it to resolve then remove the file
again (to trigger the incident again).
At this point I still have an open, but acknowledged, incident.

The new trigger does not open a new alert, but just add
another "triggered" event to the current open triggered alert… Not
quite what I want… I need to think this through and play with it.

Looking at the pagerduty I can see that it has the same
incident key… Maybe if I had a different incident key it would issue a
different alert?

I have a second check called check_second_file.json

Which checks for

> sudo vi /etc/sensu/conf.d/check_second_file.json

Which works the same way as my last check but looks for a
different file.

If I remove the first file and trigger the alert and
acknowledge it. Then I trigger the
second Sensu alert, do I get two alerts in pagerduty?

Yes I do! So, all it
needs is a unique incident Key!

This actually works exactly like I want it to. I want each type of Sensu Check to only have
one open pagerduty alert at a time. If
an alert is open and it triggers again I want it to be absorbed into the last
alert.

But if you don't want that and you want each one to trigger
a new alert you could put a timestamp in the incident key. The following code does exactly that.

43200 seconds is 12 hours.
The handlers for this check will only re-run every 12 hours.

That may help out.
But for me I will probably set this to 3600 and not use the epoch
timestamp.

Different Pagerduty alerts

What if you want to have different Sensu checks and you want
thos checks to trigger different pagerduty service alerts?

Luckily someone thought of that when writing the pagerduty.rb
code. You can designate a pager_team in
your check and use that pager_team's API alert key. This makes it easy to have specific pagerduty
alerts triggered per Sensu Check.

> sudo vi /etc/sensu/conf.d/check_file.json

Edit your check and designate a pager_team (you make up the
team name)

"pagerduty_desc": "This is a test for a
message https://www.google.com",

"playbook" : "Go get notes on how to fix
this at https://www.yahoo.com",

"subscribers": [

"check-from-sensu-master",

"client-1",

"client-2",

"aws-client"

]

}

}

}

The pagerduty_desc will append to the description. The playbook will replace the message sent to
pagerduty. (I am using playbook because I have seen it used by other sensu
handlers, I am not sure if it’s in standard use or not?)

If they are not present in the check the normal description
and message will be sent.