Log Insight 3.3: Webhooks

In this post, I would like to drill into the new webhook alert option available in Log Insight 3.3. Read on to learn more!

Defining webhooks

My colleague Alan Castonguay put it well when he said, “Webhooks are user-defined HTTP callbacks. Something happens in an application (e.g. alert fired, change made, commit merged, etc) that some remote service cares about. It doesn’t matter why they care, they just want to be told that some event happened. It could be for auditing, or to start a workflow, or something else.”
Given how generic the webhook definition is, webhooks provide a lot of flexibility, but also incur some administrative overhead — more on this in a minute.

Configuring webhooks in LI

There are two places in LI where webhooks can be configured:

For system notification: under the General page in the Administration section

For user alerts: while creating a new user alert or by editing an existing user alert

In both cases, a space separated list of webhook URLs should be specified. Note that authenticated webhooks are not supported in the LI UI as of version 3.3. You can of course send traffic over HTTPS, but the destination cannot require authentication.

Output of a LI webhook

The output of a LI webhook depends on the type of webhook (i.e. user or system) and type of query (user only). Let’s see an example for each:System

1

2

3

4

5

6

7

8

9

{

"AlertName":" Admin Alert: Worker node has returned to service (Host = 127.0.0.2)",

"messages":[

{

"text":"This notification was generated from Log Insight node (Host = 127.0.0.2, Node Identifier = a31cad22-65c2-4131-8e6c-27790892a1f9).\n\nA worker node has returned to service after having been in maintenance mode.\n\nThe Log Insight master node (Host: <a href='https://10.113.236.182:9443/'>https://10.113.236.182:9443/</a>, Node Identifier: 88fc9956-bf9a-428b-806a-22ff07636273) reports that worker node has finished maintenance and exited maintenance mode. The node will resume receiving configuration changes and serving queries. The node is also now ready to start receiving incoming log messages. If an external load balancer is configured to distribute messages among workers, the administrator should add this node back to the pool of nodes receiving incoming messages.\n\nThis message was generated by your Log Insight installation, visit the <a href='https://www.vmware.com/support/pubs/log-insight-pubs.html'>Documentation Center</a> for more information.",

"timestamp":1458665320514,"fields":[]

}

]

}

User: Message Query

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

{

"AlertType":1,

"AlertName":"Hello World Alert",

"SearchPeriod":300000,

"HitCount":0.0,

"HitOperator":2,

"messages":[

{

"text":"hello world 1",

"timestamp":1451940578545,

"fields":[

{

"name":"Field_1",

"content":"Content 1"

},

{

"name":"Field_2",

"content":"Content 2"

}

]

},

{

"text":"hello world 2",

"timestamp":1451940561008,

"fields":[

{

"name":"Field_1",

"content":"Content 1_2"

},

{

"name":"Field_2",

"content":"Content 2_2"

}

]

}

],

"HasMoreResults":false,

"Url":"https://10.11.12.13/s/8pgzq6",

"EditUrl":"https://10.11.12.13/s/56monr",

"Info":"This is an alert for all the 'Hello World' messages",

"NumHits":2

}

User: Aggregation Query

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

{

"AlertType":2,

"AlertName":"field_1 aggregated alert",

"SearchPeriod":300000,

"HitCount":2.0,

"HitOperator":2,

"messages":[

{

"fields":[

{

"name":"Field_1",

"content":"Content 1"

}

]

}

],

"HasMoreResults":false,

"Url":"https://10.11.12.13/s/r25g3s",

"EditUrl":"https://10.11.12.13/s/n3gsed",

"Info":null,

"NumHits":1

}

Sending a LI webhook to a 3rd party

If you read the release notes, you will see mention of third-party applications like Slack. If you go ahead and enable an incoming webhook in Slack and then leverage the new test alert option for user alerts in LI, you will receive the following error:
Wait, I thought LI webhooks allow me to send notifications to third-party solutions? They do, but since LI sends a webhook in its own proprietary format and third-party solutions expect incoming webhooks to be in their proprietary format we have a problem. The solution is to write shim between LI and the third-party solution which translates LI format to third-party format. I will talk about how to write shims in a future post.

Requesting more LI webhook functionality

You might look at the existing LI webhook capability and wish for some additional functionality (e.g. built-in shims, authentication support, etc). If so, please head over to the Log Insight Community and file or vote for a feature. Note the LI team wanted to get this feature out early so feedback could be gathered and more functionality could be added (welcome to agile development!) so please be sure to participate in the community.

Summary

LI webhooks provide further extensibility for alerts in LI. They are simple to enable, but as of LI 3.3 do not support authentication and require a shim be written in order to translate the outputted format into a third-party understandable format. For suggestions on future enhancements to the webhook functionality, be sure to head over to the Log Insight community.

Related

2 Comments

Thanks Steve. Is this webhook mechanism a potential replacement for the laborious task of vCenter alarm creation / management?
For one client alone, we have 35 independent vCenter platforms. Each alarm in each vCenter must be modified to generate an SNMP trap and if we want to create a custom alarm – we have to do it 35 times.

Thanks for the comment! Well I guess it depends — what are you creating alerts for? If the data is available in either ESXi syslog events or can be collected from the VC database then yes LI can be used for this (with or without webhooks), but if no then you will need the alerts so you have an event you can trigger on (in which case I would recommend some automation for VC like PowerCLI). I hope this helps!