Tag: Slack

In this post we are going to see how to monitor Docker containers resource usage statistics and send alarm notifications to a Slack channel. The Docker Engine allows you to see these statistics running the docker stats command. It returns a live data stream for running containers.

1

2

3

4

5

$docker stats

CONTAINER CPU%MEM USAGE/LIMIT MEM%NETI/OBLOCKI/O

1285939c1fd30.07%796KiB/64MiB1.21%788B/648B3.568MB/512KB

9c76f7834ae20.07%2.746MiB/64MiB4.29%1.266KB/648B12.4MB/0B

d1ea048f04e40.03%4.583MiB/64MiB6.30%2.854KB/648B27.7MB/0B

Here you can find the official Docker documentation of the command: Docker stats If you are wondering what Slack is, let me just say that it is an instant messaging and collaboration system based on channels.
You can read more here: Slack

We are going to monitor the container’s resources using a Python script. There are a lot of container management systems but I found some of them too complicated or not very useful. If you want something light, easy and open-source I suggest you Portainer.io (I am going to write a post about it).

We are going to use the docker-py Python client library to connect to the Docker Remote API.
Here you can find the library Github repository: docker-py.
If you do not want to use Python, here you can find a list of client libraries for other programming languages: Docker Remote API client libraries

Install the library with pip:

1

pip install docker

Connect to the Docker Deamon and to the Docker Remote API (specify the Docker server address):

1

2

3

import docker

client=docker.APIClient(base_url='unix://var/run/docker.sock')

client_env=docker.from_env()

For each running containers we can now stream the resource statistics. To list all the running containers use the .containers.list() method.

1

running_containers=client_env.containers.list()

To stream the statistics for a given container use the client.stats method. It takes the container name as argument and returns a generator (Wiki Python Generator).

1

2

3

forxinclient.stats(container.name,decode=True):

# print the streaming statistics

print(x)

Here you can find the official documentation of Low-level API: docker-py low-level API. The stats method returns a JSON with the following format (note the CPU and memory usage information):

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

{

"networks":{

"eth0":{

"rx_bytes":1296,

"tx_packets":8,

"rx_errors":0,

"rx_packets":16,

"tx_errors":0,

"tx_dropped":0,

"tx_bytes":648,

"rx_dropped":0

}

},

"read":"2017-01-09T20:20:24.908807372Z",

"precpu_stats":{

"throttling_data":{

"throttled_time":0,

"throttled_periods":0,

"periods":0

},

"cpu_usage":{

"percpu_usage":None,

"total_usage":0,

"usage_in_usermode":0,

"usage_in_kernelmode":0

},

"system_cpu_usage":0

},

"memory_stats":{

"limit":8354066432,

"usage":13770752,

"failcnt":0,

"max_usage":13963264

},

"cpu_stats":{

"cpu_usage":{

"percpu_usage":[

122260753,

33873591

],

"total_usage":156134344,

"usage_in_usermode":0,

"usage_in_kernelmode":50000000

},

"system_cpu_usage":1314890000000

}

}

We are going now to analyze the resource usage statics and eventually notify an alarm message to Slack (if the usage of some resources exceed our thresholds).
I am not going to post here how to extract the resource usage from the JSON object, but you can find the full code in this Github repository: mz1991/docker-stats-slack.

To send the notification to Slack we will use the Slack Webhook integration. Webhooks are a simple way to post messages from external sources into Slack. They make use of normal HTTP requests with a JSON payload that includes the message text and some options.
You can read more here: Slack Incoming Webhooks. I assume you configured the Slack Incoming Webhook integration for your Slack team and you have the Webhook URL. To configure the Incoming Webhook integration for your Slack team, you can use the following URL: https://[your_slack_team].slack.com/apps/A0F7XDUAZ-incoming-webhooks

With the Webhook integration we do not need any Slack library, to post a message to the Slack channel we just need to post (HTTP post) a message to the Webhook URL endpoint.

To post the message we are going to use the Request class (built-in in the Python urllib3 module).

1

2

3

4

5

6

7

8

9

10

import json

from urllib.request import urlopen,Request,HTTPError,URLError

req=Request(HOOK_URL,json.dumps(slack_message).encode("utf-8"))

try:

response=urlopen(req)

response.read()

except HTTPError ase:

print(repr(e))

except URLError ase:

print(repr(e))

The posted message will look like this:

We saw how to stream the containers statistics and how to post an alarm message to a Slack channel.
I build a Python script that uses a set of environment variables for the Slack channel configuration and the resource usage thresholds.

These are the environment variables needed:

SLACK_WEBHOOK_URL: the webhook url for your Slack team

SLACK_CHANNEL: the channel id (where the message will be posted)

SLACK_USERNAME: the username for the incoming messages

SLACK_EMOJI: the emoji for the incoming messages

MEMORY_PERCENTAGE: maximum percentage of RAM memory used for each container. When the percentage of used memory will exceed this threshold, an alarm will be posted to the Slack channel

CPU_PERCENTAGE: maximum percentage CPU usage for each container. When the CPU percentage usage will exceed this threshold, an alarm will be posted to the Slack channel

SLEEP_TIME (seconds) : interval between each message posted to Slack. Number of seconds between messages, unique for container.

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the cookie policy. If you want to know more or withdraw your consent to all or some of the cookies, please refer to the coockie policy. Got it!Reject.