Introduction

In this post I’m going to show how to integrate collectd (the system statistics collection daemon) with Elasticsearch (distributed, RESTful search and analytics engine) using Logstash (an open source, server-side data processing pipeline) and visualize in Kibana (Elasticsearch visualization tool) and demonstrate some of its use cases in OpenStack (a set of software tools for building and managing cloud computing platforms for public and private clouds).

There are other usful tools out there to collect system statistics information. Such Nagios, but I personally found collectd a good solution for my research purposes.

Assumptions

Let’s assume you already have a very basic knowledge of collectd and ELK stack. I will go through installing OpenStack in another post and needless to say, that you don’t need OpenStack in order to use collectd and/or ELK stack! I also assume you are using Ubuntu 16.04 server.

Install collectd on a Node

I need following modules

intel_rdt (it won’t work inside virtual machines)

libvirt (if you are using KVM/OpenStack)

lvm (if you are using LVM)

sensors (it won’t work inside virtual machines)

netlink

perl

exec

interface

logfile (for debugging purposes)

memory

cpu

network (in order to send collected data to Elasticsearch)

processes (in order to keep an eye of a specific process, if you want)

top (my written Perl plugin)

Now, we are going to run following commands as root in order to install collectd. To find out which packages need to be installed for your plugins of choice, check collectd’sGitHub page:

Also, we need to make Perl bindings. Inside the collectd main folder, run followings:

cd bindings/perl/
perl Makefile.PL
make
make install

Enable msr kernel module to use intel_rdt correctly:

modprobe msr

In order to sensors plugin work, please run following commands as root:

$ sensors-detect
Some south bridges, CPUs or memory controllers contain embedded sensors. Do you want to scan for them? This is totally safe. (YES/no): YES
Some Super I/O chips contain embedded sensors. We have to write to standard I/O ports to probe them. This is usually safe. Do you want to scan for Super I/O sensors?(YES/no): YES
Some hardware monitoring chips are accessible through the ISA I/O ports. We have to write to arbitrary I/O ports to probe them. This is usually safe though. Yes, you do have ISA I/O ports even if you do not have any ISA slots! Do you want to scan the ISA I/O ports? (YES/no): YES
Lastly, we can probe the I2C/SMBus adapters for connected hardware monitoring devices. This is the most risky part, and while it works reasonably well on most systems, it has been reported to cause trouble on some systems. Do you want to probe the I2C/SMBus adapters now? (YES/no): YES
Do you want to add these lines automatically to /etc/modules? (yes/NO) yes
$ /etc/init.d/kmod start

Its time to configure collectd using /opt/collectd/etc/collectd.conf file. Uncomment following lines to enable mentioned plugins (consider set ValuesPercentage true in CPU plugin AND set our Elasticsearch node IP address in network plugin instead of {YOUR_ELASTICSEARCH_IP} placeholder AND set your host name instead of {YOUR_HOST_NAME} placeholder):

Where <pwd> is the password for the Kibana user from the previous section (in this case etlPv88yBBHanx3Xwf3g) and <server_ip> and <server_name> is the server’s IP address and server name (to enable remote access).

Also, add the followings to the /etc/elasticsearch/elasticsearch.yml :

network.host: <server_ip>
xpack.security.enabled: false

Now you can reboot Elasticsearch and Kibana:

/etc/init.d/elasticsearch restart
/etc/init.d/kibana restart

Check Kibana’s status using following URL:http://<server_ip>:5601/status

In your Kibana, you have to create a default index pattern. So at first, login to your Kibana web interface > Management > Index Patterns > Create Index Pattern and use collectd_* as your index pattern.

In Discover section you can see your data is coming:

Then I created a visualization like below (I add full resolution screenshot so that you can see the variables):

OpenStack Integration

Now that you know how to create visualizations, you may have multiple servers running (hosts, controller, and VMs) and with a single Elasticsearch installation, you can receive all their data. You can create a sophisticated “Dashboard” to see everything in one place:

Final Word

Hopefully, this post will help you in your work and research but don’t hesitate to ask your questions via the comments sections below and I will gladly help as much as I can. I will improve this post time by time, so don’t hesitate to mention mistakes in this so I will fix them. Also please let me know if you want me to write about some specific topic.