This tutorial shows you how to use Elasticsearch, Fluentd, and Kibana to build an open source stack that helps you manage complex data systems.

Subscribe now

Get the highlights in your inbox every week.

Managing an infrastructure of servers is a non-trivial task. When one cluster is misbehaving, logging in to multiple servers, checking each log, and using multiple filters until you find the culprit is not an efficient use of resources.

The first step to improve the methods that handle your infrastructure or applications is to implement a centralized logging system. This will enable you to gather logs from any application or system into a centralized location and filter, aggregate, compare, and analyze them. If there are servers or applications, there should be a unified logging layer.

Thankfully, we have an open source stack to simplify this. With the combination of Elasticsearch, Fluentd, and Kibana (EFK), we can create a powerful stack to collect, store, and visualize data in a centralized location.

Let’s start by defining each component to get the big picture. Elasticsearch is an open source distributed, RESTful search and analytics engine, or simply an object store where all logs are stored. Fluentd is an open source data collector that lets you unify the data collection and consumption for better use and understanding of data. And finally, Kibana is a web UI for Elasticsearch.

There are other ways to collect logs, like running a small Fluentd forwarder in each host, but that’s beyond the scope of this article.

Requirements

We will install each component in its own Docker container. With Docker we can deploy each component faster, focusing in EFK rather than distro-specific bits, and we can always delete the containers and start all over again. We will be using official, upstream images.

The multiple parameters with -e are environment variables that are passed to the container to change configurations.

We define a custom ulimit to disable swapping for performance and node stability.

Containers, by design, are ephemeral. This means they don’t store data, so in order to keep the data and logs safe, we need to create a volume and mount it inside the container. In our case it is mounted to /usr/share/elasticsearch/data. This is the path where Elasticsearch stores the data.

Verify the volume was created:

$ sudo docker volume ls
[...]
local elasticdata

This volume will survive even if you delete the container.

Great! Elasticsearch is running. Let’s move on.

Running Kibana

Kibana is a much simpler command. Execute the following command to spin it:

The block <source> enables the syslog plugin, the port and address where it will listen. The block <match rsyslog.**> will match all logs coming from the syslog plugin and will send the data to Elasticsearch.

Your Unified Logging Stack is deployed. Now it’s time to point configure your host's rsyslog to send the data to Fluentd.

Log into each of the nodes you want to collect logs from, and add the following line at the end of /etc/rsyslog.conf :

*.* @<Docker Host>:42185

Then restart rsyslog service:

$ sudo systemctl restart rsyslog

Don’t forget to check Kibana—all your logs are going to be there.

Wrapping everything up with Docker Compose

We can use Docker Compose to combine all the steps we did previously into a single command. Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file (known as a Compose file) to configure an application's services; in our case, EFK services.

With one simple YAML file, your proof of concept is ready to be deployed anywhere, with consistent results. When you have tested the solution thoroughly, don't forget to read the official Elasticsearch, Fluentd, and Kibana documentation to make your implementation production grade.

As you play with the EFK (and Docker) you will recognize how practical it is, and your life as a sysadmin will never be the same.

Further reading

Topics

About the author

Michael Zamot - Michael Zamot is an open source enthusiast whose passion began in 2004, when he discovered Linux. Ever since then he has worked and played with various open source projects, including Linux, OpenStack, OpenShift/Kubernetes and many more, and participated in community events by teaching, conducting workshops, and providing technical support and mentorship. He currently works for Red Hat as a Cloud Consultant, designing, deploying, and supporting complex cloud solutions.

Footer

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat.

Opensource.com aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. Red Hat and the Red Hat logo are trademarks of Red Hat, Inc., registered in the United States and other countries.