DevOps Tools Introduction #13: Log Management and Analysis

April 3, 2018 - by

While running, most applications, containers and virtual machines constantly generate information about numerous events. These events can be anything from severe errors to a simple notice that the server successfully answered a request. Collecting and analysing this log data become challenging in a dynamic architecture or microservice environment. The DevOps Tools Engineer exam covers log management and analysis in objective 705.2. The Elastic Stack, which includes the combination of tools Logstash, Elasticsearch and Kibana, is used as a reference implementation. From these tools, Logstash is the component which usually causes the most configuration effort and is the central focus of the objective.

Equipped with this knowledge, set up your first Elastic Stack. stack-docker provides a Docker Compose file which sets up the components of the Elastic Stack -- and much more. Use this file to both gain more Docker experience and to set up Logstash, Elasticsearch, Kibana and, later, filebeat. Alternatively, follow Sébastien Pujadas’ elk-docker guide for setting up the Elastic Stack Docker.

Now that you have a playground, give a closer look to the Configure logstash guide. Follow all the subchapters, they cover important topics which are mentioned in the objectives.

A critical aspect of any logging infrastructure is collecting logging information and shipping it to the central log server. Filebeat is a modern approach to this need; it collects log data and sends it to another process, such as a Logstash server. The Filebeat documentation provides an overview of Filebeat along with the recommended getting started guide. The Logstash documentation describes Filebeat's counterpart, the Beats input plugin.

Besides the modern Filebeat, LPI also expects you to use syslog to ship log data to Logstash. In case you're not familiar with syslog, Aaron Leskiw’s introduction to syslog is a good place to start. You might also want to review the manpage of syslog.conf(5). To turn Logstash into a syslog server, the Syslog input should be configured.

In addition to the Beats and Syslog input plugins, Logstash's functionality can be extended through the use of numerous Input, Output and Filter plugins. Browse through these indexes to learn more about the modules which are related to the technologies covered in the DevOps Tools Engineer exam. Take a special look at the Grok filter and Email alerting as they are explicitly mentioned in the exam objectives.

Elasticsearch is responsible for storing the log data. While this sounds unspectacular, indexes and data retention should be configured within Elasticsearch to support the analysis of log data. The Elasticsearch documenation’s getting started guide gives you an initial overview of Elastisearch itself. Afterwards, learn more about indexes and retiring data in Elasticsearch.

Fabian Thorns is the Director of Certification Development at Linux Professional Institute, LPI. He is M.Sc. Business Information Systems, a regular speaker at open source events and the author of numerous articles and books. Fabian has been part of the exam development team since 2010. Connect with him on LinkedIn, XING or via email (fthorns at lpi.org).

Your name

Comment *

LPI is a non profit organization.

LPI is the global certification standard and career support organization for open source professionals. With more than 600,000 exams delivered, it's the world’s first and largest vendor-neutral Linux and open source certification body. LPI has certified professionals in over 180 countries, delivers exams in 9 languages, and has hundreds of training partners.

Our purpose is to enable economic and creative opportunities for everybody by making open source knowledge and skills certification universally accessible.