Log Management with ELK

Managing system and application logs from cluster nodes

You can pipe system and application logs from the nodes in a DC/OS cluster to an Elasticsearch server.

What this document does and does not cover

This document describes how to send Fluent Bit output from each node to a centralized Elasticsearch instance. This document describes how to directly stream from Fluent Bit into Elasticsearch. Logstash is not used in this architecture. If you are interested in filtering, parsing and understanding the logs with an intermediate Logstash stage, see the Logstash documentation and the example in Filtering logs with ELK.

This document does not explain how to set up and configure an Elasticsearch server. This document does not describe how to set up secure TLS communication between the Fluent Bit instances and Elasticsearch. For details on how to achieve this, see the Fluent Bit and Elasticsearch documentation.

Prerequisites

An existing Elasticsearch installation that can ingest data for indexing

All DC/OS nodes must be able to connect to your Elasticsearch server on the port used for communication between Elasticsearch and Fluent Bit (9200 by default)

A location on each DC/OS node for your custom Fluent Bit config. This tutorial will use /etc/fluent-bit/.

Step 1: Master nodes

For each master node in your DC/OS cluster, create a file /etc/fluent-bit/fluent-bit.conf that includes the default master Fluent Bit config and adds your configuration for the Elasticsearch output plugin. For more information on configuring Fluent Bit to send logs to Elasticsearch, see the Fluent Bit documentation.

Step 2: Agent nodes

For each agent node in your DC/OS cluster, create a file /etc/fluent-bit/fluent-bit.conf that includes the default master Fluent Bit config and adds your configuration for the Elasticsearch output plugin. For more information on configuring Fluent Bit to send logs to Elasticsearch, see the Fluent Bit documentation.