Knowledge Base

Docker Logging with Fluent Bit and Elasticsearch

Latest versions of Docker comes with a logging layer feature which allows to define specific drivers that can handle the Container applications logs, specifically the ones that are send to the standard output (stdout) and standard error (stderr) interfaces.

Starting from Docker v1.8, it provides a Fluentd Logging Driver which implements the Forward protocol. Fluent Bit have native support for this protocol, so it can be used as a lightweight log collector. On this article we will demonstrate how to collect Docker logs with Fluent Bit and aggregate them back to a Elasticsearch database.

Docker Logs

Every message than a containerized application writes to stdout or stderr interface, is packaged and associated with some metadata:

The records comes with the container name, container ID, interface among others. Our goal is to connect Docker logging driver with Fluent Bit so then we can send the logs to Elasticsearch:

Requirements

In order to follow this tutorial, make sure that you have an updated Linux system, check that Docker is installed and verify that Elasticsearch service is up and running. For more details about how to accomplish this, check their official documentation:

The configuration above says the core will try to flush the records every 5 seconds. It will listen for Forward messages on TCP port 24224 and deliver them to a Elasticsearch service located on host 192.168.2.3 and TCP port 9200. By default it will match all incoming records.

Note that Elasticsearch requires the Index and Type, this can be confusing if you are new to Elasticsearch, if you have used some relational database before, they can be compared to the database and table concepts.

Running the Docker Container

The following command will run a simple echo command in a container that prints out a message to the standard output (stdout). Pay attention to the extra given options which specify where the logs should go:

When specifying the fluentd driver, it will assume that will forward the logs to localhost on TCP port 24224. If you want to change that value you can use the --log-opt fluentd-address=host:port option

Query Elasticsearch

After five seconds you will be able to check the records in your Elasticsearch database, do the check with the following command: