Welcome!

Module 4 - Logging with EFK

Difficulty:Beginner

Estimated Time:20 minutes

Welcome to the Digital Academy "Kubernetes CNCF" series. This is Module 4 - Logging with EFK.

This scenario takes you through the basics of deploying a logging solution on Kubernetes. The premise is all the log streams generated by the containers are aggregated into a central datastore. From that datastore, queries and filters produce views from the aggregated logs.

Containers should only produce logs as event streams and leave the aggregation and routing to other services on Kubernetes. This pattern is emphasized as factor 11 Logs of the The Twelve Factors App methodology.

Commonly the three components ElasticSearch, Fluentd, and Kibana (EFK) are combined for the stack. Sometimes stack use Fluent Bit instead of Fluentd. Fluent Bit is mostly functionally the same, but lighter in features and size. Other solutions sometimes use Logstash (ELK) instead of Fluentd.

In the following steps you will learn:

How to deploy ElasticSearch, Fluentd, and Kibana

How to generate log events and query then in Kibana

Forwarding: Fluent Bit

- fluentbit.io

Fluentd is an open source data collector, that lets you unify the data collection and consumption for a better use and understanding of data. In this stack Fluent Bit runs on each node (DaemonSet) and collects all the logs from /var/logs and routes them to ElasticSearch.

This example could use a lighter variation of Fluentd called Fluent Bit. Perhaps EfK, with a lower case 'f' is apropos. Alen Komljen covers the reason why in his blog.

Another variation for logging is the ELK stack that includes Logstash as a substitution for the Fluent aggregation solution.

Aggregation: ElasticSearch

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

Viewing: Kibana

Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data.

For Kubernetes there are a wide variety of ways to assemble EFK together, especially with a production or business critical clusters. Some solutions may leverage an ElasticSearch service outside the cluster, perhaps offered by a cloud provider. For any solution that's deployed to Kubernetes it's recommended to use Helm charts. Even with Helm charts there are a variety of solutions evolving and competing with each other.

However, this scenario is aimed to show how you can get a working stack up with reasonable ease so you can see how the components are installed and work with each other.

Congratulations!

You've completed the scenario!

Scenario Rating

Conclusion

This stack is a good example of how Kubernetes can be used to bring distinct tools together so they can work in concert for a larger solution. In this case for log aggregation. Because Fluent Bit is installed as a DaemonSet it will be on every node dutifully collection the log streams and sending them to ElasticSearch where in turn Kibana offers a viewport into specific data based on your queries.

It's important your application also logs the transaction correlation IDs as a way to gather log events from a known transaction. This is also true for transaction tracing (a separate Katacoda scenario, Transaction Tracing).

Each one of the three components is highly configurable and this scenario provides a starting point for getting this observability pattern ready for production.

Lessons Learned

With these steps you have learned:

How to configure and deploy ElasticSearch, Fluent Bit, and Kibana on Kubernetes

Install ElasticSearch

Deploy the public Helm chart for ElasticSearch. The chart's default settings are appropriately opinionated for a production deployment. Here, some of the default settings are downsized to fit in this KataCoda cluster.

ElasticsSearch will start in a few minutes and you can observe its progress. Be patient, as it takes time for ElasticSearch to initialize, even with this smaller configuration.

watch kubectl get deployments,pods,services --namespace=logs

Once complete, the 3 Pods will move to the running state. Soon the Deployment status will move to the available (1) state. Use this clear to ctrl-c and clear the shell or press ctrl-c to break out of the watch.

Deploy Fluent Bit

Create the configuration for Fluent Bit.

Install Fluent Bit and pass the ElasticSearch service endpoint as a chart parameter. This chart will install a DaemonSet that will start a Fluent Bit pod on each node. With this each Fluent Bit services will collects the logs from each node and stream it to ElasticSearch.

Security caution. This NodePort exposes the logging to the outside world intentionally for demonstration purposes. However, for production Kubernetes clusters never expose the Kibana dashboard service to the world without any authentication.

Kibana will start in a few moments and you can observe its progress.

watch kubectl get deployments,pods,services --namespace=logs

Once complete, the kibana pod will move to the running state. It will be a few moments and the Deployments will eventually move to the available (1) state. Use this clear to ctrl-c and clear the shell or press ctrl-c to break out of the watch.

You know have a full EFK stack running. Granted its smaller and not configure to he highly available or with access protection, but these 5 pods comprise is a functional solution to get started.

...
2019-03-27T11:06:25+0000 INFO takes the value and converts it to string.
2019-03-27T11:06:29+0000 DEBUG first loop completed.
2019-03-27T11:06:31+0000 ERROR something happened in this execution.
2019-03-27T11:06:46+0000 WARN variable not in use.
...

Inspect the actual log events now being generated with this log command.

kubectl logs deployment/random-logger

Don't be alarmed by the messages, these are just samples.

View Log Events

Access

KataCoda has exposed the NodePort 31000 to access Kibana from your browser.

Access Kibana. There is also a tab above the command line area labeled Kibana that takes you to the same Kibana portal.

Security

Tip: There are no credentials to access this EFK stack through Kibana. For real deployments, you would never expose this type of information without at least an authentication wall. Logs typically reveal lots of dirty laundry and attack vector opportunities.

Portal

To see the logs collected from the random-logger service follow these steps in the Kibana portal.

When Kibana appears for the first time there will be a brief animation while it initializes.

On the Welcome page click Explore on my own.

From the left-hand menu select the top Discover item.

In the form field Index pattern enter _kubernetescluster-*

It should read "Success!" and Click the > Next step button on the right.

In the next form select timestamp from the dropdown labeled Time Filter field name.

From the bottom-right of the form select Create index pattern.

In a moment a list of fields will appear.

Again, from the left-hand menu select the top Discover item.

The log list will appear.

Refine the list a bit by selecting log near the bottom the left-hand Selected fields list.

When you hover over or click on the word log, click the Add button to the right of the label.

The log list now is filtered to show log events from the random-logger service. You can expand each event to reveal further details.

From here you can start to appreciate the amount of information this stack can provide. More information is in the Kibana documentation.

Debugging Scenarios

Help

Katacoda offerings an Interactive Learning Environment for Developers. This course uses a command line and a pre-configured sandboxed environment for you to use. Below are useful commands when working with the environment.

cd <directory>

Change directory

ls

List directory

echo 'contents' > <file>

Write contents to a file

cat <file>

Output contents of file

Vim

In the case of certain exercises you will be required to edit files or text. The best approach is with Vim. Vim has two different modes, one for entering commands (Command Mode) and the other for entering text (Insert Mode). You need to switch between these two modes based on what you want to do. The basic commands are: