Fluentd is a flexible and robust event log collector, but Fluentd doesn’t have own data-store and Web UI.
If you want to analyze the event logs collected by Fluentd, then you can use Elasticsearch and Kibana :)

Elasticsearch is an easy to use Distributed Search Engine and Kibana is an awesome Web front-end for Elasticsearch.

After Fluentd flushed received events to Elasticsearch, you can analyze the event logs via Kibana!
Following image is one panel example:

Kibana has some built-in panels, so you can create own dashboard easily. See Kibana demo

Advanced tips

If your service has a high traffic, then fluent-plugin-elasticsearch sometimes get stucked.
In this case, built-in out_roundrobin plugin is useful.
You can distribute a write request to elasticsearch nodes for load balancing.

Of course, putting Queue with multiple fluentd nodes is an another approach.

Conclusion

This article introduced Fluentd, Elasticsearch and Kibana combination to analyze the event logs.
These components are easy to setup and work fine, so you can try this framework soon!
I heard many companies have already tried / deployed this setup on production :)

In this week, Fluentd v0.10.43 has been released.
Since this version, Fluentd introduced log_level parameter in Input / Output plugin.
It enables you can set different log level separated from global log level, e.g. -v, -q command line option.

This article shows “How to support log_level option in your plugin.”

log_level option use cases

Disable in_tail warning

in_tail prints “pattern no match” warning when receives invalid log. It is useful information for almost users, but some users want to ignore this log for other important plugin warning.

In this case, you can set “log_level error” in in_tail configuration to disable “pattern no match”.

12345

<source>type tail
...
log_levelerror</source>

Debugging

Without log_level, we get many verbose logs using -vv command line option for one plugin. With log_level, you can set verbose configuration in only one plugin.

It is useful for debugging a plugin on acutual environment.

12345

<matchfoo.**>type unstable_plugin
...
log_level trace
</match>

Support log_level option in your plugin

This section is for plugin developers.

First of all, Fluentd provides $log object as heretofore. So all plugin should work without changing on Fluentd v0.10.43 or later.

To suppot log_level is very easy. Replace $log with log. Following example is fluent-plugin-td’s diff: