Free Alternative to Splunk Using Fluentd

Splunk is a great tool for searching logs, but its high cost makes it prohibitive for many teams. In this article, we present a free and open source alternative to Splunk by combining three open source projects: Elasticsearch, Kibana, and Fluentd.

Elasticsearch is an open source search engine known for its ease of use. Kibana is an open source Web UI that makes Elasticsearch user friendly for marketers, engineers and data scientists alike.

By combining these three tools (Fluentd + Elasticsearch + Kibana) we get a scalable, flexible, easy to use log search engine with a great Web UI that provides an open-source Splunk alternative, all for free.

In this guide, we will go over installation, setup, and basic use of this combined log search solution. This article was tested on Mac OS X Mountain Lion. If you’re not familiar with Fluentd, please learn more about Fluentd first.

fluent-plugin-elasticsearch comes with a logstash_format option that allows Kibana to search stored event logs in Elasticsearch.

Once everything has been set up and configured, we’ll start td-agent.

$ sudo /etc/init.d/td-agent start

Set Up rsyslogd

In our final step, we’ll forward the logs from your rsyslogd to Fluentd. Please add the following line to your /etc/rsyslog.conf, and restart rsyslog. This will forward your local syslog to Fluentd, and Fluentd in turn will forward the logs to Elasticsearch.

*.* @127.0.0.1:42185

Please restart the rsyslog service once the modification is complete.

$ sudo /etc/init.d/rsyslog restart

Store and Search Event Logs

Once Fluentd receives some event logs from rsyslog and has flushed them to Elasticsearch, you can search the stored logs using Kibana by accessing Kibana’s index.html in your browser. Here is an image example.

To manually send logs to Elasticsearch, please use the logger command.

$ logger -t test foobar

When debugging your td-agent configuration, using out_copy + out_stdout will be useful. All the logs including errors can be found at /etc/td-agent/td-agent.log.

Conclusion

This article introduced the combination of Fluentd and Kibana (with Elasticsearch) which achieves a free alternative to Splunk: storing and searching machine logs. The examples provided in this article have not been tuned.

If you will be using these components in production, you may want to modify some of the configurations (e.g. JVM, Elasticsearch, Fluentd buffer, etc.) according to your needs.