Collecting and Analyzing Ruby on Rails Logs

Scenario

You have an application written in Rails and want to collect data into MongoDB, HDFS, Elasticsearch, et. al. for analytics/search.

Logging directly into MongoDB/HDFS/Elasticsearch is not highly recommended since synchronous logging is slow/potentially hazardous for the backend. You can build asynchronous logging into your application, but Fluentd can sit between your application and backend systems to achieve reliable, asynchronous logging.

Rails comes with an excellent logging API, which we will take advantage of to connect Rails logger to Fluentd.

We assume that Rails is 4.0 and above in the rest of this article.

Setup

Set up Fluentd with in_forward. By default, Fluentd has this enabled. It is the following section in the configuration file: