For those interested in Logsene, our Logstash + Syslog + Elasticsearch + Kibana service mentioned in the talk, we’ll notify you when Logsene becomes fully (and freely!) available next month if you leave your name on the Logsene page.

Below is a sketchnote of the whole talk, which was printed and given to all attendees. Click on the image to get the full resolution.

Post navigation

2 thoughts on “Presentation: On Centralizing Logs”

The architecture involving Logstash was just an example, it’s not something I’m currently using in production. That said, here are the the reasons for providing this example in the first place 🙂
– Lumberjack is a lot lighter on your server (e.g.: waaay lower memory consumption) than a Logstash agent. If that isn’t a problem for you, I’d go with a simpler architecture of using Logstash directly. Less moving pieces == less trouble 🙂
– Redis in this case is used as a buffer. For example, if the ES cluster can’t keep up with a spike of load, or needs to be restarted.
– I didn’t know about the Redis river. Although I still don’t know about the 0.90.5 one. I see one for 0.90.1:https://github.com/sksamuel/elasticsearch-river-redis
and one for 0.20.x:https://github.com/leeadkins/elasticsearch-redis-river

Anyway, I’m guessing rivers should work as well, if you do all the processing you need with the Logstash that’s placed before Redis. I think using Logstash after Redis instead of a river would still make sense if:
a) you’re using the elasticsearch_http output, which gives you the flexibility of having a different ES version than the one Logstash is built against
b) you want to do most of the processing (such as grok) after Redis. This makes sense if you make the Logstash before Redis to do little/no processing, so that it doesn’t become a bottleneck. If you do the heavy processing after Redis and Logstash proves to be too slow, data gets buffered in Redis, giving you a chance to fire up another Logstash instance and fix the problem. If the same slowness happens with the Logstash before Redis, you might lose data.

I’m very interested in your experience and architecture choices.
Is it possible to you to answer this questions ?

– why did you used Lumberjack and not directly Logstash client ?

Logstash 1.2.1 can push directly to ElasticSearch 0.90.5,
– why did you used Redis ?
– in this case, what is the desired function used by Redis ?

ElasticSearch 0.90.5 has a Redis river.
For more I know it’s possible to add a filters and codecs into Logstash but it’s the same thing too onto ElasticSearch
– What is the interest to use a second Logstash between Redis and Elasticsearch ?

Services

About

Contact

Apache Lucene, Apache Solr and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S.
and in other countries. Sematext Group, Inc. is not affiliated with Elasticsearch BV.