Stretching Horizons with Elasticsearch

Logmatic.io is an Operations Data Platform for Log and Machine events. They help companies improve their software and business performance by leveraging their machine data.

Improving your Operations with Machine Data

Our customers continuously send us data straight out of their systems: logs, events and metrics. All the data we receive is then funneled into a single hub so that our customers can enjoy a highly efficient monitoring and investigation tool to improve their software. The most advanced users go beyond the technical realm and extract business insights from the systems they oversee.

Logs are received in real time, and searching or analyzing can be made instantly. And by instantly, we don’t mean in one minute or two seconds later, we mean now! Everyday troubleshooting and data investigating becomes way more efficient as you get a clear overview and can spot the needle in the haystack in no time. All the heavy lifting of the gazillion data entries is our problem, not our clients’.

Fig. 1: Logmatic App Screenshots

Processing highly Heterogeneous Client Data

To provide a single and adaptive service, we need to manage and treat our clients’ data in many different ways. Each one of our customers has his own particularities.

They obviously vary in volume and richness of their data. Some send tons of web server access logs, others send a long tail of business KPIs and server metrics. As an illustration, one of our media clients sends us custom logs coming from the app embedded in their set-top boxes, while another of our clients in the real-time video advertising industry sends us java logs and stats to monitor their RTB activity. While that is happening, we have an e-retailer client sending us apache logs and looking at user agent parsing to improve their customer experience on their website. And another client, in the entertainment industry, needs us to collect and analyse events from customers’ browsers using a javascript library.

In addition to heterogeneous data, some clients want to study dashboards aggregating billions of events, while others are only interested in surfacing the few log lines pertaining to that very specific transaction in their system. More often than not, customers want both.

Thus, flexibility is one of the major technical challenges we face in building and developing the our SaaS platform. It was clear from the start we needed to build Logmatic.io using a very accommodating technology: Elasticsearch.

Elasticsearch’s inherent flexibility

Elasticsearch can be tweaked to all of our specific needs due to its high tunability. As we all know, writing a heavy application entails fine-tuning your cluster and many other challenges. So, when confronted with these obstacles, we always find what we need by just browsing through the extensive documentation, blogs and forums in the incredibly active Elasticsearch community - we never feel stuck anywhere.

We also found it very easy to integrate and extend. Being a Java shop, working with Elasticsearch was particularly pleasant. It comes bundled with a very efficient and comprehensible Java client. It even has a plugin that allows us to build our secret recipe right into Elasticsearch. That is how we managed to optimize our analytics, both in terms of speed for some queries and in terms of advanced analytics that are not provided natively by Elasticsearch.

Stability is everything when building a system used for critical operations, so we have to be sure that a single user cannot bring the whole system down. In order to prevent users from taking damaging actions on the platform, we started building a barrier around all Elasticsearch related functionalities. Even then, since we don’t have much control on the number of fields, their cardinality or type, we could potentially face queries eating up all available memory (field data anyone?). With Elasticsearch 1.4, we were able to use doc_values extensively. This makes memory usage much more predictable and lets users run aggregations over billions of documents without impacting others. Innovation is always around the corner! In the end, we managed to build stability in the system along with the flexibility offered to customers.

What’s next?

We’re always looking forward to Elasticsearch releases, and right now, we are waiting for 2.0 and aggregation pipelines. It should help us deliver even better and more flexible operational intelligence experiences, so stay tuned!

Emmanuel Gueidan is co-founder and CTO at Logmatic.io, a Paris-based startup focused on enabling innovative operations and high performance applications through log management. Emmanuel has over 10 years experience building BI platforms. Dedicated to R&D and driven by innovation, he previously worked for Dassault Systems and took over development of several cutting-edge pieces of the core product at Quartet FS, working on one of the first in-memory analytics real-time business intelligence tools.