50 shades of ES

Track

Metodologije i alati

Datum i vrijeme

srijeda, 09. svibanj 2018., 12:00

Dvorana

Dvorana B

Trajanje

30'

What does it take to setup a 40+ node ES cluster able to handle tens of thousands of insertions while serving thousands of search queries?In this presentation I will try to guide you through our expierence at Infobip and what we:- failed at,- wanted to know before failing and- learned on the way.Our concrete use case will presented together with all its aspects from hardware consideration to client service implementation.The use case is simple, provide a way to query half a year of business logs to clients and internal support looking for transaction logs, clients wanting aggregated reports and business analysts wanting to indulge in the data in indicent ways.Implementation considerations are many:Do we need one or more clusters? How many machines? Which ones? Where to put them? How to organise indices?Who or what is going to manage the indices?Do we need some kind of API gateway for the clients?Where to store the indices configuraitons?How to test the cluster?How to fill the cluster?How to migrate the data?What to do when ingestion/search rate is slow?Cluster is down, why o'why?How to join kafka streams in ES?Using Ansible for managing the cluster.Cluster and dataflow monitoring stack.And many more...

O predavaču

Software engineer at Inofbip for almost 4 years, where I work on several high throughput and mission critical systems. For about a year and a half I've been developing part of Infobip's data pipeline and managing several ElasticSearch clusters.I am quite eager about learning new stuff in both professional and private life. The less I know about it the more interesting it is.