Earlier I had blogged here at Toolbox about measuring performance data of a news feed micro-service using Apache Zeppelin. Recently, I blogged about measuring performance data of MySql vs PostGreSql vs Docker using that news feed service. This time, I wrote a custom job, called perf3, that would aggregate the performance data per minute and update Elastic Search with the throughput, mean, median, and 95th percentile metrics for each entity and operation.

Why did I change how I collected the metrics? Wasn't Apache Zeppelin good enough? This time, I wanted to let the test run longer and I wanted to see results as the test was ongoing. The Zeppelin approach was to collect all the raw metrics then analyze the data later. This approach shows you the data in near real-time.

Here are the internals to the perf3 application. At application start time, a performance metric Akka Actor is initialized for every combination of entity and operation to be expected. A scheduled timer is set up to fire off a metric aggregation event to each Actor once a minute. Then the application starts consuming messages from the feed topic in Kafka. These are the messages that the micro-service produces.

As each performance metric event is received, it is added to a collection for that entity and operation. When the metric aggregation event is received, the current collection is processed and a performance metric update event is sent to the elastic search actor. The collection is then reset in order to be ready for the next minute's worth of performance metric events. No explicit synchronization is needed due to how the Akka Actor architecture works.

The screen shots of performance data graphs from the blog are taken of Kibana which works with Elastic Search to easily surface near real-time performance metrics as funneled through Kafka.