what’s the rate of processing kafka messages?

Recently I was implementing a Kafka worker as part of a project that consumes messages from the Kafka queue. As I was new to this messaging system, I didn’t know how to make sure that the worker is functioning as it should. Fortunately, the DropWizard framework provides us with a marvelous metrics endpoint that produces statistics in a neat JSON object. Furthermore, it’s possible to make use of this information by checking the average processing rate for the messages. MetricRegistry provides us with Timer, therefore we can make use of it in our Kafka worker. Timer can be used to determine how much time it takes for our Kafka worker to process the message that has been taken from the queue.

passing an MetricRegistry object to the Kafka worker

In the application class, we need to pass the MetricRegistry object as a Kafka worker constructor method parameter. This can be done by calling metrics method on DropWizard environment as follows:

implementing the Timer

This allows us to make use of MetricRegistry object, therefore we can register a new timer called – KafkaWorkerTimer.

After this, we can make use of the timer in the process method as follows.

This means when the message is taken from the queue, we are starting taking time by timer.time() and when we have finished working with specific message we can call stop() method on individalTime.

MetricRegistry allows us to invoke timer and call various methods, for an example to check what’s the mean of past fifteen minutes for processing the Kafka messages.

conclusion

However, if we would like to check DropWizard application’s health, it wouldn’t be good to use the processing rate of messages as it could give us a false alarm. In a case when the Kafka queue is empty, the DropWizard application would be marked as unhealthy, even though it’s in a healthy condition for the queue and the application. In my next blog post I will write about DropWizard health checks and how to determine if the application is unhealthy by checking if the Kafka queue is lagging.

If you enjoyed the read, drop us a comment below or share the article, follow us on Twitter or subscribe to our #MetaBeers newsletter. Before you go, grab a PDF of the article, and let us know if it’s time we worked together.