Improve Apache Kafka performance with flash storage.

Flash storage enables higher throughput which means a higher number of sustained messages per second can be processed.

My official role at Micron is Principal Storage Solutions Engineer. What that title really means is that I work to determine how flash storage can improve the performance and results of workload applications in the real world. To that end, I decided to evaluate one of the most used distributed messaging systems in the Big Data ecosystem, Apache Kafka. I wanted to see how Micron SSDs could be best applied to Apache Kafka and what sort of gains might be had by doing so. If you aren’t familiar with Apache Kafka, you might want to check out this, http://kafka.apache.org/, for a quick primer.

Monitoring the various resources involved, i.e. CPU, memory, disk activity and network, I saw that the primary bottleneck factors were disk and networking, in that order.

It’s A Throughput Game

What I found is that, with Apache Kafka, it was a throughput game. The Kafka developers have done an excellent job of passing writes directly to the kernel page cache to minimize any amount of I/O related issues. However, regardless of this development feature, the I/O still eventually translates into sequential writes being made to the Kafka partition (topic log file). So obviously, the higher the throughput of the disk being used, the more significant increases in performance should be achieved.

Once it was figured out how best to test Apache Kafka as well as what parameters to best use, a simple test was done using their built-in producer test scripts. A total of 600 million 100 byte messages were sent from 3 producers to a single Kafka broker.

The test consisted of the following: (No tuning was done, only default configurations were used.)

A total of 600 topics were generated.

Each producer was assigned 200 topics, specific to that producer.

Each producer created 1,000,000 messages per topic.

The message size used was 100 bytes per message.

Hardware used:

Each server, the 1 broker and 3 producers were identical in configuration.

For each test, the Apache Kafka broker partitions were located on drive being tested.

The results were as follows:

Drive

Drive Throughput MB/s

I/O per Second

Messages (MBytes)

6TB 7.2k HDD

160

300

13.16

Micron 5100 ECO 1920GB SSD

500

2,000

37.94

Micron 9100 Pro 3.2TB NVMe

960

7,500

62.21

As you can see from the above table, the higher the throughput translates into higher I/O per second which for Apache Kafka, means a higher number of sustained messages per second can be processed, shown in MB/s.

Conclusion

Using a higher throughput disk device in your Apache Kafka configuration, such as the Micron 5100 series SSD or a Micron NVMe SSD, will significantly increase performance in your Apache Kafka installation.