Kafka Performance: System-Level Broker Tuning

Operating system related kernel parameters affect overall performance of Kafka. These parameters can be configured via sysctl at runtime. To make kernel
configuration changes persistent (that is, use adjusted parameters after a reboot), edit /etc/sysctl.conf. The following sections describe some important kernel
settings.

File Descriptor Limits

As Kafka works with many log segment files and network connections, the Maximum Process File Descriptors setting may need to be increased in some cases in
production deployments, if a broker hosts many partitions. For example, a Kafka broker needs at least the following number of file descriptors to just track log segment files:

The Maximum Process File Descriptors setting can be monitored in Cloudera Manager and increased if usage requires a larger value than the default
ulimit (often 64K). It should be reviewed for use case suitability.

To review FD limit currently set for a running Kafka broker, run cat /proc/KAFKA_BROKER_PID/limits, and look for
Max open files.

To see open file descriptors, run:

lsof -p KAFKA_BROKER_PID

Filesystems

Linux records when a file was created (ctime), modified (mtime) and accessed (atime). The
value noatime is a special mount option for filesystems (such as EXT4) in Linux that tells the kernel not to update inode information every time a file is accessed
(that is, when it was last read). Using this option may result in write performance gain. Kafka is not relying on atime. The value relatime is another mounting option that optimizes how atime is persisted. Access time is only updated if the previous atime was earlier than the current modified time.

To view mounting options, run mount -l or cat /etc/fstab command.

Virtual Memory Handling

Kafka uses system page cache extensively for producing and consuming the messages. The Linux kernel parameter, vm.swappiness, is a value from 0-100 that
controls the swapping of application data (as anonymous pages) from physical memory to virtual memory on disk. The higher the value, the more aggressively inactive processes are swapped out from
physical memory. The lower the value, the less they are swapped, forcing filesystem buffers to be emptied. It is an important kernel parameter for Kafka because the more memory allocated to the swap
space, the less memory can be allocated to the page cache. Cloudera recommends to set vm.swappiness value to 1.

To check memory swapped to disk, run vmstat and look for the swap columns.

Kafka heavily relies on disk I/O performance. vm.dirty_ratio and vm.dirty_background_ratio are kernel parameters that
control how often dirty pages are flushed to disk. Higher vm.dirty_ratio results in less frequent flushes to disk.

To display the actual number of dirty pages in the system, run egrep "dirty|writeback" /proc/vmstat

Networking Parameters

Kafka is designed to handle a huge amount of network traffic. By default, the Linux kernel is not tuned for this scenario. The following kernel settings may need to be tuned based on use
case or specific Kafka workload:

If this documentation includes code, including but not limited to, code examples, Cloudera makes this available to you under the terms of the Apache License, Version 2.0, including any required
notices. A copy of the Apache License Version 2.0 can be found here.