Producer using broker list does not load balance requests across multiple partitions on a broker

Details

Description

https://issues.apache.org/jira/browse/KAFKA-129 introduced a bug in the load balancing logic of the Producer using broker.list.
Since the broker.list doesn't specify the number of partitions in total, it should ideally pick a broker randomly, and then send the produce request with partition id -1, so that the EventHandler routes the request to a random partition.
Instead of that, it defaults to 1 partition on each broker and ends up using the Partitioner to pick a partitions amongst the available ones.

Activity

Fixing the load balancing strategy for the broker.list option on the Producer. With this change, when broker.list is used, the producer will pick a random broker and send request to partition -1 on that broker.

Neha Narkhede
added a comment - 20/Oct/11 17:34 Fixing the load balancing strategy for the broker.list option on the Producer. With this change, when broker.list is used, the producer will pick a random broker and send request to partition -1 on that broker.

1. We should guard logger.debug with the isDebugEnabled check.
2. The method name getNumPartitionsForTopic is misleading since the return value is a list of partitions, instead of number of partitions. Can we rename it to sth like getPartitionListForTopic? Ditto for the name of the variable being assigned to.

Jun Rao
added a comment - 20/Oct/11 18:53 1. We should guard logger.debug with the isDebugEnabled check.
2. The method name getNumPartitionsForTopic is misleading since the return value is a list of partitions, instead of number of partitions. Can we rename it to sth like getPartitionListForTopic? Ditto for the name of the variable being assigned to.