DefaultEventHandler can send more produce requests than necesary

Details

Description

In DefaultEventHandler, for a batch of messages, it picks a random partition per message (when there is no key specified). This means that it can send up to P produce requests where P is the number of partitions in a topic. A better way is probably to pick a single random partition for the whole batch of messages. This will reduce the number of produce requests.

Guozhang Wang
added a comment - 09/Jul/13 01:54 Proposed code change:
1. Add a Map<Topic, PartitionIndex> to DefaultEventHandller.
2. At the start of handle(events: Seq[KeyedMessage [K,V] ]) function, clear the map.
3. In getPartition function, if key == null, check if there is already an entry in the map with the given topic:
3.1. If not, choose one partition from the availablePartitions, and add that partition to the map, return this partition index.
3.2. If yes, check if its leaderBrokerIdOpt is defined; if yes, return this partition index directly, otherwise throw an exception.