Description

When a KafkaConsumerflowlet starts after failover, and when there are huge number of messages left in the topic to be picked up, the flowlet tries to process all the backlog of messages at once and because of their existing limitation on Transaction timeout, we get in to a infinite loop of “Transaction timeout Exceptions”

We need at the minimum two configurable properties in Kafka Consumer Flowlet

1. Able to configure number of messages to fetch in a single batch
2. Able to configure a time interval for periodic transaction commits