CDAP Community Issue Trackerhttps://issues.cask.co
This file is an XML representation of an issueen-us7.8.27800203-04-2018[CDAP-7731] Add fetch size and transaction flush interval configuration for Kafka Consumer Flowlethttps://issues.cask.co/browse/CDAP-7731
CDAP<p>When a KafkaConsumerflowlet starts after failover, and when there are huge number of messages left in the topic to be picked up, the flowlet tries to process all the backlog of messages at once and because of their existing limitation on Transaction timeout, we get in to a infinite loop of “Transaction timeout Exceptions”</p>
<p>We need at the minimum two configurable properties in Kafka Consumer Flowlet</p>
<p>1. Able to configure number of messages to fetch in a single batch<br/>
2. Able to configure a time interval for periodic transaction commits</p>
CDAP-7731Add fetch size and transaction flush interval configuration for Kafka Consumer FlowletImprovementCriticalResolvedFixedSagar KapareBrett LamportTRFri, 2 Dec 2016 09:52:52 -0800Sat, 15 Apr 2017 07:12:38 -0700Tue, 4 Apr 2017 16:10:41 -07003.5.24.2.04.1.13.5.44.0.2CDAP Services04<p>Implemented against release/cdap-3.5-compatible: <a href="https://github.com/caskdata/cdap-packs/pull/84" class="external-link" rel="nofollow">https://github.com/caskdata/cdap-packs/pull/84</a></p><p>This fix needs to be ported to 4.0.2, 4.1.1 and 4.2</p><p>Assigning to <a href="https://issues.cask.co/secure/ViewProfile.jspa?name=sagar" class="user-hover" rel="sagar">Sagar Kapare</a> to port to upstream branches</p><p>PR for release/cdap-4.0-compatible branch - <a href="https://github.com/caskdata/cdap-packs/pull/85" class="external-link" rel="nofollow">https://github.com/caskdata/cdap-packs/pull/85</a></p><p>PR for release/cdap-4.1-compatible branch - <a href="https://github.com/caskdata/cdap-packs/pull/86" class="external-link" rel="nofollow">https://github.com/caskdata/cdap-packs/pull/86</a></p><p>PR for develop (4.2.0 compatible) branch - <a href="https://github.com/caskdata/cdap-packs/pull/87" class="external-link" rel="nofollow">https://github.com/caskdata/cdap-packs/pull/87</a></p>DevelopmentRank1|hzzpg7:Release NotesAdded fetch size and transaction flush interval configurations to the Kafka Consumer Flowlet.