Table of Contents |
---|
Status
Current state: Under Discussion Accepted
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
[DISCUSS] KIP-281: ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User
Vote thread: [VOTE] KIP-281: ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User
Vote results:
- Binding +4: Jason Gustafson, Ismael Juma, Rajini Sivaram and Guozhang Wang
- Non-binding +4: Ted Yu, Moshe Blumberg, Alex Dunayevsky, and Dhruvil Shah
- No -1 votes.
JIRA:
Jira | ||||||
---|---|---|---|---|---|---|
|
PULL REQUEST: https://github.com/apache/kafka/pull/4818
Motivation
ConsumerPerformance fails to consume all messages on topics with large number of partitions due to a relatively short default polling loop timeout (1000 ms) that is not reachable and modifiable by the end user.
...
- Increasing polling loop timeout is probably a good thing to handle this case
- Increasing it too much by default (say, 5 minutes) may lead to a slower feedback from the tool in some cases
- Making polling loop timeout optionally changeable is a good thing since it would give us more control for the corner cases
- Printing a warning message in case the timeout is exceeded should also be a good thing to know what's going on
Public Interfaces
Providing the command line tool kafka-consumer-perf-test.sh with an optional --polling-loop-timeout parameter:
parameter name | defaults to | type | time unit | usage |
---|---|---|---|---|
--polling-loop-timeout | 6000010000 | Long | ms | optional |
Proposed Changes
- Increasing polling loop timeout in ConsumerPerformance implementation up to 60 10 000 ms
- Making polling loop timeout reachable and modifiable by the end user from the command line kafka-consumer-perf-test tool
- If Print a warning message if the polling loop timeout is exceeded, print a warning message
Proposed changes in code available in this PR: https://github.com/apache/kafka/pull/4818
Compatibility, Deprecation, and Migration Plan
- Positive impact on existing users since the utility generates more accurate results
- The proposed parameter is optional, will not break anything
Rejected Alternatives
..None.