Table of Contents |
---|
Status
Current state: Under Discussion
...
PULL REQUEST: https://github.com/apache/kafka/pull/4818
Motivation
ConsumerPerformance fails to consume all messages on topics with large number of partitions due to a relatively short default polling loop timeout (1000 ms) that is not reachable and modifiable by the end user.
...
- Increasing polling loop timeout is probably a good thing to handle this case
- Increasing it too much by default (say, 5 minutes) may lead to a slower feedback from the tool in some cases
- Making polling loop timeout optionally changeable is a good thing since it would give us more control for the corner cases
- Printing a warning message in case the timeout is exceeded should also be a good thing to know what's going on
Public Interfaces
Providing the command line tool kafka-consumer-perf-test.sh with an optional --polling-loop-timeout parameter:
parameter name | defaults to | type | usage |
---|---|---|---|
--polling-loop-timeout | 60000 | Long | optional |
Proposed Changes
- Increasing polling loop timeout in ConsumerPerformance implementation up to 60 000 ms
- Making polling loop timeout reachable and modifiable by the end user from the command line kafka-consumer-perf-test tool
- If the polling loop timeout is exceeded, print a warning message
Proposed changes in code available in this PR: https://github.com/apache/kafka/pull/4818
Compatibility, Deprecation, and Migration Plan
- Positive impact on existing users since the utility generates more accurate results
- The proposed parameter is optional, will not break anything
Rejected Alternatives
...