...
the new KafkaConsumer#commitSync
will accept user-specified timeout.
Proposed Changes
One change shall be made to KafkaConsumer#poll()
whose calls in Kafka internals (specifically fetchCommittedOffsets()
) will block indefinitely. This is the exact same problem as what other methods face (such as committed
() and position()
). To avoid blocking indefinitely, the new methods will take user-determined timeout to define the maximum amount of time for which a method blocks.
...
1. KafkaConsumer#poll()
, since it returns offsets, will return nothing (or what KIP-288 suggests is to throw a TimeoutException while metadata is updating).
2. A ClientTimeoutException
will be introduced to allow users to more clearly identify the reason why the method timed out. (e.g. LeaderNotAvailable, RequestTimeout etc)
...
Feasible and Rejected Alternatives
Please see KIP-288 for other rejected alternatives.
In discussion, many have raised the idea of using a new config to set timeout time for methods which is being changed in this KIP. It would not be recommended to use one config for all methods. However, we could use it for similar methods (e.g. methods which call updateFetchPositions()
will block using one timeout configured by the user). In this manner, we could incorporate both the config and the added timeout parameter into the code.
Another alternative of interest is that we should add a new overload for poll()
, particularily since the changing the old method can become unwieldly between different Kafka versions. To elaborate, a Timeout parameter will also be added to the poll() overload.
One alternative was to add a timeout
parameter to the current position()
and other methods. However, the changes made by the user will be much more extensive then basing the time constraint on requestTimeoutMs
because the method signature has been changed.
...