Current state: Under Discussion
Discussion thread: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Eversince KIP-266 was concluded, several new versions of KafkaConsumer's methods has been introduced. Meanwhile, older versions has been deprecated in favor of these newer methods which allows us to control how much time each individual method blocks. Therefore, there has been a pressing need to migrate Kafka Streams from these old API to newer ones. It has also been noted that some of the older methods which Kafka Streams call blocks infinitely, migrating Kafka Streams will help solve that issue.
Currently, there are a couple of options that are available to us:
- A couple of old configurations were related with blocking in Streams:
RETRY_BACKOFF_MS_CONFIG. However, it has been noted that the application of these two configs has been relatively restricted, mainly being used in global consumer's methods (per current design). We might remove these configs in favor of a larger timeout.
- Another concern is the introduction of another config for blocking. We might use a global configuration (similar to
request.timeout.ms) or allow a user-specified timeout which is passed in via methods (the latter option was favored over the former in KIP-266).
We will change the KafkaConsumer methods used in Kafka Streams to the newer methods created in KIP-266.
Compatibility, Deprecation, and Migration Plan
The older methods will be deprecated in favor of the newer ones found in Kafka Streams if the user-specified timeout approach is found to be more applicable to this KIP.
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.