Current state: "Accepted" [VOTE] KIP-224: Add configuration parameters `retries` and `retry.backoff.ms` to Streams API
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Streams API can fail with a TimeoutException if a global state (or global KTable) is used. This is an issue if the brokers are temporarily not available, as no retry strategy is provided, and thus the
GlobalStreamThread (and consequently the whole KafkaStreams instance) dies. To make Streams API robust against such issues, it would be required to implement a configurable retry strategy.
We suggest to add a new configuration parameters to
StreamsConfig namely "retries" similar to KafkaConsumer and KafkaProducer. To keep the current "fail-fast" strategy for default config, default value for "retries" could be 0. Furthermore, we would use already existing parameter "retry.backoff.ms" for this retry strategy.
We would apply both parameters in GlobalStateManagerImpl to guard against TimeoutException for KafkaCosumer#partitionsFor and KafkaConsumer#endOffsets.
To align with current design, we would also pass the new parameter to producer and consumer if there is no "producer." or "consumer." prefix. Thus, users can still configure different values for consumer/producer if needed.
Compatibility, Deprecation, and Migration Plan
- This change is backwards compatible as we only add new parameters and also keep the current behavior with default parameter settings.
We add tests with mocking the global consumer client to throw TimeoutException to check if retry parameters are respected.