Status
Current state: Under Discussion
Discussion thread: here
JIRA:
-
KAFKA-16264Getting issue details...
STATUS
Draft PR: https://github.com/apache/kafka/pull/15888
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
While tuning Producer ID expiration on Produce State Manager, there is a mismatch on how the configurations are exposed: while producer.id.expiration.ms
is exposed as dynamic public configuration, producer.id.expiration.check.interval.ms
is static and internal.
This does not allow to adjust this values when needed: e.g. if there are too many producer ids, an operator may want to reduce the expiration to remove the producer ids faster; but without checking the expiration faster there's no benefit and the producer ids will pill up anyway.
This has lead to issues like - KAFKA-16229Getting issue details... STATUS
Public Interfaces
Broker configuration producer.id.expiration.check.interval.ms to be: public (i.e. not internal) and dynamic.
Config:
name:
producer.id.expiration.check.interval.ms
- default value: 600000 (10 min)
doc:
The interval at which to remove producer IDs that have expired due to <code>producer.id.expiration.ms</code> passing.
Proposed Changes
In practice, this means re-scheduling the Producer Expiration check thread in UnifiedLog component.
Compatibility, Deprecation, and Migration Plan
No compatibility issues. Config already exists, and will just be documented and available as dynamic config.
Test Plan
Extend UnifiedLogTest to validate re-scheduling of Producer Expiration check thread.
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.