Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-16264 - Getting issue details... STATUS

Draft PR: https://github.com/apache/kafka/pull/15888 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

While tuning Producer ID expiration on Produce State Manager, there is a mismatch on how the configurations are exposed: while producer.id.expiration.ms is exposed as dynamic public configuration, producer.id.expiration.check.interval.ms is static and internal.

This does not allow to adjust this values when needed: e.g. if there are too many producer ids, an operator may want to reduce the expiration to remove the producer ids faster; but without checking the expiration faster there's no benefit and the producer ids will pill up anyway.

This has lead to issues like KAFKA-16229 - Getting issue details... STATUS

Public Interfaces

Broker configuration producer.id.expiration.check.interval.ms to be: public (i.e. not internal) and dynamic.

Config:

  • name: 

    producer.id.expiration.check.interval.ms
  • default value: 600000 (10 min)
  • doc: 

    The interval at which to remove producer IDs that have expired due to <code>producer.id.expiration.ms</code> passing.

Proposed Changes

In practice, this means re-scheduling the Producer Expiration check thread in UnifiedLog component. 

Compatibility, Deprecation, and Migration Plan

No compatibility issues. Config already exists, and will just be documented and available as dynamic config.

Test Plan

Extend UnifiedLogTest to validate re-scheduling of Producer Expiration check thread.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.

  • No labels