Status

Current state: Under Discussion

Discussion thread: https://lists.apache.org/thread/crlnb6bjh1srzyyzcgr9g7js38vrt657

JIRA: KAFKA-18996 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-813 introduced "read-only state stores" in Kafka Streams. These state stores use their source topic as a changelog for recovery rather than creating a dedicated changelog topic. While this functionality was added to the Processor API (Topology.addReadOnlyStateStore), it is not currently accessible through the high-level DSL API. This improvement would expose this functionality through the StreamsBuilder class.

Read-only state stores are useful for efficiently sharing state across applications, especially with the introduction of tiered storage support, where compacted topics can grow very large. By using the source topic as a changelog, applications can avoid duplicating large amounts of data. Making this functionality available in the high-level DSL would provide a more user-friendly interface to this feature.

Public Interfaces

org/apache/kafka/streams/StreamsBuilder will be extended with an addReadOnlyStateStore method, allowing the changelog topic to be passed in.

Proposed Changes

The proposed DSL method has a simplified signature compared to the Processor API method, maintaining consistency with other DSL methods like StreamsBuilder.addGlobalStore(_)), which use the same parameter pattern:

StreamsBuilder addReadOnlyStateStore(final StoreBuilder<?> storeBuilder,
									 final String topic,
    								 final Consumed<KIn, VIn> consumed,
									 final ProcessorSupplier<KIn, VIn, Void, Void> stateUpdateSupplier)

Compatibility, Deprecation, and Migration Plan

  • This change is fully backwards compatible as it only adds new methods
  • No migration is needed
  • No features are being deprecated

Test Plan

Integration tests will be added to verify:

  1. Basic read-only state store functionality in the DSL
  2. State recovery after restart
  3. Thread shutdown behaviour

Rejected Alternatives

None. This is a straightforward exposure of existing functionality to a higher-level API.

  • No labels