Child pages
  • KIP 230: Name Windowing Joins
Skip to end of metadata
Go to start of metadata

Status

Current state: "Under Discussion"

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, the internal name of a windowing join is generated at runtime using a counter to ensure uniqueness within a running topology. When a topology is changed and redeployed, different names can be generated for the same window. This results in lost windowing state without intervention modifying offsets to re-consume the input topics.

Public Interfaces

Proposed Changes

The proposed change would be to

  • Add an optional configuration parameter "joinName" to `org.apache.kafka.streams.kstream.Joined`. 
  • If provided, this would be used by KStreamImpl to generate internal names when building a join.
    • Windowing joins: The joinName would be used in the naming of the internal topics
    • Kstream-KTable joins: The joinName would be used in the naming of repartition topics, if they are needed.
  • If not provided the existing name generation would be used.
  • If a naming conflict occurs, generate a Topology Builder exception. This matches the behavior of existing methods that accept user-provided names, such as `reduce`:
    • org.apache.kafka.streams.errors.TopologyBuilderException: "Invalid topology building: Topic reduction-same-name-repartition has already been registered by another source."

Example changes:

JoinName Option added to Joined:

 

Joined

 

 
JoinName used to generate internal topic name, if provided:
KStreamImpl.java

 

Compatibility, Deprecation, and Migration Plan
  1. Since this is an opt-in, optional parameter there would be no impact on existing code.
  2. The join builder methods already accept a Joined instance for configuration, so no API changes are needed in the builder.



  • No labels