Status
Current state: Under Discussion
Discussion thread:
JIRA:
-
KAFKA-10357Getting issue details...
STATUS
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Kafka Streams uses repartition topics to repartition the data when a key-based operation, e.g., aggregation or join, follows a key-changing operation, e.g., map. Additionally, Kafka Streams uses changelog topics to replicate the data in its state store for fault-tolerance. If any required repartition topic or changelog topic does not exist, it is created during a rebalance. More precisely these internal topics are created during the computation of the assignment. Consequently, if one of the internal topics is deleted between rebalances, it will be silently recreated as an empty topic. If the deletion of the internal topics was a mistake or – even worse – a malicious act, data might be lost. Kafka Streams users would either not notice it or notice it too late to to limit the damage by stopping processing.
Public Interfaces
Proposed Changes
Compatibility, Deprecation, and Migration Plan
Since metrics are only added and no other metrics are modified, this KIP should not
- affect backward-compatibility
- deprecate public interfaces
- need a migration plan other than adding the new metrics to its own monitoring component