Current state: Adopted
Discussion thread: here
Since KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention, the
replication.policy.separator has been used to control the name of internal topics for
DefaultReplicationPolicy, causing backward compatibility issues with offset syncs and checkpoint topics when users set a customized
replication.policy.separator. This impacted 3.1, 3.2, 3.3, 3.4, 3.5 versions.
Currently, the only way to work around this is either by providing a new version of ReplicationPolicy (which can optionally subclass the DefaultReplicationPolicy class) that overrides the ReplicationPolicy.offsetSyncsTopic and ReplicationPolicy.checkpointsTopic methods to use old topics if users still want to use the old internal topics, or by deleting the old internal topics and letting MM2 set up the new ones.
In this KIP, we are proposing to add
replication.policy.internal.topic.separator.enabled to allow customers to control this using configurations instead of providing a new implementation for both checkpoints and offset sync topics. The heartbeat topic isn't controlled by the separator at all.
whether or not to use replication.policy.separator to control internal topic names
- The KIP need to be released as part of the bugfix release to the last 3 versions
- The KIP need to be backported into 3.1 and 3.2 (any other version reached EOL post KIP-690).
Compatibility, Deprecation, and Migration Plan
Users who upgrade from Kafka < 3.1 and wish to disable controlling internal topics using the separator will need to set `replication.policy.internal.topic.separator.enabled` to false. Any users upgrading from > 3.1 don't need to do anything.
- Introduce a LegacyReplicationPolicy that inherits
DefaultReplicationPolicythat provide an override for ReplicationPolicy.offsetSyncsTopic and ReplicationPolicy.checkpointsTopic methods to use old topics and ignore the usage of the topic's separator. The advantage of this approach is avoiding introducing a new property to only fix a backward compatibility bug. However, this approach has a high risk of diversion between this new class and
DefaultReplicationPolicyat some point in the future. It also adds another class to maintain for MM2 without adding a big value for the connector besides fixing the backward compatibility bug.