Status
Current state: Under Discussion
Discussion thread: here
JIRA: here
PR: here
Motivation
With DefaultReplicationPolicy, the heartbeats topic name is hard-coded. Instead, this should be configurable, so users can avoid collisions with the "heartbeats" topics of other systems.
Public Interfaces
New MirrorConnectorConfig:
| Name | Type | Default | Description |
|---|---|---|---|
| default.replication.policy.heartbeats.topic.name | String | heartbeats | Name of the heartbeats topic |
Since it is not a significant change, I implemented it in advance. Draft PR: https://github.com/apache/kafka/pull/15200
Compatibility, Deprecation, and Migration Plan
- Since this feature does not change the default topic name, existing users won't be affected, unless they change the property intentionally.
- If we change the behavior in flight (e.g.: start the MM2 with default:
heartbeatstopic, stop MM2, then start it with a newcustom-heartbeatstopic) then the original topic becomes stale/dangling and the MM2 will use the new topic to track the heartbeats
Test Plan
I tested the change manually on 2 local clusters.
I wrote a couple of unit test and also wrote an integration test. However, I am not included the integration test in the PR because it seemed overkill for this small change, I don't want to introduce extra heavyweight tests, to slow down pipeline without a good reason (but, I am happy to include it if you think it is necessary)
Documentation Plan
We should introduce the new property in the 3.8.1 MirrorMaker Common Configs section
Rejected Alternatives
- Divert from the DefaultReplicationPolicy and use a custom one.
This is a feasible solution proposed by Chris Egerton KIP-690, however, I think that it would be beneficial for the users to be able to configure the heartbeat topic name with the DefaultReplicationPolicy in an easy, no code way.