Status
Current state: Under Discussion
Discussion thread: here
JIRA: here
PR: here
Motivation
With DefaultReplicationPolicy, the heartbeats topic name is hard-coded. Instead, this should be configurable, so users can avoid collisions with the "heartbeats" topics of other systems.
Public Interfaces
New MirrorConnectorConfig:
| Name | Type | Default | Description |
|---|---|---|---|
| deafult.replication.policy.heartbeats.topic.name | String | heartbeats | Name of the heartbeats topic |
Since it is not a significant change, I implemented it in advance. Draft PR: https://github.com/apache/kafka/pull/15200
Compatibility, Deprecation, and Migration Plan
- Since this feature does not change the default topic name, existing users won't be affected, unless they change the property intentionally.
- If we change the behavior in flight (e.g.: start the MM2 with default:
heartbeatstopic, stop MM2, then start it with a newcustom-heartbeatstopic) then the original topic becomes stale/dangling and the MM2 will use the new topic to track the heartbeats
Test Plan
I tested the change manually on 2 local clusters.
I wrote a couple of unit test and also wrote an integration test. However, I am not included the integration test in the PR because it seemed overkill for this small change, I don't want to introduce extra heavyweight tests, to slow down pipeline without a good reason (but, I am happy to include it if you think it is necessary)
Documentation Plan
We should introduce the new property in the 3.8.1 MirrorMaker Common Configs section
It should be highlighted in the documentation that this newly introduced property is not compatible with anything else but the DefaultReplicationPolicy
This solution was rejected in the KIP-690 as an alternative
The scope of this change is smaller than the KIP-690, as it only wants to make the heartbeats topic name configurable.
The configuration of the mirror maker is not straight forward with all that options. However, most of those properties are hidden from the user with a reasonable default value, so in my opinion it should not confuse the users, but it is there if someone want to modify it.
Renaming an internal topic should not require implementing a new replication policy class and attaching it to the classpath in my opinion.
Rejected Alternatives
- Divert from the DefaultReplicationPolicy and use a custom one.
This is a feasible solution proposed by Chris Egerton KIP-690, however, I think that it would be beneficial for the users to be able to configure the heartbeat topic name with the DefaultReplicationPolicy in an easy, no code way.