This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: Accepted
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
MirrorSourceConnector allows replicating topics between Kafka clusters. Users have a number of configuration options to tweak the behavior of the replication, e.g. topic filter, consumer and producer configs. One particular behavior that is hard-coded is the replication of the heartbeats topic. In most scenarios, this is not an issue, but in some use-cases, users may want to opt-out of the replication of the heartbeats topic(s). This KIP proposes a configuration which allows users to turn heartbeats replication off.
One such use-case is if users want to use multiple MirrorSourceConnectors in their replication. It is possible to require different topics to be replicated with different configurations between the same 2 clusters. In this case, users can use multiple MirrorSourceConnector instances, and use the topics filter to match different groups of topics. This way, the different MirrorSourceConnector instances will replicate different topics, and can use differently tuned configurations.
For example, if there are source topics using gzip compression (all prefixed with gzip-), and there are topics using lz4 compression (all prefixed with lz4-), users might want to keep the exact compression types without putting the load on the target brokers, and performing the compression on the client side. This is not possible with a single MirrorSourceConnector, as it can only support one set of producer configurations. But using 2 Connector instances, users can specify the topic filters in a mutually exclusive way, and perform the replication with different producer configs:
mirror-gzip-connector:
topics: gzip-*
producer.compression.type: gzip
mirror-lz4-connector:
topics: lz4-*
producer.compression.type: lz4
The only issue left is if there is a heartbeats topic in the source cluster. Heartbeats topics are always replicated, so even if the topic filter is properly configured to be mutually exclusive, the heartbeats topic itself will be picked up by both Connectors. By providing a property to configure this behavior, users can explicitly turn off the heartbeats replication in their MirrorSourceConnector instances, only leaving one of them to replicate the topic.
Public Interfaces
New property in MirrorSourceConfig:
- heartbeats.replication.enabled (default: true): When enabled, heartbeats topics are always replicated, regardless of the topic filter settings.
Proposed Changes
The logic inside MirrorSourceConnector (shouldReplicate method) when choosing topics to replicate will be updated by also considering the heartbeats.replication.enabled configuration.
Compatibility, Deprecation, and Migration Plan
- The new configuration is fully backward compatible, as its default value will keep the old behavior. Existing users do not need to perform any kind of migration.
Test Plan
Since this is a fairly small change, unit tests should be sufficient.
Rejected Alternatives
- If KIP-1016 passes, users can intentionally "misconfigure" the heartbeats topic name, setting it to a non-existent topic. This can achieve the same end result. This is a counter-intuitive solution to the proposed problem.