Status
Current state: Accepted
Discussion thread: here
JIRA: KAFKA-14740
Release: 3.5.0
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
MirrorSourceConnector registers a few metrics to track the mirroring of records from the source to the target cluster. Its metrics have the "target", "topic", "partition" tags. By default, with DefaultReplicationPolicy, the topic name contains the source cluster alias as its prefix, for example us-west.topic where us.west is the source cluster alias. However as it's part of the topic tag it's oftentimes not easily usable in monitoring tools. In addition, with IdentityReplicationPolicy the source cluster alias does not appear in the metrics at all.
With multiple mirroring routes, it can be helpful to aggregate metrics based on a specific source or target cluster to better gauge the usage and state of mirroring deployments.
Public Interfaces
1) A new configuration setting for MirrorSourceConnector:
Name: add.source.alias.to.metrics
Description: Deprecated. Whether to tag metrics with the source cluster alias. Metrics have the target, topic and partition tags. When this setting is enabled, it adds the source tag. This configuration will be removed in Kafka 4.0 and the default behavior will be to always have the source tag.
Type: boolean
Default: false
2) A new tag in MirrorSourceConnector metrics
If add.source.alias.to.metrics is set to true, a new tag, source, will be added to all metrics registered by MirrorSourceConnector. It will contain the source cluster alias. For example:
kafka.connect.mirror:type=MirrorSourceConnector,source=one,target=two,topic=newtopic,partition=0
The new tag will be added in the front of the list of tags so the tag order is the same as MirrorCheckpointConnector metrics.
Proposed Changes
Add add.source.alias.to.metrics to MirrorSourceConfig
Update MirrorSourceMetrics to have a new tag, source, and set it to the source cluster alias if add.source.alias.to.metrics is set to true.
Compatibility, Deprecation, and Migration Plan
Adding a tag to metrics may break queries in monitoring tools that serialize tags in metric names. For this reason this feature is gated by a configuration. The plan to is always add the new tag and delete add.source.alias.to.metrics in the next major release.
Test Plan
This will be tested using unit tests.
Rejected Alternatives
- Add the source tag at the end of the list of tags: Since this has similar compatibility implications than the proposed method, I favored having the same tag order in MirrorSourceConnector and MirrorCheckpointConnector.
- Create new metrics (with the source tag) while keeping the existing one: While this solves the compatibility issue, this is creating a lot of extra metrics since these are per topic/partition that is being mirror.