Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-19168 - Getting issue details... STATUS

Motivation

Each MirrorMaker task creates its own Metrics instance without passing any tags. While most metrics created by tasks won't collide because they have tags for a specific topic partition or group, the kafka_metrics_count, which is always created, always collides.

So, for example, if you run two MirrorSourceConnector tasks, the "kafka.connect.mirror:type=kafka-metrics-count" metric is shown as 13, even though there are actually 26 metrics, 13 for each task.

Furthermore, "kafka.connect.mirror:type=kafka-metrics-count" metric is created without any tags, unlike other task metrics, leading to collisions. Even with the connector name and task index added, it was still insufficient. To ensure kafka-metrics-count is displayed correctly, this KIP changes the source and target as default tags when creating the Metrics instance. Otherwise, using only the connector name and task index can still lead to collisions; for example, if there are two clusters, A and B: Cluster A → B creates its own Metrics instance, and Cluster B → A creates its own as well.

This KIP proposes providing the tags (connector name and task index), and setting the source and target as default tags when creating the Metrics instance for each task, so that MirrorMaker task metrics can be reported per task without collisions.

Public Interfaces

1) A new configuration setting for MirrorSourceConnector and MirrorCheckpointConnector in MirrorConnectorConfig:

Name: add.connector.and.task.to.metrics

Description: Deprecated. Whether to tag metrics with the connector name and task index, and whether source and target tags are added to kafka-metrics-count. This configuration will be removed in Kafka 5.0, at which point the default behavior will be to always use the behavior proposed by this KIP.

Type: boolean

Default: false

2) New tags in MirrorSourceConnector metrics and MirrorCheckpointConnector metrics:

If add.connector.and.task.to.metrics is set to true, new tags, connector name and task index, will be added to all metrics registered by MirrorSourceConnector and MirrorCheckpointConnector, and source and target tags will be added to kafka-metrics-count. For example:

• kafka.connect.mirror:type=kafka-metrics-count,connector=my-connector,task=0,source=A,target=B
• kafka.connect.mirror:type=MirrorSourceConnector,connector=my-connector,task=0,source=A,target=B,topic=newtopic,partition=0
• kafka.connect.mirror:type=MirrorCheckpointConnector,connector=my-connector,task=0,source=A,target=B,topic=newtopic,partition=0

The new tags will be added at the front of the list of tags so the tag order is the same as KIP-877.

List of all affected metrics:

For kafka.connect.mirror:type=MirrorSourceConnector,

• record-count
• record-age-ms
• record-age-ms-min
• record-age-ms-max
• record-age-ms-avg
• replication-latency-ms
• replication-latency-ms-min
• replication-latency-ms-max
• replication-latency-ms-avg
• byte-rate

For kafka.connect.mirror:type=MirrorCheckpointConnector,

• checkpoint-latency-ms
• checkpoint-latency-ms-min
• checkpoint-latency-ms-max
• checkpoint-latency-ms-avg

Proposed Changes

• Add add.connector.and.task.to.metrics to MirrorConnectorConfig
• Add the new tags, connector name and task index, to MirrorSourceMetrics and MirrorCheckpointMetrics, and set source and target as default tags when creating the Metrics instance to ensure correct display, if add.connector.and.task.to.metrics is set to true. In other words, these four tags will be passed as default tags when creating the Metrics instance.
• Set the name of the connector (connector) and the task ID (task) to be the same as defined in KIP-877 for Tasks.

Compatibility, Deprecation, and Migration Plan

Adding new tags to metrics can break queries in monitoring tools. For this reason, we have a configuration that enables or disables this change. The plan is to always add the new tags and delete the add.connector.and.task.to.metrics configuration in the next major release, Kafka 5.0.

Test Plan

Unit tests will be added to verify the correct tagging behavior for the metrics.

Rejected Alternatives

1. Create new metrics with new tags while keeping the existing one

Reason for Rejection: While this solves the compatibility issue without requiring a new config to control enabling, it results in a lot of extra metrics since these are per topic/partition that is being mirrored. This design decision and trade-off take reference from: KIP-911: Add source tag to MirrorSourceConnector metrics.

2. Also set topic and partition as default tags when creating the Metrics instance

Reason for Rejection:  The same metric instance could be used for several topics and partitions, so setting them as default tags is neither reasonable nor ideal.

3. Add the new tags at the end of the list of tags

Reason for Rejection:  Keeping the tag order consistent with what is defined in KIP-877 is preferable for clarity and consistency.