Status

Current state: Accepted

Discussion thread: here

Voting thread: here

JIRA: KAFKA-19149

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Both MirrorSourceConnector and MirrorCheckpointConnector emit metrics. However there was no mechanisms for connectors to register metrics when MirrorMaker was created, so they uses a "workaround" to register their metrics. Each connector creates its own Metrics instance and creates new instances of the MetricsReporter that are set via the metric.reporters configuration. MirrorHeartbeatConnector does not have any metrics.

Since Kafka 4.1, we have KIP-877 which provides a way for connectors to register metrics. We should update the MirrorMaker connectors to use that feature. This will also serve as an example for 3rd party connectors by demonstrating how to use KIP-877.

Public Interfaces

1) A new configuration setting for MirrorSourceConnector:

Name: metric.names.formats
Description: Deprecated. The formats in which metrics are emitted. Valid values are legacy and new. When set to legacy, the metrics have the following name "kafka.connect.mirror:type=MirrorSourceConnector,source=<SOURCE>,target=<TARGET>,topic=<TOPIC>,partition=<PARTITION>". When set to new, the metrics have the following name "kafka.connect:type=plugins,connector=<CONNECTOR>,task=<TASK>,source=<SOURCE>,target=<TARGET>,topic=<TOPIC>,partition=<PARTITION>". When set to "legacy,new" the connector will emit metrics with both names, this can be useful when migrating to the new format but it doubles the amount of metrics that are emitted.
In Kafka 5.0 the legacy format and this configuration will be removed and the metrics will always use the new names.
Type: List
Default: legacy

2) A new configuration setting for MirrorCheckpointConnector:

Name: metric.names.formats

Deprecated. The formats in which metrics are emitted. Valid values are legacy and new. When set to legacy, the metrics have the following name "kafka.connect.mirror:type=MirrorCheckpointConnector,source=<SOURCE>,target=<TARGET>,group=<GROUP>,topic=<TOPIC>,partition=<PARTITION>". When set to new, the metrics have the following name "kafka.connect:type=plugins,connector=<CONNECTOR>,task=<TASK>,source=<SOURCE>,target=<TARGET>,group=<GROUP>,topic=<TOPIC>,partition=<PARTITION>". When set to "legacy,new" the connector will emit metrics with both names, this can be useful when migrating to the new format but it doubles the amount of metrics that are emitted.
In Kafka 5.0 the legacy format and this configuration will be removed and the metrics will always use the new names.
Type: List
Default: legacy

Proposed Changes

This KIP proposes updating MirrorSourceConnector and MirrorCheckpointConnector to use KIP-877 to register their metrics. However this will alter the metric names, so new configurations, metric.names.formats, are introduced to enable the migration.

From Kafka 5.0 metrics will always have the new names. In the meantime, the metric.names.formats configurations will default to legacy to maintain the original names. The configurations are deprecated immediately and will be removed in Kafka 5.0. Users wanting to migrate to the new metrics before Kafka 5.0 can set the configurations to new.

Legacy names:
- kafka.connect.mirror:type=MirrorSourceConnector,source=<SOURCE>,target=<TARGET>,topic=<TOPIC>,partition=<PARTITION>
- kafka.connect.mirror:type=MirrorCheckpointConnector,source=<SOURCE>,target=<TARGET>,group=<GROUP>,topic=<TOPIC>,partition=<PARTITION>

New/KIP-877 names:
- kafka.connect:type=plugins,connector=<CONNECTOR>,task=<TASK>,source=<SOURCE>,target=<TARGET>,topic=<TOPIC>,partition=<PARTITION>
- kafka.connect:type=plugins,connector=<CONNECTOR>,task=<TASK>,source=<SOURCE>,target=<TARGET>,group=<GROUP>,topic=<TOPIC>,partition=<PARTITION>

Where:

• <CONNECTOR> is the name of the created connector, not the class name
• <TASK> is the task Id, for example 0
• <SOURCE> is the source cluster alias
• <TARGET> is the target cluster alias
• <TOPIC> is the topic name
• <PARTITION> is the partition Id
• <GROUP> is the consumer group Id

If metric.names.formats is not set or contains legacy, a warning will be logged informing that the metrics names will changes from Kafka 5.0 and suggesting to update to the new names.

Compatibility, Deprecation, and Migration Plan

By default the new configurations will be set to legacy so it does not change the current behavior. The configurations will be marked as deprecated immediately.

In the next major version, Kafka 5.0:

• the metric.names.formats configurations will be removed
• the MirrorMaker connectors will only emit metrics via KIP-877 with the new names

From Kafka 5.0 or by setting metric.names.formats to new, users monitoring these metrics will need to adjust their monitoring systems to use the new metric names.

Test Plan

This feature will be tested using unit and integration tests.

Rejected Alternatives

• Use a boolean configuration to either use the legacy or new metrics names: When migrating between formats, in case the monitoring systems are not updated correctly with the new metric names, this risks completely losing observability. By allowing users to have both metrics, they can ensure continuity in their monitoring.