DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Status
Current state: [Accepted]
Vote thread: here
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Kafka has been gradually moving coordinator metrics from Yammer Metrics to Kafka Metrics. As part of KAFKA-14519, `GroupCoordinatorMetrics` was introduced for the new coordinator metrics work. KAFKA-15870 then moved the newly added group coordinator metrics from Yammer Metrics to Kafka Metrics.
However, this migration is still incomplete. Several group coordinator metrics are already available through Kafka Metrics, including `group-count` metrics for different group
protocols. But some important classic-group metrics are still exposed only through Yammer Metrics.
This KIP completes that migration for `GroupCoordinatorMetrics`. It deprecates the remaining Yammer-based metrics and adds equivalent Kafka Metrics coverage. This gives the group coordinator a clearer and more consistent metrics interface while preserving compatibility during the deprecation period.
Public Interfaces
1. Yammer-Based Metrics to be Deprecated
The following Yammer metrics will be marked as `@Deprecated`:
Yammer Metric Name | Replacement Kafka Metric | Type | Description |
|---|---|---|---|
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroups | kafka.server:type=group-coordinator-metrics,name=group-count,protocol=classic | Gauge<Long> | The total number of groups using the classic rebalance protocol |
kafka.coordinator.group:type=GroupMetadataManager,name=NumOffsets | kafka.server:type=group-coordinator-metrics,name=offset-count | Gauge<Long> | The number of offsets currently retained for Classic, Consumer, and Streams Groups |
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsPreparingRebalance | kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=PreparingRebalance | Gauge<Long> | The number of classic groups in PreparingRebalance state |
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsCompletingRebalance | kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=CompletingRebalance | Gauge<Long> | The number of classic groups in CompletingRebalance state |
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsStable | kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=Stable | Gauge<Long> | The number of classic groups in Stable state |
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsDead | kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=Dead | Gauge<Long> | The number of classic groups in Dead state |
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsEmpty | kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=Empty | Gauge<Long> | The number of classic groups in Empty state |
2. New Kafka metrics are to be added.
The following Kafka metrics will be added to provide equivalent functionality:
Metric Name | Tags | Attribute | Type | Status |
|---|---|---|---|---|
kafka.server:type=group-coordinator-metrics,name=group-count,protocol=classic | protocol=classic | group-count | Gauge<Long> | Existing |
kafka.server:type=group-coordinator-metrics,name=offset-count | none | offset-count | Gauge<Long> | New |
kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=PreparingRebalance | state = PreparingRebalance | classic-group-count
| Gauge<Long>
| New |
kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=CompletingRebalance | state = CompletingRebalance | classic-group-count
| Gauge<Long>
| New |
kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=Stable | state = Stable | classic-group-count
| Gauge<Long>
| New |
kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=Dead | state = Dead | classic-group-count
| Gauge<Long>
| New |
kafka.server:type=group-coordinator-metrics,name=classic-group-count,state=Empty | state = Empty | classic-group-count
| Gauge<Long>
| New |
Proposed Changes
1. New Kafka Metrics
The new metrics listed in Public Interfaces will be registered as tagged gauges in GroupCoordinatorMetrics , following the same pattern already used for existing metrics like
group-count{protocol=...} and consumer-group-count{state=...} .
Note: The existing offset-commit-count metric is a cumulative counter (meter) of offset commit operations. The new offset-count is a gauge reflecting the number of offsets currently
retained by the group coordinator. These serve different purposes and both will coexist.
2. Deprecation of Yammer Metrics
The corresponding Yammer metrics will be annotated as @Deprecated but will remain functional during the deprecation period. Each deprecated metric has a direct Kafka Metrics
replacement as described in the Public Interfaces section.
3. Internal Changes
GroupCoordinatorMetrics will be updated to register the new gauges, expose them through the existing metrics framework, and remove them during coordinator shutdown.
Compatibility, Deprecation, and Migration PlanKafka 4.4 (Deprecation Period)
As described in the Proposed Changes, the Yammer metrics in `GroupCoordinatorMetrics` will be deprecated, but they will remain available during the deprecation period.
Kafka 5.0 (Removal)
The deprecated Yammer metrics in `GroupCoordinatorMetrics` will be removed.
At that point, the group coordinator metrics surface will be exposed through Kafka Metrics only
Test Plan
This change will be covered by unit tests in `GroupCoordinatorMetricsTest` and `GroupCoordinatorMetricsShardTest`.
The tests will verify that the Kafka Metrics replacements listed in the Public Interfaces section are registered with the expected metric names and tags, including:
1. group-count with protocol=classic
2. offset-count with no tags
3. classic-group-count with each classic group state tag:
- state=PreparingRebalance
- state=CompletingRebalance
- state=Stable
- state=Dead
- state=Empty
The tests will also verify that:
1. The deprecated Yammer metrics remain registered and functional during the deprecation period.
2. The new Kafka Metrics report values equivalent to the corresponding deprecated Yammer metrics listed in the Public Interfaces section.
3. Metric values are aggregated correctly across active coordinator shards.
4. The `offset-count` gauge reflects committed timeline state, matching the existing shard counter behavior.
5. Both Kafka Metrics and Yammer metrics are removed when `GroupCoordinatorMetrics` is closed.
Rejected Alternatives
N/A