Status

Current state: [Under Discussion]

Discussion thread: here

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka has been gradually moving coordinator metrics from Yammer Metrics to Kafka Metrics. As part of KAFKA-14519, `GroupCoordinatorMetrics` was introduced for the new coordinator metrics work. KAFKA-15870 then moved the newly added group coordinator metrics from Yammer Metrics to Kafka Metrics.

However, this migration is still incomplete. Several group coordinator metrics are already available through Kafka Metrics, including `group-count` metrics for different group
protocols. But some important classic-group metrics are still exposed only through Yammer Metrics.

This KIP completes that migration for `GroupCoordinatorMetrics`. It deprecates the remaining Yammer-based metrics and adds equivalent Kafka Metrics coverage. This gives the group coordinator a clearer and more consistent metrics interface while preserving compatibility during the deprecation period.

Public Interfaces

1. Yammer-Based Metrics to be Deprecated

The following Yammer metrics will be marked as `@Deprecated`:


Yammer Metric Name

Replacement Kafka Metric
Type
Description
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroups
kafka.server:type=group-coordinator-metrics,protocol=classic
Gauge<Long>
The total number of groups using the classic rebalance protocol
kafka.coordinator.group:type=GroupMetadataManager,name=NumOffsets
kafka.server:type=group-coordinator-metrics
Gauge<Long>
The number of offsets currently retained for Classic, Consumer, and Streams Groups
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsPreparingRebalance
kafka.server:type=group-coordinator-metrics,state=PreparingRebalance
Gauge<Long>
The number of classic groups in PreparingRebalance state
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsCompletingRebalance
kafka.server:type=group-coordinator-metrics,state=CompletingRebalance
Gauge<Long>
The number of classic groups in CompletingRebalance state
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsStable
kafka.server:type=group-coordinator-metrics,state=Stable
Gauge<Long>
The number of classic groups in Stable state
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsDead
kafka.server:type=group-coordinator-metrics,state=Dead
Gauge<Long>
The number of classic groups in Dead state
kafka.coordinator.group:type=GroupMetadataManager,name=NumGroupsEmpty
kafka.server:type=group-coordinator-metrics,state=Empty
Gauge<Long>
The number of classic groups in Empty state


2. New Kafka metrics are to be added.

The following Kafka metrics will be added to provide equivalent functionality:

Metric Name

Tags
Attribute
Type
Status
kafka.server:type=group-coordinator-metrics,protocol=classic
protocol=classic

group-count

Gauge<Long>
Existing
kafka.server:type=group-coordinator-metrics
none
group-offset-count
Gauge<Long>
New
kafka.server:type=group-coordinator-metrics,state=PreparingRebalance

state = PreparingRebalance

classic-group-count

 

Gauge<Long>

 

New
kafka.server:type=group-coordinator-metrics,state=CompletingRebalance

state = CompletingRebalance

classic-group-count

 

Gauge<Long>

 

New
kafka.server:type=group-coordinator-metrics,state=Stable

state = Stable

classic-group-count

 

Gauge<Long>

 

New
kafka.server:type=group-coordinator-metrics,state=Dead

state = Dead

classic-group-count

 

Gauge<Long>

 

New 
kafka.server:type=group-coordinator-metrics,state=Empty

state = Empty

classic-group-count

 

Gauge<Long>

 

New

Proposed Changes

  1. New Kafka Metrics

  The new metrics listed in Public Interfaces will be registered as tagged gauges in GroupCoordinatorMetrics , following the same pattern already used for existing metrics like
  group-count{protocol=...}  and consumer-group-count{state=...} .

  Note: The existing offset-commit-count metric is a cumulative counter (meter) of offset commit operations. The new group-offset-count is a gauge reflecting the number of offsets currently
  retained by the group coordinator. These serve different purposes and both will coexist.

  2. Deprecation of Yammer Metrics

  The corresponding Yammer metrics will be annotated as @Deprecated  but will remain functional during the deprecation period. Each deprecated metric has a direct Kafka Metrics
  replacement as described in the Public Interfaces section.

  3. Internal Changes

  GroupCoordinatorMetrics  will be updated to register the new gauges, expose them through the existing metrics framework, and remove them during coordinator shutdown.


Compatibility, Deprecation, and Migration Plan

Kafka 4.4 (Deprecation Period)

As described in the Proposed Changes, the Yammer metrics in `GroupCoordinatorMetrics` will be deprecated, but they will remain available during the deprecation period.

Kafka 5.0 (Removal)

The deprecated Yammer metrics in `GroupCoordinatorMetrics` will be removed.

At that point, the group coordinator metrics surface will be exposed through Kafka Metrics only

Test Plan

N/A

Rejected Alternatives

N/A

  • No labels