DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Status
Current state: Under Discussion
Discussion thread: here
JIRA: KAFKA-20725
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
KIPs like KIP-392: Allow consumers to fetch from closest replica, KIP-1150: Diskless Topics, and KIP-1123: Rack-aware partitioning for Kafka Producer, apart from other goals, aim to reduce inter-rack traffic. This brings potential benefits such as lower latency and costs, especially in cloud environments where inter-rack (i.e. inter-availability zone) traffic is often billed.
While these initiatives succeed in this goal, the aspect of observability could be improved. Knowing how much traffic goes within a rack and how much crosses rack borders would be valuable to operators, for example, to highlight misconfiguration. This KIP proposes to close this gap with additional metrics.
Public Interfaces
New metrics
Producer
In the MBean kafka.producer:type=producer-topic-metrics,client-id=<id>,topic=<topic>:
| Metric | Description | Existing counterpart |
|---|---|---|
byte-in-rack-rate | The average number of bytes sent in the same rack per second for a topic. | byte-rate |
byte-in-rack-total | The total number of bytes sent in the same rack for a topic. |
|
Consumer
In the MBean kafka.consumer:type=consumer-fetch-manager-metrics,client-id=<id>:
| Metric | Description | Existing counterpart |
|---|---|---|
bytes-consumed-in-rack-rate | The average number of bytes consumed per second from brokers in the same rack as the consumer. | bytes-consumed-rate |
bytes-consumed-in-rack-total | The total number of bytes consumed from brokers in the same rack as the consumer. | bytes-consumed-total |
In the MBean kafka.consumer:type=consumer-fetch-manager-metrics,client-id=<id>,topic=<topic>:
| Metric | Description | Existing counterpart |
|---|---|---|
bytes-consumed-in-rack-rate | The average number of bytes consumed per second for a topic from brokers in the same rack as the consumer. | bytes-consumed-rate |
bytes-consumed-in-rack-total | The total number of bytes consumed for a topic from brokers in the same rack as the consumer. | bytes-consumed-total |
Broker
In the MBean kafka.server:type=BrokerTopicMetrics,name=ReplicationBytesOutInRackPerSec:
| Metric | Description | Existing counterpart |
|---|---|---|
ReplicationBytesOutInRackPerSec | Bytes of replication traffic sent by the broker. | ReplicationBytesOutPerSec |
In the MBean kafka.server:type=BrokerTopicMetrics,name=ReplicationBytesInInRackPerSec:
| Metric | Description | Existing counterpart |
|---|---|---|
ReplicationBytesInInRackPerSec | Bytes of replication traffic received by the broker. | ReplicationBytesInPerSec |
In the MBean kafka.server:type=BrokerTopicMetrics,name=ReassignmentBytesOutInRackPerSec:
| Metric | Description | Existing counterpart |
|---|---|---|
ReassignmentBytesOutInRackPerSec | Bytes of reassignment traffic sent by the broker. | ReassignmentBytesOutPerSec |
In the MBean kafka.server:type=BrokerTopicMetrics,name=ReassignmentBytesInInRackPerSec:
| Metric | Description | Existing counterpart |
|---|---|---|
ReassignmentBytesInInRackPerSec | Bytes of reassignment traffic received by the broker. | ReassignmentBytesInPerSec |
Proposed Changes
The KIP proposes to complement certain existing metrics with in-rack counterparts (see Public Interfaces). The in-rack metric values will be less or equal to the corresponding "full" metrics, i.e. the will represent a fraction of traffic that went in-rack.
Compatibility, Deprecation, and Migration Plan
No impact is expected for existing users, as no existing interfaces or behavior is being changed.
Test Plan
The new metrics will be tested the same way as their existing counterparts, preferably by extending the same test classes.
Rejected Alternatives
None.