Table of Contents |
---|
This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: Draft Accepted
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
As per the Tiered Storage feature introduced in KIP-405, we added several metrics related to reads(from) and writes(to) for remote storage. The naming convention that was followed is confusing to the users.
Public Interfaces
...
For eg. in regular Kafka, BytesIn means bytes written to the log, and BytesOut means bytes read from the log. But with tiered storage, the concepts are reversed.
- RemoteBytesIn means "Number of bytes read from remote storage per second"
- RemoteBytesOut means "Number of bytes written to remote storage per second"
We should rename the tiered storage related metrics to remove any ambiguity.
Public Interfaces
The following metrics will be renamed:
Original Name | Description | New Name |
kafka.server:type=BrokerTopicMetrics, name=RemoteBytesInPerSec, topic=([-.w]+) | The number of bytes read from remote storage per second |
for a topic | kafka.server:type=BrokerTopicMetrics, name=RemoteFetchBytesPerSec, topic=([-.w]+) |
kafka.server:type=BrokerTopicMetrics, name=RemoteReadRequestsPerSec, topic=([-.w]+) | The number of remote storage read requests per second |
for a topic | kafka.server:type=BrokerTopicMetrics, name=RemoteFetchRequestsPerSec, topic=([-.w]+) |
kafka.server:type=BrokerTopicMetrics, name=RemoteReadErrorPerSec, topic=([-.w]+) | The number of remote storage read errors per second |
for a topic | kafka.server:type=BrokerTopicMetrics, name=RemoteFetchErrorsPerSec, topic=([-.w]+) |
kafka.server:type=BrokerTopicMetrics, name=RemoteBytesOutPerSec, topic=([-.w]+) | The number of bytes copied to remote storage per second |
for a topic | kafka.server:type=BrokerTopicMetrics, name=RemoteCopyBytesPerSec, topic=([-.w]+) | |
kafka.server:type=BrokerTopicMetrics, name=RemoteWriteRequestsPerSec, topic=([-.w]+) | The number of remote storage write requests per second for a topic | kafka.server:type=BrokerTopicMetrics, name=RemoteCopyRequestsPerSec, topic=([-.w]+) |
kafka.server:type=BrokerTopicMetrics, name=RemoteWriteErrorPerSec, topic=([-.w]+) | The number of remote storage write errors per second |
Name
Description
for a topic | kafka.server:type=BrokerTopicMetrics, name=RemoteCopyErrorsPerSec, topic=([-.w]+) |
The following metrics will be added.
kafka.server:type= |
BrokerTopicMetrics, name= |
RemoteBytesInPerSec |
Total number of |
org.apache.kafka.storage.internals.log:type=RemoteIndexCache, name=MissCount
The number of cache misses for the remote index cache.
org.apache.kafka.storage.internals.log:type=RemoteIndexCache, name=EvictionCount
The number of entries evicted from the remote index cache.
org.apache.kafka.storage.internals.log:type=RemoteIndexCache, name=LoadSuccessCount
The number of successful cache loads for the remote index cache.
org.apache.kafka.storage.internals.log:type=RemoteIndexCache, name=LoadFailureCount
The number of failed cache loads for the remote index cache.
org.apache.kafka.storage.internals.log:type=RemoteIndexCache, name=TotalLoadTime
Total load time (success and failure) for the remote index cache.
org.apache.kafka.storage.internals.log:type=RemoteIndexCache, name=EvictionWeight
The sum of weights of entries evicted from the remote index cache.
Compatibility, Deprecation, and Migration Plan
Test Plan
bytes read from remote storage per second on a broker | kafka.server:type=BrokerTopicMetrics, name=RemoteFetchBytesPerSec | |
kafka.server:type=BrokerTopicMetrics, name=RemoteReadRequestsPerSec | Total number of remote storage read requests per second on a broker | kafka.server:type=BrokerTopicMetrics, name=RemoteFetchRequestsPerSec |
kafka.server:type=BrokerTopicMetrics, name=RemoteReadErrorPerSec | Total number of remote storage read errors per second on a broker | kafka.server:type=BrokerTopicMetrics, name=RemoteFetchErrorsPerSec |
kafka.server:type=BrokerTopicMetrics, name=RemoteBytesOutPerSec | Total number of bytes copied to remote storage per second on a broker | kafka.server:type=BrokerTopicMetrics, name=RemoteCopyBytesPerSec |
kafka.server:type=BrokerTopicMetrics, name=RemoteWriteRequestsPerSec | Total number of remote storage write requests per second on a broker | kafka.server:type=BrokerTopicMetrics, name=RemoteCopyRequestsPerSec |
kafka.server:type=BrokerTopicMetrics, name=RemoteWriteErrorPerSec | Total number of remote storage write errors per second on a broker | kafka.server:type=BrokerTopicMetrics, name=RemoteCopyErrorsPerSec |
Compatibility, Deprecation, and Migration Plan
The metrics that are being renamed were only added as part of the same release. Hence, we do not need to maintain backward compatibility for the renamed metrics.
Test Plan
Existing Unit Tests for the metrics being renamed.
Rejected Alternatives
None
...