You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As owner of Kafka cluster, but not the owner of clients, I am not capable of telling which topics/partitions are being actively read from (= have running clients that are sending fetch requests).

The existing metric kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic=topic aggregates both fetch requests made by clients and replicas.

So for partitions with replication factor > 1, even if the partition is no longer read from by clients, the above metric is still growing due to requests made by replicas.

Public Interfaces

The following new broker-side metrics are proposed:

  • kafka.server:type=BrokerTopicMetrics,name=ConsumerFetchRequestsPerSec
  • kafka.server:type=BrokerTopicMetrics,name=ConsumerFetchRequestsPerSec,topic=mytopic (one per topic)
  • kafka.server:type=BrokerTopicMetrics,name=ReplicationFetchRequestsPerSec
  • kafka.server:type=BrokerTopicMetrics,name=ReplicationFetchRequestsPerSec,topic=mytopic (one per topic)

Each of them is a meter similar to already-existing TotalFetchRequestsPerSec metric.

Proposed Changes

  1. Add new metrics mentioned above
  2. Update ConsumerFetchRequestsPerSec metric only when the fetch request is coming from client only (-1 != replicaId). Otherwise, update ReplicationFetchRequestsPerSec.

Compatibility, Deprecation, and Migration Plan

These are new metrics and there should be no impact on existing users.

Rejected Alternatives

  1. Do not provide new metrics - does not satisfy the need in environment where cluster owner has no knowledge about clients.
  2. Provide only one of metrics, as ConsumerFetchRequestsPerSec.count + ReplicationFetchRequestsPerSec.count == TotalFetchRequestsPerSec.count.
    `com.yammer.metrics.core.Meter` used underneath is exponentially weighted, so it would not be true for *Rate metrics.
  3. As proposed change, just change TotalFetchRequestsPerSec to be a gauge that adds values from ConsumerFetchRequestsPerSec and ReplicationFetchRequestsPerSec.
    Same as point 2.


  • No labels