This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • KIP-293 Add new metrics for consumer/replication fetch requests
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Current »

Status

Current state: "Under Discussion"

Discussion thread: here

Voting threadhere

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As owner of Kafka cluster, but not the owner of clients, I am not capable of telling which topics/partitions are being actively read from (= have running clients that are sending fetch requests).

The existing metric kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic=topic aggregates both fetch requests made by clients and replicas.

So for partitions with replication factor > 1, even if the partition is no longer read from by clients, the above metric is still growing due to requests made by replicas.

Public Interfaces

The following new broker-side metric is proposed:

  • kafka.server:type=BrokerTopicMetrics,name=ConsumerFetchRequestsPerSec
  • kafka.server:type=BrokerTopicMetrics,name=ConsumerFetchRequestsPerSec,topic=mytopic (one per topic)

It would be similar to the already-existing TotalFetchRequestsPerSec metric.

Proposed Changes

  1. Add new metric mentioned above
  2. Update ConsumerFetchRequestsPerSec metric only when the fetch request is coming from client only (-1 != replicaId).

Compatibility, Deprecation, and Migration Plan

These are new metrics and there should be no impact on existing users.

Rejected Alternatives

  1. Do not provide new metrics - does not satisfy the need in environment where cluster owner has no knowledge about clients.
  2. Provide two metrics: ConsumerFetchRequestsPerSec and ReplicationFetchRequestsPerSec:
    1. benefits: user is capable of accessing the desired metric without the need to find the values of others (what might be beneficial to e.g. jmxterm users)
    2. drawbacks: increased footprint (extra metric per topic & need to update it per request)


  • No labels