Current state: "Under Discussion"
Discussion thread: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
As owner of Kafka cluster, but not the owner of clients, I am not capable of telling which topics/partitions are being actively read from (= have running clients that are sending fetch requests).
The existing metric kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic=topic aggregates both fetch requests made by clients and replicas.
So for partitions with replication factor > 1, even if the partition is no longer read from by clients, the above metric is still growing due to requests made by replicas.
The following new broker-side metric is proposed:
- kafka.server:type=BrokerTopicMetrics,name=ConsumerFetchRequestsPerSec,topic=mytopic (one per topic)
It would be similar to the already-existing TotalFetchRequestsPerSec metric.
- Add new metric mentioned above
- Update ConsumerFetchRequestsPerSec metric only when the fetch request is coming from client only (-1 != replicaId).
Compatibility, Deprecation, and Migration Plan
These are new metrics and there should be no impact on existing users.
- Do not provide new metrics - does not satisfy the need in environment where cluster owner has no knowledge about clients.
- Provide two metrics: ConsumerFetchRequestsPerSec and ReplicationFetchRequestsPerSec:
- benefits: user is capable of accessing the desired metric without the need to find the values of others (what might be beneficial to e.g. jmxterm users)
- drawbacks: increased footprint (extra metric per topic & need to update it per request)