Current state: Under Discussion
Discussion thread: Discussion Thread
We have had many situations where clients have been throttled due to exceeding their produce/fetch or request quota. Unfortunately, with possibly varying quotas assigned across clients, the only means of preforming capacity planning around throttling requires dynamically fetching the Zk quota config. While this can be done using the
AdminClient under KIP-546 it leads to disjoint data sources - requiring additional work to implement alerting and capacity planning for clients nearing their quota.
- Emit the quota (upper-bound) value tagged at the
client-id/usergranularity as an additional attribute to the
- Provide broker level configuration to enable the recording of the quota value.
- In the case the configuration is not set, no additional recording overhead should be added to the user request (i.e. the caching and creation of an additional sensor).
A new boolean configuration property "
client.quota.value.metric.enable" has been added as a static kafka.server.
KafkaConfig. As expected, this config must be set to "
true" in order to enabled emitting the quota metric.
An additional attribute is added to each of of the
|MBean Name||Current Attributes||Proposed New Attribute|
As mentioned above, the following boolean config will be added to
kafka.server.KafkaConfig with the property defined as "
To facilitate emitting the new metric, an additional
Sensor is needed that records the quota value. To fit into the mechanism used for caching client sensors, as well as obtaining the write lock for recording. This sensor is only created in the case where the quota is enabled.
Following this, on each
Produce API request the quota value is then recorded.
Compatibility, Deprecation, and Migration Plan
No impact will be seen on existing users and no migration plan will be necessary.
From a design perspective, recording quota values as a metric isn't ideal in the sense that they are largely static (as metrics go). In the case it would make more sense to recording the "available capacity" that a client has available at a given time as a rate. However, in order for the rate to have the correct value some additional work would be needed and it should be noted that as long as the sensor configs (window size and number of windows) this corresponding rate can be calculated after the fact easily.