Status

Current state: "Voting“

Discussion thread: https://lists.apache.org/thread/npdqdmghngwcrh8664gnfmo1zdkjch4k

Vote thread: https://lists.apache.org/thread/glpx2fb2hglbqp1dod6pwjs2f02mnlq1

JIRA:   KAFKA-19566 - Getting issue details... STATUS

Motivation

In KAFKA-18225, we identified that ClientQuotaCallback#updateClusterMetadata is not supported in KRaft mode, even though it is supported in Zookeeper mode. In Kafka 4.0, we addressed this gap by implementing the missing functionality; however, some limitations still remain:

  1. In ClientQuotaCallback#updateClusterMetadata, we pass a Cluster object to this method. The Cluster is an immutable object that holds a lot of information, and a new Cluster is created every time there is a change in metadata. For large clusters with many partitions, this can result in significant memory pressure on Kafka. (see KAFKA-18239)
  2. Some information in the Cluster object is unnecessary or confusing. For instance, Cluster#controller refers to a random broker node in a KRaft cluster, which is retained for backward compatibility as it represents the Zookeeper controller. Furthermore, in Zookeeper mode, we parse the listener using the listener name from the request. However, in KRaft mode, there is no listener name in the updateClusterMetadata path. As a result, the callback may receive multiple partition info entries for each listener name. (See KAFKA-19122).

To address the above issues, we proposed an enhancement in KIP-1162: Redesign ClientQuotaCallback#updateClusterMetadata. While it introduces several API changes, considering that this method has remained unimplemented for years since the introduction of KRaft—and no users have raised concerns—it doesn’t seem worthwhile to invest significant effort into it. However, leaving a non-functional API in place is still awkward, and we should discourage its continued use. Therefore, we propose deprecating the updateClusterMetadata method, and remove it entirely in Kafka 5.0.

Public Interfaces

Deprecating ClientQuotaCallback#updateClusterMetadata

public interface ClientQuotaCallback extends Configurable {
	...
     /**
     * This callback is invoked whenever there are changes in the cluster metadata, such as 
     * brokers being added or removed, topics being created or deleted, or partition leadership updates.
     * This is useful if quota computation takes partitions into account.
     * Topics that are being deleted will not be included in `cluster`.
     *
	 * @deprecated since 4.2 and should not be used any longer.
     * @param cluster Cluster metadata including partitions and their leaders if known
     * @return true if quotas have changed and metric configs may need to be updated
     */
	@Deprecated(since = "4.2", forRemoval = true)
    boolean updateClusterMetadata(Cluster cluster);
	...
}

Proposed Changes

Deprecating ClientQuotaCallback#updateClusterMetadata. No alternative method will be introduced, because the method showed no demand during its three-year absence before 4.0, and its reintroduced form contains multiple pitfalls that would require significant effort to fix.

Compatibility, Deprecation, and Migration Plan

This KIP does not introduce any alternative methods, nor does it remove existing functionality—everything remains fully backward compatible. The ClientQuotaCallback#updateClusterMetadata method will be deprecated now and scheduled for removal in Kafka 5.0.

Test Plan

We will rely on the existing CustomQuotaCallbackTest.java to ensure the deprecated method still works.

Rejected Alternatives

  • While KIP-1162 proposes a redesign, this approach adds considerable engineering cost and complexity, yet brings little practical benefit, as the method has seen almost no demand and the existing implementation already exposes multiple pitfalls.


  • No labels