Status

Current state: Voting

Discussion thread: here 

Vote thread: here 

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In the current Kafka ecosystem, we have the kafka-broker-api-versions.sh tool to retrieve the supported API versions from each broker. This is an extremely useful feature that provides administrators and users with a clear view of the broker's RPC version.

However, we lack a similar direct observability mechanism for the controller. The Controller's version and its supported APIs determine the capabilities of the cluster. Currently, for an administrator to confirm the Controller's capabilities, they don't have any method to retrieve controller's RPC version.

To enhance the transparency and observability of Kafka cluster administration, the controller should like brokers, support a public interface to query its supported API versions. This would provide the following benefits:

  1. Verify Upgrades: After a rolling upgrade of controllers, administrators would have a definitive command to verify that the new controller is active and provides the expected RPC version.

  2. Feature Discovery: Automation tools and administrators could first query the controller to determine if a specific feature is supported before attempting to use it, thus avoiding failures due to version incompatibility.

  3. Simplify Tooling Development: It provides a standard and reliable way for external management tools to obtain the capability set of the controller.

Public Interfaces

  • A new argument, --bootstrap-controller, is introduced to accept a comma-separated list of host and port pairs for connecting to controller nodes.

Proposed Changes

After this KIP, the legacy admin client will be removed, and the current AdminClient implementation will be used exclusively to reduce maintenance overhead.

We will introduce InternalDescribeFeatureResult, which extends DescribeFeatureResult, and leverage the DescribeFeatures API to collect API version information from all brokers.

This proposal also introduces a new argument that performs the following steps when --bootstrap-controller is used:

  1. Send DescribeCluster RPC to the controller:
    When the admin client is initialized with --bootstrap-controller, it will first issue a DescribeCluster request to the controller node to retrieve the metadata of the cluster, including all controller endpoints.

  2. Retrieve and query all controllers:
    After obtaining the list of controller endpoints from the DescribeClusterResponse, the admin client will send an ApiVersionsRequest to each controller.

  3. Collect ApiVersionsResponse from controllers:
    Each controller will respond with its supported API versions. The admin client will aggregate these responses.

This change ensures that when the admin tool is bootstrapped directly from the controller, it can accurately discover and communicate with all controllers in the quorum.

Compatibility, Deprecation, and Migration Plan

  • This change only adds new arguments and behaviour to kafka-broker-api-version.sh, maintaining full backward compatibility.

Test Plan

  • Integration test and unit test should cover this feature.

Rejected Alternatives

  • We can use --bootstrap to retrieve all controller and broker, the additional argument is not necessary but we need to identity what type the node is and inconsistent with other kafka tools.
  • No labels