Status

Current state: Accepted

Discussion thread: here 

Vote thread: here 

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In the current Kafka ecosystem, we have the kafka-broker-api-versions.sh tool to retrieve the supported API versions from each broker. This is an extremely useful feature that provides administrators and users with a clear view of the broker's RPC version.

However, we lack a similar direct observability mechanism for the controller. The Controller's version and its supported APIs determine the capabilities of the cluster. Currently, for an administrator to confirm the Controller's capabilities, they don't have any method to retrieve controller's RPC version.

To enhance the transparency and observability of Kafka cluster administration, the controller should like brokers, support a public interface to query its supported API versions. This would provide the following benefits:

  1. Verify Upgrades: After a rolling upgrade of controllers, administrators would have a definitive command to verify that the new controller is active and provides the expected RPC version.

  2. Feature Discovery: Automation tools and administrators could first query the controller to determine if a specific feature is supported before attempting to use it, thus avoiding failures due to version incompatibility.

  3. Simplify Tooling Development: It provides a standard and reliable way for external management tools to obtain the capability set of the controller.

However, extending kafka-broker-api-versions.sh directly to support controllers introduces a conceptual mismatch. API version inspection is no longer a broker-specific operation in KRaft mode, where controllers are independently addressable cluster nodes with their own RPC interfaces and administrative responsibilities.

As cluster capability inspection continues to evolve, API version discovery should be treated as a cluster-level administrative operation rather than functionality tied specifically to brokers.

Therefore, this KIP proposes moving API version inspection functionality into the kafka-cluster.sh tool by api-versions subcommand, this provides a unified and extensible administrative interface capable of inspecting both broker and controller API capabilities while aligning with Kafka's broader cluster tooling direction.

Public Interfaces

  • A new api-versions subcommand is added to kafka-cluster.sh to retrieve supported API versions from both brokers and controllers. The subcommand supports:
    • --bootstrap-server for querying broker API versions
    • --bootstrap-controller for querying controller API versions

    The --bootstrap-controller argument accepts a comma-separated list of host and port pairs for connecting to controller nodes.

  •  For each node returned by the cluster, one block is printed:

      <host>:<port> (id: <nodeId> rack: <rack> isFenced: <bool>) -> (
      <ApiName>(<apiKey>): <minVersion> to <maxVersion> [usable: <usableVersion>],
      <ApiName>(<apiKey>): UNSUPPORTED,
      ...
  )

Proposed Changes

1. Tool Integration and Redirection

To streamline the Kafka command-line toolsuite, the functionality of kafka-broker-api-versions.sh will be integrated into the kafka-cluster.sh tool as a new subcommand: api-versions.

  • Legacy Tool Transformation: A deprecation warning will be issued to encourage users to transition to the new command.

2. Technical Implementation

We will introduce InternalDescribeFeatureResult, which extends DescribeFeatureResult, and leverage the DescribeFeatures API to collect API version information from all brokers.

This proposal also introduces a new argument that performs the following steps when --bootstrap-controller is used:

  1. Send DescribeCluster RPC to the controller:
    When the admin client is initialized with --bootstrap-controller, it will first issue a DescribeCluster request to the controller node to retrieve the metadata of the cluster, including all controller endpoints.

  2. Retrieve and query all controllers:
    After obtaining the list of controller endpoints from the DescribeClusterResponse, the admin client will send an ApiVersionsRequest to each controller.

  3. Collect ApiVersionsResponse from controllers:
    Each controller will respond with its supported API versions. The admin client will aggregate these responses.

This change ensures that when the admin tool is bootstrapped directly from the controller, it can accurately discover and communicate with all controllers in the quorum.

Compatibility, Deprecation, and Migration Plan

  • Deprecate Kafka-broker-api-version.sh, Kafka-broker-api-version.bat and it will be removed next major version release.
  • The broker API version query implementation in kafka-cluster.sh api-versions --bootstrap-server will align with the current behavior and output of kafka-broker-api-versions.sh

Test Plan

  • Integration test and unit test should cover this feature.

Rejected Alternatives

  • We can use --bootstrap to retrieve all controller and broker, the additional argument is not necessary but we need to identity what type the node is and inconsistent with other kafka tools.
  • No labels