Status

Current state: "Accepted"

Vote thread: here

Discussion thread: here

JIRA: https://issues.apache.org/jira/browse/KAFKA-18775

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, when using MetadataQuorumCommand to add a controller, users must provide a controller.properties configuration file. This file is required for the command to retrieve the metadata local path and endpoints needed to add voters. However, this approach has several limitations:

Limited Accessibility: The node executing the tool must have direct access to the metadata path of the node being added or removed. This restricts the ability to use node A to manage node B, as node A may not have access to the metadata folder on node B.
Dependency on Node Configuration: The tool requires access to the configuration of the node being managed.

However, the essential information for these operations — the directory UUID and endpoints — is already available from the active controller’s in-memory state and the ClusterImage.

Leveraging these sources allows us to simplify voter addition and removal, enabling the command to run without direct access to the target node’s metadata directory.

Public Interfaces

CLI

Adding a controller

For adding a controller, introduces a new option —-controller-id for the add-controller subcommand.

Add a new controller with bootstrap server

bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 add-controller --controller-id <id>

Add a new controller with bootstrap controller

bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 add-controller --controller-id <id>

Removing a controller

For removing a controller, the —-controller_directory_id option is no longer required.

Remove a controller with bootstrap server

bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 remove-controller --controller-id <id>

Remove a controller with bootstrap controller

bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 remove-controller --controller-id <id>

Public APIs

The Admin API will make the options-based overloads the preferred APIs for adding and removing voters.

AddRaftVoterOptions will be extended to include optional voterDirectoryId and endpoints fields, in addition to the existing optional clusterId. If voterDirectoryId is empty, the Admin client sends Uuid.ZERO_UUID, allowing the active controller to derive the voter's directory ID from its in-memory observer state. If endpoints is empty, the Admin client sends an empty listener collection, allowing the active controller to derive the voter's endpoints from its in-memory state. If either field is provided, the provided value is used.

RemoveRaftVoterOptions will be extended to include an optional voterDirectoryId, in addition to the existing optional clusterId. If voterDirectoryId is empty, the Admin client sends Uuid.ZERO_UUID, allowing the active controller to derive the voter's directory ID from the current voter set. If it is provided, the provided directory ID is used to identify the voter.
Existing overloads that take voterDirectoryId and endpoints as positional arguments will remain for compatibility but will be deprecated. For those overloads, the positional arguments take precedence over any corresponding values configured in the options object.

AddRaftVoterOptions.java

public class AddRaftVoterOptions extends AbstractOptions<AddRaftVoterOptions> {
    private Optional<String> clusterId = Optional.empty();
    private Optional<Uuid> voterDirectoryId = Optional.empty();
    private Set<RaftVoterEndpoint> endpoints = Set.of();

    public AddRaftVoterOptions setClusterId(Optional<String> clusterId) {
        this.clusterId = clusterId;
        return this;
    }

    public Optional<String> clusterId() {
        return clusterId;
    }

    public AddRaftVoterOptions setVoterDirectoryId(Optional<Uuid> voterDirectoryId) {
        this.voterDirectoryId = Objects.requireNonNull(voterDirectoryId);
        return this;
    }

    public Optional<Uuid> voterDirectoryId() {
        return voterDirectoryId;
    }

    public AddRaftVoterOptions setEndpoints(Set<RaftVoterEndpoint> endpoints) {
        this.endpoints = Set.copyOf(Objects.requireNonNull(endpoints));
        return this;
    }

    public Set<RaftVoterEndpoint> endpoints() {
        return endpoints;
    }
}

RemoveRaftVoterOptions.java

public class RemoveRaftVoterOptions extends AbstractOptions<RemoveRaftVoterOptions> {
    private Optional<String> clusterId = Optional.empty();
    private Optional<Uuid> voterDirectoryId = Optional.empty();

    public RemoveRaftVoterOptions setClusterId(Optional<String> clusterId) {
        this.clusterId = clusterId;
        return this;
    }

    public Optional<String> clusterId() {
        return clusterId;
    }

    public RemoveRaftVoterOptions setVoterDirectoryId(Optional<Uuid> voterDirectoryId) {
        this.voterDirectoryId = Objects.requireNonNull(voterDirectoryId);
        return this;
    }

    public Optional<Uuid> voterDirectoryId() {
        return voterDirectoryId;
    }
}

Admin.java

addRaftVoter

Admin.java

/**
 * Add a new voter node to the KRaft metadata quorum.
 * 
 * <p>
 * This is a convenience method which allows the active controller to derive the
 * voter's directory ID and endpoints from its in-memory state. It is not idempotent:
 * if multiple observers have the same node ID, the request may fail because the
 * target voter cannot be identified unambiguously.
 *
 * <p>
 * To validate the target voter or override the derived values, use
 * {@link #addRaftVoter(int, AddRaftVoterOptions)} with
 * {@link AddRaftVoterOptions#setVoterDirectoryId(Optional)} or
 * {@link AddRaftVoterOptions#setEndpoints(Set)}.
 *
 * @param voterId The node ID of the voter to add.
 */
default AddRaftVoterResult addRaftVoter(int voterId) {           
    return addRaftVoter(voterId, new AddRaftVoterOptions());
}

/**
 * Add a new voter node to the KRaft metadata quorum.
 *
 * <p>
 * The clusterId in {@link AddRaftVoterOptions} is optional.
 * If provided, the operation will only succeed if the cluster id matches the id
 * of the current cluster. If the cluster id does not match, the operation
 * will fail with {@link InconsistentClusterIdException}.
 * If not provided, the cluster id check is skipped.
 *
 * <p>
 * If {@link AddRaftVoterOptions#voterDirectoryId()} is empty, the active controller
 * derives the voter's directory ID from its in-memory observer state. If
 * {@link AddRaftVoterOptions#endpoints()} is empty, the active controller derives
 * the voter's endpoints from its in-memory state.
 *
 * <p>
 * This operation is not idempotent when the directory ID is omitted: if multiple
 * observers have the same node ID, the target voter cannot be identified
 * unambiguously and the request will fail.
 *
 * <p>
 * To validate the target voter or override the derived values, use
 * {@link AddRaftVoterOptions#setVoterDirectoryId(Optional)} and
 * {@link AddRaftVoterOptions#setEndpoints(Set)}.
 *
 * @param voterId  The node ID of the voter to add.
 * @param options  Additional options for the operation.
 */
AddRaftVoterResult addRaftVoter(int voterId, AddRaftVoterOptions options);

/**
 * Add a new voter node to the KRaft metadata quorum.
 *
 * @param voterId           The node ID of the voter.
 * @param voterDirectoryId  The directory ID of the voter.
 * @param endpoints         The endpoints that the new voter has.
 * @deprecated Since 4.4. Use {@link #addRaftVoter(int, AddRaftVoterOptions)} instead.
 * This method will be removed in Apache Kafka 5.0.
 */
@Deprecated(since = "4.4", forRemoval = true)
default AddRaftVoterResult addRaftVoter(
    int voterId,
    Uuid voterDirectoryId,
    Set<RaftVoterEndpoint> endpoints
) {
    return addRaftVoter(
        voterId,
        new AddRaftVoterOptions()
            .setVoterDirectoryId(Optional.of(voterDirectoryId))
            .setEndpoints(endpoints)
    );
}

/**
 * Add a new voter node to the KRaft metadata quorum.
 *
 * <p>
 * When this overload is used, the {@code voterDirectoryId} and {@code endpoints}
 * arguments take precedence over any corresponding values in {@code options}.
 *
 * @param voterId           The node ID of the voter.
 * @param voterDirectoryId  The directory ID of the voter.
 * @param endpoints         The endpoints that the new voter has.
 * @param options           Additional options for the operation, including optional cluster ID.
 * @deprecated Since 4.4. Use {@link #addRaftVoter(int, AddRaftVoterOptions)} instead.
 * This method will be removed in Apache Kafka 5.0.
 */
@Deprecated(since = "4.4", forRemoval = true)
default AddRaftVoterResult addRaftVoter(
    int voterId,
    Uuid voterDirectoryId,
    Set<RaftVoterEndpoint> endpoints,
    AddRaftVoterOptions options
) {
    return addRaftVoter(
        voterId,
        new AddRaftVoterOptions()
            .setClusterId(options.clusterId())
            .setVoterDirectoryId(Optional.of(voterDirectoryId))
            .setEndpoints(endpoints)
            .timeoutMs(options.timeoutMs())
    );
}

removeRaftVoter

/**
 * Remove a voter node from the KRaft metadata quorum.
 *
 * <p>
 * This is a convenience method which allows the active controller to derive the
 * voter's directory ID from the current voter set.
 *
 * <p> Note: Since 4.2.0, if {@code controller.quorum.auto.join.enable} is set to true the controller
 * must be shutdown before removing the controller from the voter set to prevent the removed
 * controller from automatically joining again.
 *
 * @param voterId The node ID of the voter to remove.
 */
default RemoveRaftVoterResult removeRaftVoter(int voterId) {
    return removeRaftVoter(voterId, new RemoveRaftVoterOptions());
}

/**
 * Remove a voter node from the KRaft metadata quorum.
 *
 * <p>
 * The clusterId in {@link RemoveRaftVoterOptions} is optional.
 * If provided, the operation will only succeed if the cluster id matches the id
 * of the current cluster. If the cluster id does not match, the operation
 * will fail with {@link InconsistentClusterIdException}.
 * If not provided, the cluster id check is skipped.
 *
 * <p>
 * If {@link RemoveRaftVoterOptions#voterDirectoryId()} is empty, the active controller
 * derives the voter's directory ID from the current voter set. Otherwise, the provided
 * directory ID is used to identify the voter.
 *
 * <p> Note: Since 4.2.0, if {@code controller.quorum.auto.join.enable} is set to true the controller
 * must be shutdown before removing the controller from the voter set to prevent the removed
 * controller from automatically joining again.
 *
 * @param voterId  The node ID of the voter to remove.
 * @param options  Additional options for the operation.
 */
RemoveRaftVoterResult removeRaftVoter(int voterId, RemoveRaftVoterOptions options);

/**
 * Remove a voter node from the KRaft metadata quorum.
 *
 * @param voterId           The node ID of the voter.
 * @param voterDirectoryId  The directory ID of the voter.
 * @deprecated Since 4.4. Use {@link #removeRaftVoter(int, RemoveRaftVoterOptions)} instead.
 * This method will be removed in Apache Kafka 5.0.
 */
@Deprecated(since = "4.4", forRemoval = true)
default RemoveRaftVoterResult removeRaftVoter(
    int voterId,
    Uuid voterDirectoryId
) {
    return removeRaftVoter(
        voterId,
        new RemoveRaftVoterOptions().setVoterDirectoryId(Optional.of(voterDirectoryId))
    );
}

/**
 * Remove a voter node from the KRaft metadata quorum.
 *
 * <p>
 * When this overload is used, the {@code voterDirectoryId} argument takes precedence
 * over any corresponding value in {@code options}.
 *
 * <p> Note: Since 4.2.0, if {@code controller.quorum.auto.join.enable} is set to true the controller
 * must be shutdown before removing the controller from the voter set to prevent the removed
 * controller from automatically joining again.
 *
 * @param voterId           The node ID of the voter.
 * @param voterDirectoryId  The directory ID of the voter.
 * @param options           Additional options for the operation, including optional cluster ID.
 * @deprecated Since 4.4. Use {@link #removeRaftVoter(int, RemoveRaftVoterOptions)} instead.
 * This method will be removed in Apache Kafka 5.0.
 */
@Deprecated(since = "4.4", forRemoval = true)
default RemoveRaftVoterResult removeRaftVoter(
    int voterId,
    Uuid voterDirectoryId,
    RemoveRaftVoterOptions options
) {
    return removeRaftVoter(
        voterId,
        new RemoveRaftVoterOptions()
            .setClusterId(options.clusterId())
            .setVoterDirectoryId(Optional.of(voterDirectoryId))
            .timeoutMs(options.timeoutMs())
    );
}

RPC Changes

AddRaftVoterRequest.json

diff --git a/clients/src/main/resources/common/message/AddRaftVoterRequest.json b/clients/src/main/resources/common/message/AddRaftVoterRequest.json
index 74b7638ea2..27a6e5face 100644
--- a/clients/src/main/resources/common/message/AddRaftVoterRequest.json
+++ b/clients/src/main/resources/common/message/AddRaftVoterRequest.json
@@ -18,7 +18,7 @@
   "type": "request",
   "listeners": ["controller", "broker"],
   "name": "AddRaftVoterRequest",
-  "validVersions": "0-1",
+  "validVersions": "0-2",
   "flexibleVersions": "0+",
   "fields": [
     { "name": "ClusterId", "type": "string", "versions": "0+", "nullableVersions": "0+",

RemoveRaftVoterRequest.json

diff --git a/clients/src/main/resources/common/message/RemoveRaftVoterRequest.json b/clients/src/main/resources/common/message/RemoveRaftVoterRequest.json
index 7d11086e53..2181ecd9ff 100644
--- a/clients/src/main/resources/common/message/RemoveRaftVoterRequest.json
+++ b/clients/src/main/resources/common/message/RemoveRaftVoterRequest.json
@@ -18,14 +18,14 @@
   "type": "request",
   "listeners": ["controller", "broker"],
   "name": "RemoveRaftVoterRequest",
-  "validVersions": "0",
+  "validVersions": "0-1",
   "flexibleVersions": "0+",
   "fields": [

Proposed Changes

Server side changes

During AddRaftVoterRequest handling, if api version >= 2,
- the voter directory id is derived from in-memory LeaderState when the value is Uuid.ZERO_UUID,
- the controller endpoints are derived from ClusterImage if endpoint set is empty, note that the ClusterImage may lag behind actual state, so endpoints are not strictly idempotent.
During AddRaftVoterRequest handing, if multiple observers share the same node ID, reject with IllegalStateException indicating the duplicate node ID and instruct the user to resolve the conflict.
During RemoveRaftVoterRequest handing, if api version >=1, the voter directory id is derived from in-memory LeaderState when the value is Uuid.ZERO_UUID.

Client side changes

Two convenience methods for adding and removing controllers have been introduced in Admin.java, addRaftVoter documented with Javadoc warnings about idempotency risks, and are intended for use only when the user understands and accepts those risks.

`MetadataQuorumCommand add-controller` changes

Add a new option —-controller-id to add-controller subcommand.

new --controller-id option

        addControllerParser
            .addArgument("--controller-id", "-i")
            .help("The id of the controller to add. This option should be used with bootstrap controller.")
            .type(Integer.class)
            .action(Arguments.store());

If —-controller-id is provided, invoke new method Admin#addRaftVoter(int)
If —-command-config and —-controller-id are both provided, the config file provided by —-command-config will only be applied in Admin client initialization.
- the description for —-command-config will be changed to "Property file containing configs to be passed to Admin Client. For add-controller, the file is used to specify the controller properties as well unless --controller-id is provided."

If neither —-command-config nor —-controller-id is provided, an exception will be thrown:

throw new TerseException("You must use --command-config or --controller-id option to add a controller.");

`MetadataQuorumCommand` `remove-controller` changes

Option controller-directory-id in remove-controller subcommand

diff --git a/tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java b/tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java
index dba7951aa4..f3bdbbeffa 100644
--- a/tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java
+++ b/tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java
@@ -471,7 +471,6 @@ public class MetadataQuorumCommand {
         removeControllerParser
             .addArgument("--controller-directory-id", "-d")
             .help("The directory ID of the controller to remove.")
-            .required(true)
             .action(Arguments.store());

The —-controller-directory-id is no longer required, we can leverage on the new method Admin#removeRaftVoter(int)
If —-controller-directory-id is explicitly provided, invoke Admin#removeRaftVoter(int, Uuid)

Compatibility, Deprecation, and Migration Plan

Client API

This change is binary-compatible. Existing Admin API overloads are retained and deprecated rather than removed. The new options-based overloads are added as default interface methods, so existing Admin implementations do not need to implement new abstract methods.

The deprecated positional overloads will be removed in Apache Kafka 5.0. Users should migrate to the options-based APIs:

addRaftVoter(int voterId, AddRaftVoterOptions options)
removeRaftVoter(int voterId, RemoveRaftVoterOptions options

Adding optional fields to AddRaftVoterOptions and RemoveRaftVoterOptions is binary-compatible for existing callers.

CLI

Existing CLI options remain supported. --command-config for add-controller and --controller-directory-id for remove-controller are still accepted.

The --command-config option remains available in add-controller.

The --controller-directory-id option in remove-controller is now optional but still supported.

RPC

The simplified behavior requires AddRaftVoterRequest v2 and RemoveRaftVoterRequest v1. For add, older controllers continue to require explicit directory ID and endpoints; for remove, older controllers require explicit directory ID.

Test Plan

New test cases will be added to MetadataQuorumCommandTest.java to validate:

Adding a controller with --controller-id.
Removing a controller without explicitly providing --controller-directory-id.

Integration tests will be added for the two new methods in Admin.java.

Rejected Alternatives

Deprecate —-command-config option in add-controller and --controller-directory-id option in remove-controller.
The main reason not to deprecate these two parameters is that they were only just introduced in 4.0, so deprecating them in a 4.x release feels a bit too soon. Also, the --command-config can be used in a different user scenario, where the user can still provide the configuration file to add-controller if they already have it locally.
Using admin APIs to retrieve directory UUID and controller endpoints, but this brings extra network communication overhead.
1. The Admin#describeMetadataQuorum method can provide the directory UUID.
2. The Admin#describeConfigs method, utilizing the bootstrap.controller address, can be used to retrieve the necessary endpoints.

Space shortcuts

Child pages

Status

Motivation

Public Interfaces

CLI

Adding a controller

Removing a controller

Public APIs

AddRaftVoterOptions.java

RemoveRaftVoterOptions.java

Admin.java

RPC Changes

AddRaftVoterRequest.json

RemoveRaftVoterRequest.json

Proposed Changes

Server side changes

Client side changes

`MetadataQuorumCommand add-controller` changes

`MetadataQuorumCommand` `remove-controller` changes

Compatibility, Deprecation, and Migration Plan

Client API

CLI

RPC

Test Plan

Rejected Alternatives

Space shortcuts

Child pages

KIP-1141: Simplifying Add/Remove Voter in MetadataQuorumCommand

Status

Motivation

Public Interfaces

CLI

Adding a controller

Removing a controller

Public APIs

AddRaftVoterOptions.java

RemoveRaftVoterOptions.java

Admin.java

RPC Changes

AddRaftVoterRequest.json

RemoveRaftVoterRequest.json

Proposed Changes

Server side changes

Client side changes

MetadataQuorumCommand add-controller changes

MetadataQuorumCommand remove-controller changes

Compatibility, Deprecation, and Migration Plan

Client API

CLI

RPC

Test Plan

Rejected Alternatives

`MetadataQuorumCommand add-controller` changes

`MetadataQuorumCommand` `remove-controller` changes