Status

Current state: Accepted

Discussion thread: https://lists.apache.org/thread/6zsr70lpzqp96g4jf6kngyms4kkz1tp1

Vote thread: https://lists.apache.org/thread/dtc48g4zjdj4s4n0hm99hm0xoho5sp1n

JIRA: KAFKA-18628 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Since Apache Kafka 4.0, ZooKeeper has been removed and KRaft is the only supported mode (KIP-500). In KRaft mode, a single process can serve as a broker, a controller, or both. KIP-631 therefore introduced node.id to replace broker.id as the canonical node identifier.

However, the legacy broker.id configuration still exists and is treated as a synonym of node.id. If only one is set, the other is automatically populated with the same value. If both are explicitly set, they must be equal; otherwise, a ConfigException is thrown.

Having two configuration properties that must always hold the same value causes unnecessary confusion and contradicts the documentation, which describes node.id as required in KRaft mode.

Related ZooKeeper-era configurations such as broker.id.generation.enable and reserved.broker.max.id were already removed in Kafka 4.0, making broker.id the only remaining legacy identifier config. We should deprecate it to simplify the configuration surface and guide all users toward the single, canonical configuration: node.id.  broker.id is also the key the tiered storage plugins are configured with, so removing the broker config alone would leave the legacy name in the plugin contract.

Public Interfaces

This KIP proposes to deprecate the broker.id server configuration property in Apache Kafka 4.4 and remove it in Apache Kafka 5.0.

Notes:

• The default value of -1 for broker.id is effectively unused in KRaft mode. If node.id is set, the synonym mechanism overwrites broker.id with the value of node.id. If neither is set, node.id validation fails with a ConfigException before the default is ever used.
• The documentation describes node.id as required in KRaft mode, but it can actually be omitted currently if broker.id is set, because the synonym mechanism automatically populates node.id from broker.id. After the removal in 5.0, node.id must be explicitly set.

The behavior is the same for all process.roles (broker, controller, or combined). No changes to public APIs, network protocols, metrics, or command-line tools.

Tiered storage plugin configuration

RemoteLogManager configures RemoteStorageManager and RemoteLogMetadataManager instances with a map that includes broker.id but not node.id.

TopicBasedRemoteLogMetadataManagerConfig.BROKER_ID is deprecated and replaced by NODE_ID:

/**
 * @deprecated Use {@link #NODE_ID} instead. This key is no longer passed to plugins from Kafka 5.0 (KIP-1232).
 */
@Deprecated(since = "4.4", forRemoval = true)
public static final String BROKER_ID = "broker.id";

public static final String NODE_ID = "node.id";

Metrics reporter plugins are unaffected: they are configured through AbstractConfig#getConfiguredInstances, which passes the broker's full originals(), so both keys are already visible to them.

Proposed Changes

Phase 1: Deprecation (Apache Kafka 4.4)

1. Log deprecation warning: When broker.id is explicitly set in the server configuration, log a WARN-level message at startup:

   The 'broker.id' configuration is deprecated and will be removed in Apache Kafka 5.0. Please use 'node.id' instead.
   

2. Preserve backward compatibility: The existing synonym mechanism will continue to function. Users who only set broker.id will still have it automatically copied to node.id. The validation requiring both to be equal (if both are explicitly set) also remains unchanged.

3. Pass node.id to the tiered storage plugins alongside broker.id, and switch Kafka's own plugin implementations to read node.id. They fall back to broker.id with a WARN when node.id is absent, and fail with a ConfigException when neither key is present. The fallback never triggers inside a broker, since RemoteLogManager always passes both keys.

Phase 2: Removal (Apache Kafka 5.0)

1. Remove broker.id configuration: The BROKER_ID_CONFIG will be removed from ServerConfigs. Setting it will be silently ignored.
2. Remove synonym logic: Remove the broker.id/node.id synonym handling from AbstractKafkaConfig.populateSynonyms().
3. Retain internal brokerId() accessor: The brokerId() method will continue to exist for internal use, delegating to nodeId().

4. Stop passing broker.id to the tiered storage plugins, and remove TopicBasedRemoteLogMetadataManagerConfig.BROKER_ID.

Behavior Matrix

The following table summarizes the expected behavior for each configuration scenario across version ranges:

Note: In the 5.0 column, "ignored" means broker.id is treated as an unrecognized configuration and silently ignored. The Kafka server does not log warnings for unrecognized configurations.

Compatibility, Deprecation, and Migration Plan

Tiered storage plugins. Implementations that read broker.id from configure(Map) keep working through 4.x, since both keys are passed, and must switch to node.id before 5.0. This is the only breakage in this KIP outside the broker.

Migration Steps

Users should replace broker.id with node.id in their server configuration. If both are configured, simply remove broker.id.

Downgrade Considerations

Downgrading from 5.0 to 4.x is safe as long as the same configuration file is used — any config that was valid on 4.x before the upgrade remains valid after the rollback. The only edge case is if broker.id and node.id are set to different values on a 5.0 binary (where broker.id is ignored) and then downgraded to 4.x, which would result in a ConfigException. In practice, this cannot happen if the config was originally used on 4.x.

Tiered storage plugins. Implementations that read broker.id from configure(Map) keep working through 4.x, since both keys are passed, and must switch to node.id before 5.0. This is the only breakage in this KIP outside the broker.

Test Planㄇㄣ

Phase 1: Deprecation (Apache Kafka 4.4)

Unit tests to verify:

• A deprecation warning is logged when broker.id is explicitly set (with or without node.id).
• Backward compatibility: setting only broker.id still works (existing tests in KafkaConfigTest already create configs with only broker.id set).

Phase 2: Removal (Apache Kafka 5.0)

Unit tests to verify:

• broker.id is no longer recognized and is silently ignored.
• Setting only broker.id without node.id results in a ConfigException.

Rejected Alternatives

Throw ConfigException when broker.id is set in 5.0

Instead of silently ignoring broker.id in 5.0, we could explicitly throw a ConfigException. However, this would force users to update their configuration files before upgrading, making the upgrade process more disruptive. Silently ignoring unrecognized configurations is consistent with how other removed configurations are handled in Kafka.

Log a customized warning when broker.id is set in 5.0

We could add special-case logic to detect and warn users that broker.id has been removed. However, the Kafka server does not log warnings for unrecognized configurations, and adding a one-off check just for broker.id would be inconsistent with how other removed configurations are handled. The deprecation warning in Phase 1 already provides sufficient notice for users to migrate.

References

• KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum
• KIP-631: The Quorum-based Kafka Controller