This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Under discuss

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka's controller currently has no visibility into each broker's static configuration. When a broker registers with the controller via BrokerRegistrationRequest, it reports its supported feature versions, listeners, rack, and log directories, but not its configuration values (i.e., the settings defined in server.properties). This creates a fundamental gap that blocks one ticket and two categories of improvements:

KAFKA-20544 tracks the refactor of cordoned.log.dirs validation in ConfigurationControlManager, whose description explicitly states that the cleanup depends on making static configurations available in the controller. Today, because the controller cannot see each broker's log.dirs, validating cordoned.log.dirs ⊆ log.dirs requires a resource-specific code path inside ConfigurationControlManager and a forwarded flag plumbed through the Controller interface — a workaround that the standard ConfigDef validator path should have made unnecessary. KAFKA-20544 is concrete evidence that the missing static configuration data is already shaping controller code in undesirable ways, and removing it is the prerequisite for that cleanup.

Aligning incrementalAlterConfigs validation (KIP-1256)

The controller and broker currently behave differently when processing Admin.incrementalAlterConfigs requests. One key inconsistency is that brokers immediately reject invalid dynamic configuration values, while the controller silently drops them. To replicate broker-side validation logic, the controller needs each broker's static configuration as context — certain dynamic configuration constraints depend on the static values present on that broker. Without this information, the controller cannot perform faithful broker-equivalent validation.

Enforcing configuration constraints during MetadataVersion upgrades (KIP-1294)

As the cluster's MetadataVersion advances, new constraints on configuration values may come into effect. For example, a future MetadataVersion might raise the minimum value of log.segment.bytes from 1 byte to 3 MB. Without knowledge of each broker's static configuration, the controller cannot proactively verify that all brokers satisfy the constraints of the target version before allowing an upgrade 
to proceed. The incompatibility can only be discovered after the fact, when a broker fails on restart.

This KIP addresses the root cause shared by both problems: the controller lacks broker static configuration data. We propose that brokers include their static configurations in BrokerRegistrationRequest, and that the controller persists this information in RegisterBrokerRecord so it survives controller failovers. This KIP intentionally contains no validation logic — it provides only the infrastructure that KIP-1256 and KIP-1294 build upon.

Public Interfaces

ConfigDef

We propose a new configDef api to prevent security related configuration report to controller.

/**
 * Define a security-related configuration. The controller report policy is fixed at
 * {@link ControllerReportPolicy#NEVER}, guaranteeing the broker will never include this
 * config in {@code BrokerRegistrationRequest.StaticConfigs} (KIP-1326). Intended for
 * SSL/SASL secrets, PASSWORD-type entries, JAAS configs, and other sensitive material
 * that must never leave the broker.
 *
 * @param name          The name of the config parameter
 * @param type          The type of the config
 * @param defaultValue  The default value to use if this config isn't present
 * @param validator     The validator to use in checking the correctness of the config
 * @param importance    The importance of this config
 * @param documentation The documentation string for the config
 * @return This ConfigDef so you can chain calls
 */
public ConfigDef defineSecurity(String name, Type type, Object defaultValue, Validator validator,
                                Importance importance, String documentation) {
    return define(name, type, defaultValue, validator, importance, documentation,
            ControllerReportPolicy.NEVER);
}

BrokerRegistrationRequest

We propose to bump BrokerRegistrationRequest and BrokerRegistrationResponse to a new version to include a map of static configurations. This allows brokers to report their local server.properties settings to the Controller during the registration phase.

BrokerRegistrationResponse.json

BrokerRegistrationRecord

We propose updating RegisterBrokerRecord to Version 5. This version introduces a new field StaticConfigs, which is a collection of BrokerStaticConfig objects. This allows the Controller to persist the broker's reported static settings directly within the metadata log.

RegisterBrokerRecord.json

Proposed Changes

Broker side

A broker static config is included in BrokerRegistrationRequest.StaticConfigs iff (a) it has a Validator defined in ConfigDef, and (b) its ControllerReportPolicy is not NEVER.

The validator requirement keeps the reported set aligned with what the controller can actually enforce against future dynamic config changes — a config without a validator gives the controller nothing to validate, so reporting its value would only bloat the metadata log.

The NEVER policy is an explicit opt-out, applied via the defineSecurity(...) overload of ConfigDef. It is intended for sensitive material that must never be persisted in the metadata log. The opt-out is unconditional: the config is excluded even if it has (or later gains) a validator, so the guarantee survives future changes to the inclusion logic.

Controller side

When receiving a BrokerRegistrationRequest v6 and the current MetadataVersion supports static configs, the controller copies the StaticConfigs field into the RegisterBrokerRecord.

The in-memory BrokerRegistration object is extended with a Map<String, String> field to hold the static configs. This field is populated during ClusterControlManager.replay(RegisterBrokerRecord) and is accessible to higher-level logic (KIP-1256, KIP-1294) without requiring additional requests to the broker.

Compatibility, Deprecation, and Migration Plan

• We bump the versions of RegisterBrokerRecord, BrokerRegistrationRequest, BrokerRegistrationResponse, the old record and RPC version still support.

Test Plan

• Integration test: End-to-end broker registration with static configs in a KRaft cluster.
• Unit test: BrokerLifecycleManager populates StaticConfigs with only configs that   have a ConfigDef validator, and excludes PASSWORD-type configs.
• Unit test: ClusterControlManager.replay() populates the in-memory BrokerRegistration with static configs from the record.

Rejected Alternatives

• Only report configs with non-default values - During a rolling upgrade, different brokers may be running different Kafka versions with different defaults for the same config. A broker running an older version may omit a config because it matches its own default, but that default may differ from the controller's version. The controller cannot reliably fill in the "correct" default because it only knows its own version's defaults, not the defaults of every broker version in the cluster.

• Report all non-sensitive config - 

  Reporting all non-sensitive configs (~388 entries, ~18 KB per broker) was considered. While simpler in filtering logic, it includes 163 configs that have no ConfigDef validator. The additional ~8 KB per broker adds up in large clusters (8 MB for 1000 brokers) with no practical benefit — configs without validators cannot be validated regardless of whether they are reported. The chosen approach (only configs with validators, ~225 entries, ~10 KB per broker) keeps the payload smaller and automatically expands when new validators are added in future KIPs.