Page History

...

After KIP-1262, kafka operators no longer need to invoke kafka-storage format to start kafka. Clusters who skip formatting essentially have a bootstrapping "state" of the latest MetadataVersion and a ClusterIdRecord who is generated by the initial active controller.

Clusters who skip formatting will also have a static KRaft quorum, which is explained in the below section, so controller.quorum.voters must be defined. Operators can still end up with a dynamic quorum without formatting because static to dynamic quorum upgrades are supported.

Operators can still format clusters the same way as prior to this KIP. However, the --cluster-id  flag now becomes optional. When --cluster-id is specified, write meta.properties V2 with it. When --cluster-id  is not specified, and the formatter's resolved MV supports ClusterIdRecord, write meta.properties V2 with an empty cluster.id. If the formatter's resolved MV does not support ClusterIdRecord, formatting will fail.

When is formatting still required before starting kafka

There are a few situations where operators still need to format nodes before starting kafka, but this only applies to bootstrap controllers. Bootstrap controllers are those who participate in the initial KRaft leader election. These are the nodes specified by controller.quorum.voters in a static quorum, and the nodes who format with --standalone/--initial-controllers  in a dynamic quorum.

Formatting of bootstrap controllers is still required when operators want to:

• Specify bootstrap SCRAM credentials
• Specify bootstrap non-default feature levels or metadata version
• Bootstrap a dynamic quorum (kraft.version=1) cluster.

meta.properties will be written during kafka broker/controller startup if it doesn't exist already (from formatting)

...

Remove the requirement for formatting for brokers and observer controllers, but still require it for bootstrap controllersBootstrap controllers are those who participate in the initial KRaft leader election. These are the nodes specified by controller.quorum.voters in a static quorum, and the nodes who format with --standalone/--initial-controllers  in a dynamic quorum. 

The main reason we can relax this to say no nodes require formatting is that KRaft can elect a leader when all nodes do not have cluster.id defined.