Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-20762

Pull request (reference implementation): PR #22751

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The ZooKeeper-to-KRaft migration is designed so that a cluster can linger in the dual-write phase and roll back to ZooKeeper mode if an operator discovers a problem before the migration is finalized.

In production environments, at any point in time, hardware failures may happen, some of which are cause for replacing nodes, such as failing disks. Kafka fully supports node replacements out of the box.

These two aspects (node replacements/provisioning and KRaft migration rollback) are documented as orthogonal concepts.

However, it has been observed in production systems that node replacements during a KRaft migration, when following the documented procedures, breaks the ability to roll back. This is not documented, and as both mechanisms are important tooling for operators to keep clusters available, it's preferable to make them work in conjunction. There is no way to eliminate the risk of hardware failures during an ongoing migration, and as such, the sanctioned tooling needs to be adapted to handle them.

Today, when a broker is lost during the migration and the documented process for provisioning a replacement is followed, the replacement's storage is formatted using the kafka-storage.sh format procedure. That command unconditionally writes meta.properties with version=1. If a rollback is later required, the operator configures the brokers back into migration mode — at which point the newly provisioned broker rejects its own storage and crashes before becoming available:

java.lang.RuntimeException: Found unexpected version in /srv/kafka/meta.properties. ZK-based brokers that are not migrating only support version 0 (which is implicit when the `version` field is missing).
    at org.apache.kafka.metadata.properties.MetaPropertiesEnsemble.verify(MetaPropertiesEnsemble.java:489)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:258)


The incompatibility is not documented, and node replacement is a routine operation for managed-fleet operators. The result is a silent loss of the rollback capability that the migration is supposed to preserve, discovered only at the worst possible time: during an attempted rollback.

The broker itself already formats empty log directories with a rollback-compatible version=0 meta.properties when it starts up in migration mode, so the capability exists in the code base. It is simply not reachable through the provisioning tool that operators are instructed to use. This KIP closes that gap.

Public Interfaces

A new optional flag is added to the format subcommand of the storage tool (bin/kafka-storage.sh format):

--zk-rollback-compatible

Help text: Format the node so that it remains compatible with rolling back an in-progress KRaft migration to ZooKeeper mode.

The flag is a boolean switch and takes no argument. When it is passed, the format command performs storage formatting identically to how the broker itself does during a migration sequence, resulting in a meta.properties with version=0. When it is omitted, the command behaves exactly as before and writes version=1.

No existing flags, defaults, output, exit codes, or other behaviors change.

Proposed Changes

When --zk-rollback-compatible is set, the storage tool selects MetaPropertiesVersion.V0 instead of the default MetaPropertiesVersion.V1 when building the meta.properties for each formatted log directory. This is the same version the broker writes for itself while it is in migration mode, so a node provisioned this way can participate in a rollback in exactly the way an original migration broker can.

This fix will be made on the 3.9 development branch only, as it is the only currently supported version that still supports ZooKeeper, and by extension the only supported version where KRaft migrations are still supported.

Documentation is updated in two places in the KRaft operations guide: a new "Formatting Nodes During a KRaft Migration" subsection under provisioning explains when and how to use the flag, and the migration-phases section links to it. The documentation makes clear that the flag is only relevant while a migration can still be rolled back, and should no longer be used once the migration has been finalized.

Compatibility, Deprecation, and Migration Plan

The change is fully backward compatible. The flag is opt-in, and omitting it preserves the current behavior byte-for-byte. No configuration, no other command line option, and no on-disk default is altered for existing users.

There is nothing to deprecate and no migration required. Operators who are not performing a ZooKeeper-to-KRaft migration have no reason to use the flag and are unaffected.

Test Plan

Unit tests assert both directions of the behavior at both layers that were changed:

  • The storage tool verifies that format --zk-rollback-compatible produces a version=0 meta.properties and that a plain format still produces version=1.
  • The underlying formatter verifies the same mapping when setZkRollbackCompatible(true) is set versus the default.

Rejected Alternatives

Document the incompatibility as intended behavior

We could declare that node replacement is unsupported while a migration is rollback-eligible and simply document that constraint. This was rejected because it removes a recovery path during an already high-risk operational mode and pushes significant operational burden onto operators, for no benefit — the capability already exists in the broker and is cheap to expose.

  • No labels