...

A new metadata.version will be introduced to gate the usage of this new record. 

Proposed Changes

Changes to the KRaft Controller

We define a new concept in the controller called a record transaction. A transaction consists of a BeginMarker, a number of records, and either an EndMarker or AbortMarker. These records can be committed atomically to the Raft layer in multiple batches. By allowing a set of records to span across several record batches, we eliminate the current batch size limitation and further decouple the controller from Raft layer implementation details.

...

There will be control records in the log after the partial transaction, but these are not exposed to the controller as records.

Record Visibility

The high watermark of the Raft layer is used to control visibility of metadata updates in the controller. Since we are defining transactions that may span across Raft commits, we must not make a transaction visible to readers until the EndMarker/AbortMarker is processed.

Compatibility, Deprecation, and Migration Plan

...