...
An example topic creation using a transaction with two atomic batches:
| Record | Record in transaction | Record in batch |
| BeginMarkerRecord | 1 | 1 |
TopicRecord | 2 | 2 |
| PartitionRecord | 3 | 3 |
| PartitionRecord | 4 | 4 |
| PartitionRecord | 5 | 1 |
| PartitionRecord | 6 | 2 |
ConfigRecord | 7 | 3 |
| EndMarkerRecord | 8 | 4 |
Record Visibility
As the controller sends records to Raft, it will optimistically apply the record to its in-memory state. The record is not considered durable (i.e., "committed") until the high watermark of the Raft log increases beyond the offset of the record. Once a record is durable, it is made visible outside the controller to external clients. Since we are defining transactions that may span across Raft commits, we must extend our criteria for visibility to include whole transactions. When the controller sees that a partial transaction has been committed, it must not expose it until the entire transaction is committed. This is how the controller will provide atomicity for a transaction.
...
Another aspect of atomicity for transactions is dealing with controller failover. If a transaction has not been completed when a controller crashes, there will be a partial transaction committed to the log. The remaining records for this partial transaction are forever lost, and so we must abort the entire transaction. This will be done by the newly elected controller inserting an AbortMarkerRecordAbortMarker.
Detecting a partial transaction during failover is easily determined by examining the log. An incomplete transaction is only expected during failover and would occur at the end of the log.
...
The records fetched by the brokers will include everything up to the Raft high watermark. This will include partial transactions. The broker must buffer these records and not make them visible to its internal components. Once an EndMarkerRecord EndMarker or AbortMarkerRecord AbortMarker are seen, the buffered records can be processed or discarded, respectively.
...