👷 WORK IN PROGRESS

Authors: Greg Harris, Ivan Yurchenko, Jorge Quilcate, Giuseppe Lillo, Anatolii Popov, Juha Mynttinen, Josep Prat, Filip Yonov

Status

Current state: Under DiscussionDiscarded

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

This KIP was discarded due to a design change in KIP-1163 which made it unnecessary.

Glossary

• Diskless Topic: A topic which does not append to or serve from block storage devices.
• Object Storage: A shared, durable, concurrent, and eventually consistent storage supporting arbitrary sized byte values and a minimal set of atomic operations: put, delete, list, and ranged get.
• Object Key: A unique reference to an object within Object Storage.
• Batch: A container for Kafka records and a unit of record representation in the network protocol and storage format in both status quo Kafka and diskless topics.
• Shared Log Segment Object: An object containing a shared log segment for one or more diskless topic-partitions on the object storage. Contains record batches similar to classic Kafka topics.
• Batch Coordinate: A reference to a record batch within a shared log segment object, at some byte range.
• Diskless Batch Coordinator: Component serving as source of truth about batch coordinates and log segment objects. Establishes a global order of batches within a diskless topic, and serves coordinates to enable retrieval of batches.
• Object Compaction: Distinct from log compaction. A background asynchronous process which reads from and writes to multiple shared log segment objects to manage already-written objects.

...