Page History

...

But running Kafka on shared storage services is challenging because its classic ISR-based storage engine is designed for local disks. For example, although Kafka can be deployed on file storage services like NetApp Files, it can't benefit from the durability and elasticity of shared file storage.

At AutoMQ, our goal is This KIP aims to make Kafka stateless and enable , allowing it to run seamlessly on shared storage systems , and transforming its architecture from shared nothing to shared storage. But this KIP does not plan to merge AutoMQ into Kafka directly, as it's a major decision for the Kafka community and would require significant resources.This KIP aims to introduce a It introduces a unified log layer for deploying Kafka on various shared storage mediums, with offering the following benefits:

• A unified log layer that supports both current replication-based local file storage and shared storage simultaneously.

• Greater flexibility for major users or vendors to deploy Kafka on their own shared storage, as many large companies have their own DFS or HDFS infrastructure.

• Prevent community fragmentation, allowing users or vendors to extend the storage layer without complex modifications to Kafka, fostering greater community collaboration.

...