Abstract

Fluss is a streaming storage built for real-time analytics which can serve as the real-time data layer for Lakehouse architectures.

Proposal

Fluss is a distributed streaming storage service that provides high-throughput and sub-second latency for streaming reads and writes. Unlike traditional message queue systems, Fluss is specifically designed to be a streaming "table" storage, tailored for analytical workloads. With its columnar stream and real-time update capabilities, Fluss integrates seamlessly with Apache Flink and Lakehouse architecture, enabling high-performance, low-latency real-time analytics and stream processing.

Background

Nowadays, stream processing has become a cornerstone of modern data architectures. A common architectural pattern involves using Apache Flink as the stream processor in conjunction with a message queue such as Apache Kafka to build real-time data pipelines and warehouses.

However, this combination presents several limitations when scaling to large workloads and integrating with data lakehouse systems. This is because traditional message queues were not originally designed for stream processing and analytics. As a result, they often lack native support for key capabilities such as columnar pruning, efficient updates, changelog subscription, and querying—features that are essential for building high-performance, analytics-ready streaming pipelines.

Fluss, developed by Alibaba with core contributors from Apache communities such as Apache Flink, Apache Paimon, Apache Calcite, etc., addresses these challenges by providing:

  • Tables, not Topics. Fluss treats tables as first-class citizens instead of topics, aligning its data model with that of data lakehouses.

  • Columnar Stream Storage. By storing streaming data in a columnar format (Apache Arrow format), Fluss achieves up to 10x faster for streaming read performance.

  • Real-Time Updates and Changelog Subscription. Native support for updates and generate changelogs for low-latency incremental stream processing.

  • Streaming & Lakehouse Unification. Fluss reinforces the stream processing capability of the Lakehouse architecture, supporting both realtime processing and historical analysis.

Rationale

With its core capabilities, we have seen Fluss has already helped many Flink users significantly reduce the cost of building streaming pipelines by removing redundant state storage and minimizing unnecessary data transfers. Furthermore, Fluss enables a unified stream-batch storage architecture through integration with data lake formats such as Apache Paimon and Apache Iceberg (WIP), avoiding the need for duplicate data copies across both streaming and batch processing which saves a lot of costs.

On the other hand, as the lakehouse architecture gains momentum, the industry is increasingly adopting composable architectures, where each system specializes in its respective layer—storage, format, processing. Fluss aims to become the unified real-time data layer that bridges data streaming and data lakehouses. To achieve this vision, Fluss is designed to seamlessly integrate with various open table formats (e.g., Iceberg, Paimon) and query engines (e.g., Apache Flink, Apache Spark, Apache Doris), enabling a wide range of real-time analytics use cases.

The Apache Software Foundation provides an ideal home for Fluss as it evolves into an independent, community-driven open-source project. Being at Apache will foster deeper integration with a broad set of ecosystem projects and help Fluss grow into a foundational component of modern real-time data architectures.

Initial Goals

Although many of the core features of Fluss have been developed and verified in the production environment, there is still a lot to evolve, such as the following requirements from users:

  1. Extend Ecosystem: support more query engines, such as Spark, Doris, StarRocks, Trino, etc. And extend python ecosystem like integration with DuckDB.

  2. Improve core capabilities: especially the performance of streaming reads and updates, and the seemless interoperability with Lakehouse.

  3. Enhancement: rolling upgrade, multi-tenant, high-available and fault tolerance, etc.

Besides core feature development acceleration, we believe entering the Apache incubation will help Fluss to grow into a more vibrant and neutral open source community.

Current Status

Fluss project was started at Alibaba in 2023 with the same project name. It was open-sourced under the Apache License, Version 2.0 in November 2024. The source code is hosted at GitHub under Alibaba organization (https://github.com/alibaba/fluss).

Meritocracy

Since its open-source, Fluss has gained strong interest from dozens of companies and individuals. Within half year of being open-sourced, the project has garnered over 1,100 GitHub stars and contributions from more than 50 contributors across multiple companies. As the project entering incubation, we are committed to fostering a vibrant, diverse, and merit-driven community.

We aim to build an environment rooted in the principles of open collaboration and meritocracy. Our focus is on actively engaging with the broader community and aligning with related projects to encourage contributions and mutual growth. We are dedicated to ensuring that contributors and committers to Fluss come from a diverse range of organizations, reflecting a fair and merit-based selection process throughout the incubation period.

Community

The core maintainers of Fluss project has background of other Apache projects. Therefore, Fluss has been building a community around users and contributors in The Apache Way since it was open-sourced. All project roadmaps, design documents, and issue discussions are publicly accessible and conducted via GitHub Issues and Discussions. To further support contributor engagement, we host monthly online community calls to onboard new contributors, share updates, and foster collaboration. We believe that hosting Fluss at the ASF will provide a strong foundation to consolidate existing efforts, grow the contributor base, and strengthen connections between Fluss and other open source projects in the ecosystem.

Users

Fluss is used as a real-time streaming storage with Apache Flink together to build real-time warehouse across several business units within Alibaba. In several use cases, Fluss has reduced total costs by up to 80% compared to traditional message queues, thanks to its core abilities mentioned above. Beyond Alibaba, Fluss has also attracted interest from a growing number of external companies looking to deploy and adopt it. This aligns with our goal of joining the ASF—to establish Fluss as an independent, community-driven open-source project and accelerate its adoption across the broader industry.

Core Developers

  • Jark Wu. He is the founder of this project and a PMC member of Apache Flink, from Alibaba (GitHub ID: wuchong)

  • Yuxia Luo. He is a core developer and contributed most features of the project, and a committer of Apache Flink, from Alibaba. (GitHub ID: luoyuxia).

  • Yunhong Zheng. He is another core developer and contributed most features of the project, from Alibaba. (GitHub ID: swuferhong).

  • Feng Wang. He is the chief architect as well as a developer of the project, and a PMC member of Apache Paimon, and a committer of Apache Flink, from Alibaba. (GitHub ID: wangfengpro)

  • Benchao Li. He is an ASF Member, a PMC member and committer of Apache Calcite, and committer of Apache Flink, from ByteDance. (GitHub ID: libenchao)

  • Giannis Polyzos. He is an active advocator and developer of this project, from Ververica. (GitHub ID: polyzos)

  • Nicholas Jiang. He is an ASF Member, a PMC member of Apache Celeborn & Paimon, committer of many Apache projects, from Ant Group. (GitHub ID: SteNicholas)

  • Michael Koepf. He is a long-time active developer of this project since its open source, from Dynatrace. (GitHub ID: michaelkoepf)

Alignment

Fluss aims to store and serve real-time streaming data for stream processing and data lakehouses. This requires to integrate with various open table formats (e.g., Apache Iceberg, Apache Paimon, Apache Hudi) and query engines (e.g., Apache Flink, Apache Spark, Apache Doris). Many of the projects are from Apache. Fluss currently mainly supports Apache Paimon and Apache Flink. We plan to support others in the near future. Fluss is already under Apache License 2.0, and many of the core developers have experience on Apache projects. We believe this should foster collaboration within these Apache projects.

Known Risks

Project Name

Fluss is a German word meaning "river" which resonates deeply with the project’s vision. It symbolizes the streaming data is continuously flowing, distributing and converging into data lakes, just like a river.

Based on our search results, the term "Fluss" is not used as a trademark under any class in US and EU, so it is legal to use it as our project name.

Orphaned products

Fluss has been used within Alibaba, and its usage is growing rapidly. Alibaba has a long-time commitment to advance the Fluss project. Beyond Alibaba, many organizations are also evaluating or adopting Fluss to build more efficient and scalable streaming pipelines. The project already includes contributors from a diverse range of organizations, reflecting its growing ecosystem. We are now actively operating the community and will continue to increase the vitality of the community to attract more contributors to the community.

Inexperience with Open Source

Many of the Fluss contributors have experience working on open source projects, and by working with our mentors and the Apache community we believe we will be able to conduct ourselves in accordance with Apache Incubator guidelines.

Homogenous Developers

The current contributors are across various organizations, including Alibaba, ByteDance, AntGroup, Ververica, eBay, Dynatrace, Dream11 etc. We are committed to recruiting additional committers based on their contributions to the project.

Reliance on Salaried Developers

Most of the developers are paid by their employers to contribute to this project. Given some volunteer developers and the committers' sense of ownership of the code, the project could continue even if no salaried developers contributed to the project.

Relationships with Other Apache Products

Currently, Fluss has deep integrations with Apache Flink and Apache Paimon. It leverages Apache Flink as both the query and compute engine for streaming read and write operations, and supports ad-hoc querying capabilities. For storage, Fluss uses Apache Paimon as its Lakehouse format, enabling unified data management across streaming and batch workloads.

Looking ahead, Fluss plans to expand query engines. This includes integration with Apache Spark and Apache Doris. In addition, Fluss plans to supporting more open table formats such as Apache Iceberg and Apache Hudi . Furthermore, Fluss aims to supports protocol compatibility with Apache Kafka, allowing Kafka clients to interact with Fluss seamlessly.

An Excessive Fascination with the Apache Brand

We believe the Apache way, not the brand, will help Fluss grow and persist. We hope to make sure that a very inclusive, diverse, and meritocratic community is built outside the umbrella of a single company.

Documentation

Documentation can be found on https://alibaba.github.io/fluss-docs/.

Initial Source

The initial source code for Fluss is hosted at https://github.com/alibaba/fluss.

Source and Intellectual Property Submission Plan

As soon as Fluss is approved to join Apache Incubator, our initial committers will submit iCLA(s), SGA, and CCLA(s). The codebase is already licensed under Apache License 2.0.

We will also deprecate the initial source repository and redirect it to the new incubator project repository after approved.

External Dependencies

Apache License 2.0

  • com.aliyun.oss:aliyun-sdk-oss

  • com.aliyun:aliyun-java-sdk-core

  • com.aliyun:aliyun-java-sdk-kms

  • com.aliyun:aliyun-java-sdk-ram

  • com.amazonaws:aws-java-sdk-core

  • com.amazonaws:aws-java-sdk-dynamodb

  • com.amazonaws:aws-java-sdk-kms

  • com.amazonaws:aws-java-sdk-s3

  • com.amazonaws:aws-java-sdk-sts

  • com.amazonaws:jmespath-java

  • com.fasterxml.jackson.core:jackson-annotations

  • com.fasterxml.jackson.core:jackson-core

  • com.fasterxml.jackson.core:jackson-databind

  • com.fasterxml.jackson.dataformat:jackson-dataformat-cbor

  • com.fasterxml.woodstox:woodstox-core

  • com.github.ben-manes.caffeine:caffeine

  • com.github.stephenc.jcip:jcip-annotations

  • com.google.code.findbugs:jsr305

  • com.google.code.gson:gson

  • com.google.errorprone:error_prone_annotations

  • com.google.guava:failureaccess

  • com.google.guava:guava

  • com.google.guava:listenablefuture

  • com.google.j2objc:j2objc-annotations

  • com.jamesmurty.utils:java-xmlbuilder

  • com.nimbusds:nimbus-jose-jwt

  • com.squareup.okhttp3:okhttp

  • com.squareup.okio:okio

  • com.twitter:chill-java

  • com.ververica:frocksdbjni

  • commons-beanutils:commons-beanutils

  • commons-cli:commons-cli

  • commons-codec:commons-codec

  • commons-collections:commons-collections

  • commons-configuration:commons-configuration

  • commons-digester:commons-digester

  • commons-io:commons-io

  • commons-lang:commons-lang

  • commons-logging:commons-logging

  • io.netty:netty-buffer

  • io.netty:netty-common

  • io.netty:netty-handler

  • io.netty:netty-transport

  • io.netty:netty-transport-native-epoll

  • io.netty:netty-transport-native-unix-common

  • io.opentracing:opentracing-api

  • io.opentracing:opentracing-noop

  • io.opentracing:opentracing-util

  • io.prometheus:simpleclient

  • io.prometheus:simpleclient_common

  • io.prometheus:simpleclient_httpserver

  • joda-time:joda-time

  • net.java.dev.jets3t:jets3t

  • net.minidev:accessors-smart

  • net.minidev:json-smart

  • org.apache.commons:commons-compress

  • org.apache.commons:commons-configuration2

  • org.apache.commons:commons-lang3

  • org.apache.commons:commons-math3

  • org.apache.commons:commons-text

  • org.apache.curator:curator-client

  • org.apache.curator:curator-framework

  • org.apache.directory.api:api-asn1-api

  • org.apache.directory.api:api-util

  • org.apache.directory.server:apacheds-i18n

  • org.apache.directory.server:apacheds-kerberos-codec

  • org.apache.flink:flink-annotations

  • org.apache.flink:flink-clients

  • org.apache.flink:flink-connector-datagen

  • org.apache.flink:flink-core

  • org.apache.flink:flink-core-api

  • org.apache.flink:flink-datastream

  • org.apache.flink:flink-datastream-api

  • org.apache.flink:flink-file-sink-common

  • org.apache.flink:flink-hadoop-fs

  • org.apache.flink:flink-java

  • org.apache.flink:flink-metrics-core

  • org.apache.flink:flink-optimizer

  • org.apache.flink:flink-queryable-state-client-java

  • org.apache.flink:flink-rpc-akka-loader

  • org.apache.flink:flink-rpc-core

  • org.apache.flink:flink-runtime

  • org.apache.flink:flink-shaded-asm-9

  • org.apache.flink:flink-shaded-guava

  • org.apache.flink:flink-shaded-jackson

  • org.apache.flink:flink-shaded-netty

  • org.apache.flink:flink-shaded-zookeeper-3

  • org.apache.flink:flink-streaming-java

  • org.apache.flink:flink-table-common

  • org.apache.hadoop.thirdparty:hadoop-shaded-guava

  • org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7

  • org.apache.hadoop:hadoop-aliyun

  • org.apache.hadoop:hadoop-annotations

  • org.apache.hadoop:hadoop-auth

  • org.apache.hadoop:hadoop-aws

  • org.apache.hadoop:hadoop-common

  • org.apache.hadoop:hadoop-hdfs-client

  • org.apache.htrace:htrace-core4

  • org.apache.httpcomponents:httpclient

  • org.apache.httpcomponents:httpcore

  • org.apache.kerby:kerb-admin

  • org.apache.kerby:kerb-client

  • org.apache.kerby:kerb-common

  • org.apache.kerby:kerb-core

  • org.apache.kerby:kerb-crypto

  • org.apache.kerby:kerb-identity

  • org.apache.kerby:kerb-server

  • org.apache.kerby:kerb-simplekdc

  • org.apache.kerby:kerb-util

  • org.apache.kerby:kerby-asn1

  • org.apache.kerby:kerby-config

  • org.apache.kerby:kerby-pkix

  • org.apache.kerby:kerby-util

  • org.apache.kerby:kerby-xdr

  • org.apache.kerby:token-provider

  • org.apache.logging.log4j:log4j-api

  • org.apache.paimon:paimon-flink-1.20

  • org.apache.yetus:audience-annotations

  • org.apache.zookeeper:zookeeper

  • org.apache.zookeeper:zookeeper-jute

  • org.codehaus.jackson:jackson-core-asl

  • org.codehaus.jackson:jackson-mapper-asl

  • org.codehaus.jettison:jettison

  • org.codehaus.woodstox:stax2-api

  • org.ini4j:ini4j

  • org.javassist:javassist

  • org.jetbrains.kotlin:kotlin-stdlib

  • org.jetbrains.kotlin:kotlin-stdlib-common

  • org.lz4:lz4-java

  • org.mortbay.jetty:jetty

  • org.mortbay.jetty:jetty-sslengine

  • org.mortbay.jetty:jetty-util

  • org.objenesis:objenesis

  • org.snakeyaml:snakeyaml-engine

  • org.wildfly.openssl:wildfly-openssl

  • org.xerial.snappy:snappy-java

  • software.amazon.ion:ion-java

  • tools.profiler:async-profiler

BSD

  • com.github.luben:zstd-jni

  • org.codehaus.woodstox:stax2-api

  • org.ow2.asm:asm

BSD 2-Clause

  • dnsjava:dnsjava

BSD 3-clause

  • com.esotericsoftware.kryo:kryo

  • com.esotericsoftware.minlog:minlog

MIT License

  • org.checkerframework:checker-qual

  • org.codehaus.mojo:animal-sniffer-annotations

  • org.slf4j:slf4j-api

ICU License

  • com.ibm.icu:icu4j

CDDL 1.1 License

  • javax.xml.bind:jaxb-api

Go License

  • com.google.re2j:re2j

Eclipse Distribution License 1.0

  • jakarta.activation:jakarta.activation-api

Eclipse Public License - v 2.0

  • org.jacoco:org.jacoco.agent

JDOM License

  • org.jdom:jdom2

Cryptography

Fluss does not currently include any cryptography-related code.

Required Resources

Mailing lists

Git Repositories

Issue Tracking

The community would like to continue using GitHub Issues (but will be moved to github.com/apache/).

Other Resources

The community has already chosen GitHub actions as continuous integration tools.

Initial Committers

Sponsors

Champion

Yu Li (liyu@apache.org)

Nominated Mentors

Becket Qin (jqin@apache.org)

Jean-Baptiste Onofré (jbonofre@apache.org)

Jingsong Lee (lzljs3620320@apache.org)

Yu Li (liyu@apache.org)

Zili Chen (tison@apache.org)

Sponsoring Entity

We are expecting the Apache Incubator could sponsor this project.

  • No labels