Iggy Proposal

Abstract

Iggy is a high-performance, ultra-low latency and large-scale persistent message streaming platform written in Rust.

Proposal

Iggy provides a persistent message streaming platform built for scale and efficiency. It is designed to meet the growing demand for robust streaming solutions in diverse industries, with features including:

• High-performance, persistent append-only log for message streaming with ultra-low latency.
• Extremely high throughput for both writes and reads, powered by Rust (no garbage collection).
• Support for multiple streams, topics, partitions, and transport protocols (QUIC, TCP, HTTP).
• Granular authentication, authorization, and Personal Access Tokens (PAT).
• Client SDKs in multiple languages: Rust, C#, Go, Node.js, Python, Java, C++, and Elixir.
• Flexible message polling options: by offset, timestamp, or consumer-specific reads.
• Configurable retention policies with message expiry and auto-deletion.
• TLS and AES-256-GCM encryption support for secure data transmission and storage.
• OpenTelemetry logs, Prometheus metrics, and built-in benchmarking for monitoring and testing.
• Single binary deployment with no external dependencies, offering simplicity and ease of use.
• More detailed features list can be found here https://github.com/iggy-rs/iggy

Roadmap:

We’re actively working on the following features

• Low-level optimizations (zero-copy, rkyv etc)
• Configurable Tiered Storage (object storage infra like S3, GCS, etc)
• Shared-nothing design and io_uring support
• Clustering & data replication
• Advanced Web UI
• Plugins & extensions support

Iggy seeks to benefit from the ASF's proven governance model and large ecosystem, enabling a vibrant community and a sustainable future for the project.

Background

Iggy was conceived by Piotr Gankiewicz in April 2023 as an open-source project on GitHub under the MIT license. The project's inception was driven by Piotr's desire to learn Rust and to delve deeper into the intricacies of messaging systems. With a background in distributed systems and experience with tools like RabbitMQ, ZeroMQ, Kafka, and Aeron, Piotr aimed to build a message streaming platform that was lightweight, fast, and easy to use. 

Initially, Iggy had limited functionality, utilizing the QUIC protocol based on the Quinn library. Over time, it evolved to support multiple protocols, including TCP and HTTP, and implemented a typical architecture of streams, topics, partitions, and segments. This evolution allowed for parallel writes and reads to and from independent streams, supporting many distinct applications connected to the server. 

By May 2024, Iggy had grown into a project with around 10 contributors working on various components, including the core streaming server, SDKs for different programming languages, and tools like Web UI and CLI. The community's involvement was instrumental in expanding Iggy's ecosystem and functionality. 

In January 2025, with 30+ contributors, Iggy transitioned to the Apache 2.0 license, reflecting its commitment to open-source development and broader community collaboration. For more details on the project's origins, motivations, and evolution, you can refer to the blog posts:  

https://blog.iggy.rs/posts/building-message-streaming-in-rust/

https://blog.iggy.rs/posts/one-year-of-building-the-message-streaming/

Rationale

Iggy was developed to address specific challenges in existing message streaming platforms, aiming to provide a solution that is lightweight, high-performing, and user-friendly. Traditional platforms often come with complexities and resource demands that can be burdensome for developers and organizations. Iggy seeks to mitigate these issues through several key features:

Lightweight Resource Consumption: Built in Rust, Iggy ensures predictable resource usage without the unpredictability of garbage collection pauses, making it efficient even on limited hardware. 

High Throughput and Low Latency: Iggy is designed to handle millions of messages per second, providing consistent performance with minimal latency. 

Multi-Protocol Support: Supporting QUIC, TCP, and HTTP transport protocols, Iggy offers flexibility in integration and deployment across various environments. 

Advanced Features: Iggy includes functionalities such as user management, partitioning, consumer groups, message deduplication, retention policies, and data encryption, catering to diverse use cases. 

Comprehensive Tooling and SDKs: With a modern Web UI, a fully-featured CLI, and SDKs for multiple programming languages, Iggy simplifies management and integration into different systems. 

By focusing on these aspects, Iggy aims to provide a streamlined and efficient message streaming platform that overcomes the limitations of existing solutions. 

Initial Goals

The initial goals of the Iggy project are several-fold:

• Move the existing codebase to Apache and integrate it with the Apache development process.
• Set up the governance structure of the project.
• Plan for incremental development, and releases along with the Apache guidelines.
• Ensure all donated code is appropriately licensed under the Apache License.
• Attract contributions from the greater Apache and Rust community.
• Address the highly requested features and bug-fixes towards the next dot release.
• Benchmarking with larger datasets with a wide variety of hardware and cloud infrastructures.

Current Status

Since its inception in April 2023, Iggy has been an open-source project hosted on GitHub, fostering a vibrant and engaged community.

• Discord Server: Iggy boasts an active Discord community with over 300 users, engaging in discussions, sharing ideas, and providing support.
• GitHub Metrics:
• 2.2k+ Stars and 100+ Forks, showcasing the project’s popularity and developer interest.
• 30+ Contributors from around the globe, demonstrating the project’s diversity and inclusiveness.
• Over 850 Pull Requests and 600 Issues, reflecting an active development lifecycle and strong community involvement in the codebase.
• Crates.io Stats:
• 130 published versions, showcasing frequent updates and improvements with bi-weekly releases (sometimes weekly too, we try to move fast).
• Nearly 90,000 downloads, indicating significant adoption and trust in the platform.

Meritocracy

Iggy has operated as a meritocracy from the very beginning, fostering an environment where contributions of all kinds are recognized and valued. Since its creation as an open-source project in April 2023, Iggy has attracted significant interest and contributions from companies and individuals worldwide. Contributions are not limited to code but also include documentation, defining roadmaps, architecting solutions, and actively engaging with the community by answering user questions on the Discord channel.

We are committed to supporting new contributors and promoting a collaborative culture. Contributors who demonstrate consistent, high-quality contributions—whether technical or community-focused—are invited to join as committers, based on consensus among the core team. This approach is fully aligned with the ASF’s principle of meritocracy, ensuring that the project grows sustainably while fostering an inclusive and thriving community.

Community

Since its inception in April 2023, Iggy has been dedicated to building a vibrant and engaged community. This commitment to fostering collaboration and open development has resulted in a thriving ecosystem of contributors and users. Refer to the “Current Status” section for the community stats.

Future Growth with Apache

Joining the Apache Software Foundation will provide Iggy with a robust platform to accelerate community growth and broaden its reach. The credibility and support of the Apache family will help attract more contributors, users, and organizations, further solidifying Iggy as a leading open-source project.

Planned Community Initiatives

To nurture and expand the Iggy community, we are planning:

• Regular Meetups: In-person and virtual gatherings to connect contributors and users, share knowledge, and foster collaboration.
• Virtual Conferences: Dedicated events to showcase use cases, provide tutorials, and encourage community participation.
• Workshops and Webinars: Educational sessions to onboard new contributors and empower existing users.

Iggy’s ethos of collaboration and openness aligns perfectly with the Apache Foundation’s principles, and we are confident that joining the Apache family will enable us to grow the community at an even faster pace.

Core Developers

The core development team consists of highly skilled software engineers with expertise spanning full-stack, front-end, back-end, and data engineering. The team also includes experienced architects and leaders with deep expertise in distributed systems, real-time communications, and large-scale platform architectures.

Iggy thrives on the contributions of a diverse group of committers, representing multiple organizations and showcasing the project's open and inclusive nature. A detailed list of initial committers is provided in the “Initial Committers” section.

Alignment

Iggy has been developed using numerous Apache-licensed packages as part of its core infrastructure. The project’s core team brings extensive experience with various Apache top-level projects and libraries, which have significantly influenced Iggy’s design and architecture. Notably, Kranti Parisa, one of the core members, has contributed to Apache projects such as Lucene, Solr, and Camel, and co-authored the book Apache Solr Enterprise Search Server to promote ASF’s top-level projects. We plan to integrate closely with Apache OpenDAL for various storage layer implementations, with additional potential integrations outlined in the “Relationships with Other Apache Products” section below.

Known Risks

Project Name

Iggy - the name is an abbreviation of the Italian Greyhound, small yet extremely fast dogs, the best in their class. We strongly believe it’s the perfect name for our project as we want Iggy to be - the best message streaming platform in its class. Lightweight in terms of the resource consumption, fast (and predictable) when it comes to the throughput and latency, and easy to use when speaking of its API, SDK and configuration of the project.

We conducted a search for "Iggy" in the USPTO database and identified an existing registered trademark under the name “Iggy,” with the Goods & Services description listed as “Software development tool for creating graphic user interfaces in video games” (source: [USPTO Trademark Record](https://tsdr.uspto.gov/#caseNumber=77948806)).  

However, Iggy's focus is entirely different, targeting “Downloadable server-side computer software for collecting, processing, storing, securing, and delivering large amounts of data in real time” and “Downloadable computer software platforms that enable users to stream large amounts of data simultaneously in real time.”  

Given the distinct nature and target use cases of these two trademarks, we believe it is possible to register the trademark for “Iggy” within the category of message streaming engines and platforms.

Orphaned Products

First, the core developers will continue to work on Iggy. Additionally, Iggy is already used by Neferdata, SolidHash, Hugin.io, and many other customers across the world. A very active community on Discord, large number of user requirements are inspiring Iggy to grow. Therefore, the risk of Iggy being deprecated is minimal.

Inexperience with Open Source

Since its inception, Iggy has been developed entirely on GitHub as an open-source project. All current contributors are familiar with the open-source development process. 

While most contributors are new to the Apache Software Foundation's processes, the team benefits from the experience of Kranti Parisa, a core member of Iggy. Kranti is a seasoned open-source contributor with extensive experience in Apache projects such as Lucene, Solr, and Camel. He is also a co-author of the book Apache Solr Enterprise Search Server, written alongside long-time Apache Foundation members David Smiley, Eric Pugh, and Matt Mitchell.

Length of Incubation

Given the maturity of Iggy’s codebase, community and the experience of its committers, we expect to graduate to TLP in 1-2 years.

Homogenous Developers

The current list of committers includes developers from LaserData, several different companies plus many independent volunteers. The committers are geographically distributed across the world. They are experienced with working in a distributed environment. We hope to establish a developer community that includes contributors from several companies and we are actively encouraging new contributors via the Discord channels and public presentations of Iggy.

Reliance on Salaried Developers

It is expected that Iggy development will occur on both salaried time and on volunteer time, after hours. The majority of initial committers are independent software developers or allowed by their employer to contribute to this project. However, they are all passionate about the project, and we are confident that the project will continue even if no salaried developers contribute to the project. We are committed to recruiting additional committers including non-salaried developers.

Relationships with Other Apache Products

Iggy, as a high-performance, ultra-low-latency message streaming platform, is designed to serve as a foundational layer across backend infrastructures. Its horizontal scalability and real-time streaming capabilities make it an ideal integration point for OLAP systems, databases, and other Apache technologies used in modern data pipelines. Just to name a few:

OLAP Systems and Real-Time Analytics

Apache Druid: Iggy seamlessly integrates with Druid as a data ingestion layer, feeding real-time message streams for OLAP queries and high-speed analytics on large datasets.

Apache Kylin: Iggy can provide real-time streaming data to Kylin for building and updating OLAP cubes, enabling low-latency analytical queries on massive datasets.

Apache Pinot: As a real-time OLAP datastore, Pinot benefits from Iggy’s ability to deliver ultra-low-latency message streams, ensuring timely updates to analytical dashboards and user-facing queries.

Integration with Databases

Apache Cassandra: Iggy can feed real-time data into Cassandra, making it an excellent fit for use cases requiring high-speed writes and scalable, distributed storage, such as IoT applications and event logging.

Apache HBase: With its strong support for real-time operations, Iggy can act as an ingestion layer for HBase, enabling instant writes and updates for time-sensitive applications like monitoring or alerting systems.

Apache Derby: For lightweight database use cases, Iggy can provide a real-time data pipeline to Apache Derby, facilitating seamless integration with other components in the Apache ecosystem.

Data Warehouses (Apache Hive): Iggy can ingest and stream data into Hive-managed data lakes, enabling near-real-time data querying and analytics in batch or hybrid environments.

Stream Processing and Workflow Orchestration

Apache Flink: Iggy complements Flink by providing a robust, low-latency messaging layer for complex stream processing and event-driven applications.

Apache Spark: Iggy integrates with Spark Streaming to deliver high-throughput data streams for real-time analytics, machine learning, and ETL pipelines.

Apache NiFi: With NiFi’s data flow orchestration capabilities, Iggy serves as a reliable and scalable transport layer, ensuring low-latency data transfer between distributed systems.

Search and Indexing

Apache Solr: Iggy can be used to stream real-time updates to Solr, enabling near-instantaneous indexing for search applications.

Apache Lucene: For applications built directly on Lucene, Iggy can provide a lightweight, real-time feed of documents or data streams for indexing and querying.

An Excessive Fascination with the Apache Brand

We acknowledge that the Apache brand is one of the most recognized and respected in the open-source domain. However, the primary goal of this proposal is not to seek publicity but to expand the developer base and foster a broad, diverse community around Iggy. We believe that successful open-source projects thrive on inclusivity and collaboration, and becoming an Apache project provides a clear and proven framework to attract and nurture new contributors.  

The core development team has been deeply committed to Iggy over the past few years, addressing the fast-growing needs of data and event streaming in today’s digital landscape. While we remain dedicated to continuing its development regardless of its status, we see this as an opportunity to give back to the Apache community. Having benefited extensively from various Apache libraries and projects throughout our careers, we view this proposal as a way to contribute to the vibrant ecosystem that has supported so many innovative technologies.  

Documentation

The current documentation for Iggy is at

https://iggy.rs/

https://docs.iggy.rs/

https://blog.iggy.rs/

Initial Source

Iggy’s current source code (consists of multiple repositories) is hosted on Github:

https://github.com/iggy-rs

• https://github.com/iggy-rs/iggy -> https://github.com/apache/iggy
• https://github.com/iggy-rs/iggy-docs -> https://github.com/apache/iggy-docs
• https://github.com/iggy-rs/iggy-web-ui -> https://github.com/apache/iggy-web-ui
• https://github.com/iggy-rs/iggy-benchmarks-dashboard -> https://github.com/apache/iggy-benchmarks-dashboard
• https://github.com/iggy-rs/iggy-node-client -> https://github.com/apache/iggy-node-client
• https://github.com/iggy-rs/iggy-elixir-client -> https://github.com/apache/iggy-elixir-client
• https://github.com/iggy-rs/iggy-python-client -> https://github.com/apache/iggy-python-client
• https://github.com/iggy-rs/iggy-java-client -> https://github.com/apache/iggy-java-client
• https://github.com/iggy-rs/iggy-cpp-client -> https://github.com/apache/iggy-cpp-client
• https://github.com/iggy-rs/iggy-node-http-client -> https://github.com/apache/iggy-node-http-client
• https://github.com/iggy-rs/iggy-go-client -> https://github.com/apache/iggy-go-client
• https://github.com/iggy-rs/iggy-dotnet-client -> https://github.com/apache/iggy-dotnet-client

This is the exact codebase that we would migrate to the Apache Software Foundation following the guidelines.

Source and Intellectual Property Submission Plan

The Iggy codebase is currently hosted on Github: https://github.com/iggy-rs. This is the exact codebase that we would migrate to the Apache foundation.

The Iggy source code in Github is currently licensed under Apache License v2.0 and the copyright is assigned to Piotr Gankiewicz, LaserData Inc. All the contributions from external parties have been received under Apache style CLA. If Iggy fulfills and passes the conditions for being an Incubator project in the ASF, LaserData Inc will transition the source code ownership to the Apache Software Foundation via the Software Grant Agreement.

External Dependencies

More detailed information can be found here https://github.com/iggy-rs/iggy/blob/master/dependencies.md

Cryptography

Not applicable.

Required Resources

Mailing lists

We plan to use the following mailing lists:

• private@iggy.apache.org (with moderated subscriptions)
• dev@iggy.apache.org
• commits@iggy.apache.org
• users@iggy.apache.org

Git Repositories

We would like to use Git for source code management and enable GitHub mirroring functionality. We need the following Git repos:

• https://github.com/apache/iggy
• https://github.com/apache/iggy-docs
• https://github.com/apache/iggy-web-ui 
• https://github.com/apache/iggy-bench-dashboard
• https://github.com/apache/iggy-node-client
• https://github.com/apache/iggy-elixir-client
• https://github.com/apache/iggy-python-client
• https://github.com/apache/iggy-java-client
• https://github.com/apache/iggy-cpp-client
• https://github.com/apache/iggy-node-http-client
• https://github.com/apache/iggy-go-client
• https://github.com/apache/iggy-dotnet-client

Issue Tracking

The community would like to continue using GitHub Issues.

Other Resources

At this moment, no special resources or infrastructure are required.

Initial Committers

Grzegorz Koszyk, g.koszyk98@gmail.com, CLA signed

Hubert Gruszecki, h.gruszecki@gmail.com, CLA signed

Kranti Parisa, kranti.parisa@gmail.com, CLA signed

Patryk Huzarski, phuzarski@gmail.com, CLA signed

Piotr Gankiewicz, piotr.gankiewicz@gmail.com, CLA signed

Marvin Hansen, marvin.hansen@gmail.com, CLA signed

Bakytgerey Ashirbekov, ashrbah@gmail.com, CLA signed

José Roberto Araújo, jroberto.araujo@gmail.com, CLA signed

Thibaut Lambert, T1B0.gm@gmail.com, CLA signed

Sebastian Hindhede, sebastian.hindhede@gmail.com, CLA signed

Bartosz Cieśla, bartosz.ciesla@gmail.com , CLA signed

Corentin Filoche, corentinfiloche@hotmail.fr

Maciej Modzelewski, maciej@modzelewski.dev , CLA signed

Kyle Downey, kyle.downey@gmail.com

Sponsors

Champion

• Yonik Seeley (ASF Member, Incubator PMC Member)

Nominated Mentors

• Hao Ding  (xuanwo) (ASF Member, Incubator PMC Member)
• Yonik Seeley (ASF Member, Incubator PMC Member)
• Zili Chen (tison) (ASF Member, Incubator PMC Member)
• TianYi Lin (hulk) (ASF Member, Incubator PMC Member)

Sponsoring Entity

The Apache Incubator