This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Page tree
Skip to end of metadata
Go to start of metadata


  • Security and SSL support
  • Quota - currently DL support per stream rate limiting. It is kind of throughput quota. It would be good to also have data size quota.
  • Improve rate limiting algorithm - currently the rate limiting is a decentralized solution. It would be good to see how to evolve it into a more centralized solution and more controllable.

Core Library

  • Metadata Abstraction to integrate with different consistent metadata stores. For example, Etcd.
  • Log segment abstraction to integrate with different data stores. For example, HDFS, Memory, Amazon S3 and such.
  • Endless stream
  • Stream transformation, like filtering and projection
  • Support large records over 1 megabyte.

Write Proxy

  • Zero data copy enhancement
  • Better stream placement. For example, traffic or network bandwidth usage aware algorithms.
  • Be able to support different wire protocol. For example, gRPC.
  • Support exclusive writer. Currently you can achieve fencing only if you use core library directly. In some use cases, fencing is still required even using write proxy.
  • Exact once or transnational publish - DP-1 DP-2

Write Proxy Client

  • Multiple language bindings. E.g C++ client.


  • Different deployment modes: local, distributed and broker-like.
    • Currently you can deploy the whole stack locally or separated in a different way. It might make sense to deploy both bookie and write proxy together in same instance, in a more 'broker'-like way. It would simply the deployment if users don't use a cluster management system like mesos.
  • Easy to deploy and install to different environments
  • Be able to run distributedlog on Windows


  • Documents written in different languages. For example, Chinese.
  • Improve the documentation. For example, write proxy wire protocol, rate limiting algorithm.


  • Integrate with ingestion systems
  • Integrate with stream computing engines
  • Integrate with use cases that require a WAL
  • Libraries that make building state machines easily
  • No labels