Skip to end of metadata
Go to start of metadata
- Security and SSL support
- Quota - currently DL support per stream rate limiting. It is kind of throughput quota. It would be good to also have data size quota.
- Improve rate limiting algorithm - currently the rate limiting is a decentralized solution. It would be good to see how to evolve it into a more centralized solution and more controllable.
- Metadata Abstraction to integrate with different consistent metadata stores. For example, Etcd.
- Log segment abstraction to integrate with different data stores. For example, HDFS, Memory, Amazon S3 and such.
- Endless stream
- Stream transformation, like filtering and projection
- Support large records over 1 megabyte.
- Zero data copy enhancement
- Better stream placement. For example, traffic or network bandwidth usage aware algorithms.
- Be able to support different wire protocol. For example, gRPC.
- Support exclusive writer. Currently you can achieve fencing only if you use core library directly. In some use cases, fencing is still required even using write proxy.
- Exact once or transnational publish - DP-1 DP-2
Write Proxy Client
- Multiple language bindings. E.g C++ client.
- Different deployment modes: local, distributed and broker-like.
- Currently you can deploy the whole stack locally or separated in a different way. It might make sense to deploy both bookie and write proxy together in same instance, in a more 'broker'-like way. It would simply the deployment if users don't use a cluster management system like mesos.
- Easy to deploy and install to different environments
- Be able to run distributedlog on Windows
- Documents written in different languages. For example, Chinese.
- Improve the documentation. For example, write proxy wire protocol, rate limiting algorithm.
- Integrate with ingestion systems
- Integrate with stream computing engines
- Integrate with use cases that require a WAL
- Libraries that make building state machines easily