Page tree
Skip to end of metadata
Go to start of metadata

General

  • Security and SSL support
  • Quota - currently DL support per stream rate limiting. It is kind of throughput quota. It would be good to also have data size quota.
  • Improve rate limiting algorithm - currently the rate limiting is a decentralized solution. It would be good to see how to evolve it into a more centralized solution and more controllable.

Core Library

  • Metadata Abstraction to integrate with different consistent metadata stores. For example, Etcd.
  • Log segment abstraction to integrate with different data stores. For example, HDFS, Memory, Amazon S3 and such.
  • Endless stream
  • Stream transformation, like filtering and projection
  • Support large records over 1 megabyte.

Write Proxy

  • Zero data copy enhancement
  • Better stream placement. For example, traffic or network bandwidth usage aware algorithms.
  • Be able to support different wire protocol. For example, gRPC.
  • Support exclusive writer. Currently you can achieve fencing only if you use core library directly. In some use cases, fencing is still required even using write proxy.
  • Exact once or transnational publish - DP-1 DP-2

Write Proxy Client

  • Multiple language bindings. E.g C++ client.

Deployment

  • Different deployment modes: local, distributed and broker-like.
    • Currently you can deploy the whole stack locally or separated in a different way. It might make sense to deploy both bookie and write proxy together in same instance, in a more 'broker'-like way. It would simply the deployment if users don't use a cluster management system like mesos.
  • Easy to deploy and install to different environments
  • Be able to run distributedlog on Windows

Documentation

  • Documents written in different languages. For example, Chinese.
  • Improve the documentation. For example, write proxy wire protocol, rate limiting algorithm.

Integration

  • Integrate with ingestion systems
  • Integrate with stream computing engines
  • Integrate with use cases that require a WAL
  • Libraries that make building state machines easily
  • No labels