Status
Current state: [One of "Under Discussion", "Accepted", "Rejected"]
Discussion thread: https://lists.apache.org/thread/nkn6vn1dp5r9t2wy65qy2lt1chmph46n
JIRA or Github Issue:
Released: <Doris Version>
Google Doc: <If the design in question is unclear or needs to be discussed and reviewed, a Google Doc can be used first to facilitate comments from others.>
Motivation
Doris-kafka-connector is a scalable and reliable data transfer tool that can accurately transfer data from kafka to doris.
Related Research
Detailed Design
Currently doris-kafka-connector only supports data sink to doris, which has the following advantages:
- Easy to expand. It can be deployed on a single machine or expanded to a distributed environment according to the amount of data in kafka.
- Easy to use. The kafka framework natively supports connect, making deployment and use very simple. There are currently two deployment methods:
- Standalone mode: Standalone mode is the simplest mode, where a single process is responsible for executing all connectors and tasks.
- Distributed mode: Distributed mode provides scalability and automatic fault tolerance for Kafka Connect.
- Supports multiple data formats. Currently, it has been tested to sink json, csv, and avro data formats into doris through Doris-kafka-connector.