...
Google Doc: <If the design in question is unclear or needs to be discussed and reviewed, a Google Doc can be used first to facilitate comments from others.>
Motivation
Describe the problems you are trying to solveDoris-kafka-connector is a scalable and reliable data transfer tool that can accurately transfer data from kafka to doris.
Related Research
some research related to the function, such as the advantages and disadvantages of the design, related considerations, etc.
Detailed Design
the detailed design of the function.
Scheduling
...
Detailed Design
Currently doris-kafka-connector only supports data sink to doris, which has the following advantages:
- Easy to expand. It can be deployed on a single machine or expanded to a distributed environment according to the amount of data in kafka.
- Easy to use. The kafka framework natively supports connect, making deployment and use very simple. There are currently two deployment methods:
- Standalone mode: Standalone mode is the simplest mode, where a single process is responsible for executing all connectors and tasks.
- Distributed mode: Distributed mode provides scalability and automatic fault tolerance for Kafka Connect.
- Supports multiple data formats. Currently, it has been tested to sink json, csv, and avro data formats into doris through Doris-kafka-connector.
