...

• Application: General applications, such as doris' fe and be.
• OTel Libraty: Also known as SDK, it is responsible for collecting and exporting telemetry data in the program.
• OTel Collector: The OpenTelemetry Collector offers a vendor-agnostic implementation of how to receive, process and export telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors. This works with improved scalability and supports open-source observability data formats (e.g. Jaeger, Prometheus, Fluent Bit, etc.) sending to one or more open-source or commercial back-ends.
• Backends:  Responsible for persisting and presenting telemetry data, and providing the ability to analyze telemetry data. such as zipkin, prometheus, etc.

3. What traces can do

• Slow Query Location
  trace and span record the query time consumption, through trace you can count the longest time consuming queries over a period of time.
• Performance bottleneck analysis
  span records the time consumption of the network between fe and be nodes and the time consumption of each execution node of be, and the time consumption of each span in a batch of queries can be counted to analyze the performance bottlenecks.
• Quickly locate query failures
  Combine trace with log and metric, and quickly locate the relevant log and metric information by trace_id and span_id.

4. traces storage

opentelemetry is responsible for the collection, processing and export of telemetry data, and Back-end is responsible for the presentation and storage of telemetry data. traces-related Back-end such as zipkin supports storage components: memory, Elasticsearch, MySQL, etc., and even doris can be used as its storage component.
This means that if trace data is stored in doris, that is, it allows users to have some trace analysis capability without deploying collector and back-end, and also to get full trace analysis processing capability after deploying collector and back-end.

The following diagram shows the export and storage process of trace, with arrows indicating the direction of trace data flow. trace can be collected and stored directly by doris, or by exporting trace to collector for processing and then storing it in doris, and back-end pulling data directly from doris.

┌──────────────────────────┐
│                          │
│  doris                   │
│                          │
│  ┌───────────────────┐   │           ┌───────────────────┐
│  │                   │   │           │                   │
│  │                   │   │           │                   │
│  │ trace collection  ├───┼───────────►  otel collector   │
│  │                   │   │           │                   │
│  │                   │   │           │                   │
│  └─────┬─────────────┘   │           └─────────┬─────────┘
│        │                 │                     │
│        │      ┌──────────┼─────────────────────┘
│        │      │          │
│        │      │          │
│  ┌─────▼──────▼──────┐   │           ┌───────────────────┐
│  │                   │   │           │                   │
│  │                   │   │           │                   │
│  │ trace storage     ├───┼───────────►    back-end       │
│  │                   │   │           │                   │
│  │                   │   │           │                   │
│  └───────────────────┘   │           └───────────────────┘
│                          │
│                          │
└──────────────────────────┘

Detailed Design

Query trace collection and export:

...