...

In summary, we hope that through the Dynamic Table with declared SQL statement and freshness,  you can define the batch and streaming transformation on your data in the same way,  accelerate ETL pipeline development, automatically manage task orchestration, enhance data quality and freshness, save your time and effort.

Downstream Pipeline

Cascading Dynamic Table

CREATE DYNAMIC TABLE dws_deals(
	PRIMARY KEY (order_id, ds) NOT ENFORCED
)
PARTITIONED BY (ds)
FRESHNESS = INTERVAL '1' HOUR
AS SELECT
  o.ds
  o.order_id,
  count(*) order_cnt,
  sum(pay_amt) order_total_amt
  ...
FROM 
  dwd_orders as o
group by o.order_id, o.ds

We can create cascading DWS Dynamic Table dws_deals based on DWD's Dynamic Table dwd_orders.

OLAP Query

SELECT * FROM dwd_orders WHERE ds = '20231023' LIMIT 10;

We can also select the Dynamic Table from dwd_orders by a point query.

Public Interfaces

SQL Statements

...