...
In summary, we hope that through the Dynamic Table with declared SQL statement and freshness, you can define the batch and streaming transformation on your data in the same way, accelerate ETL pipeline development, automatically manage task orchestration, enhance data quality and freshness, save your time and effort.
Downstream Pipeline
Cascading Dynamic Table
Code Block | ||
---|---|---|
| ||
CREATE DYNAMIC TABLE dws_deals(
PRIMARY KEY (order_id, ds) NOT ENFORCED
)
PARTITIONED BY (ds)
FRESHNESS = INTERVAL '1' HOUR
AS SELECT
o.ds
o.order_id,
count(*) order_cnt,
sum(pay_amt) order_total_amt
...
FROM
dwd_orders as o
group by o.order_id, o.ds |
We can create cascading DWS Dynamic Table dws_deals based on DWD's Dynamic Table dwd_orders.
OLAP Query
Code Block | ||
---|---|---|
| ||
SELECT * FROM dwd_orders WHERE ds = '20231023' LIMIT 10; |
We can also select the Dynamic Table from dwd_orders by a point query.
Public Interfaces
SQL Statements
...