...
Code Block | ||
---|---|---|
| ||
CREATE DYNAMIC TABLE dt_sink PARTITIONED BY (ds) WITH( --hint time partition format 'partition.fields.ds.timedate-formatter' = 'yyyy-MM-dd' ) FRESHNESS = INTERVAL '1' DAY AS SELECT ds id, order_number, user_id, ... FROM source |
...
Given the scenario where the freshness is set to '1' DAY, meaning that the scheduling occurs once a day. Assuming today's date is March 2, 2024, when the batch scheduling is triggered at midnight, the framework will use the time passed by the scheduling system, combined with freshness and 'partition.fields.ds.timedate-formatter', to calculate the 'ds' partition that needs to be refreshed. For this case, it will determine that 'ds=2024-03-01' which corresponds to yesterday's partition. The framework then assembles the following SQL Statement to submit a Fink batch job.
...
If FRESHNESS = INTERVAL '1' HOUR, what would the behavior be? First, a task that schedules on an hourly basis is registered on the Scheduler. Then, with each execution, the framework will use the time provided by the scheduling system, combined with freshness and the 'partition.fields.ds.timedate-formatter', to calculate the 'ds' partition that needs to be refreshed. For this scenario, it would compute that ds='2024-03-02', which is the latest time partition.
...