Page History

...

CREATE DYNAMIC TABLE dt_sink
PARTITIONED BY (ds)
WITH(
  --hint time partition format
  'partition.fields.ds.timedate-formatter' = 'yyyy-MM-dd'
)
FRESHNESS = INTERVAL '1' DAY
AS SELECT 
  	ds
  	id,
  	order_number,
  	user_id,
	...
FROM source

...

Given the scenario where the freshness is set to '1' DAY, meaning that the scheduling occurs once a day. Assuming today's date is March 2, 2024, when the batch scheduling is triggered at midnight, the framework will use the time passed by the scheduling system, combined with freshness and 'partition.fields.ds.timedate-formatter', to calculate the 'ds' partition that needs to be refreshed. For this case, it will determine that 'ds=2024-03-01' which corresponds to yesterday's partition. The framework then assembles the following SQL Statement to submit a Fink batch job.

...

If FRESHNESS = INTERVAL '1' HOUR, what would the behavior be? First, a task that schedules on an hourly basis is registered on the Scheduler. Then, with each execution, the framework will use the time provided by the scheduling system, combined with freshness and the 'partition.fields.ds.timedate-formatter', to calculate the 'ds' partition that needs to be refreshed. For this scenario, it would compute that ds='2024-03-02', which is the latest time partition.

...