Motivation

One thing that was left unstated in the AIP was the implications for dags.

The asset decorator is just a wrapper that generates a dag with one task.  So to have "partitioned assets", we need to have a dag that is scheduled in a partition-aware way, dag runs that need to be aware of partitions, and tasks that need information about what partitions they should be processing, etc.

The cleanest way to do this essentially to make DAGs partition aware: then a DagRun can be scheduled for a specific partition, and the task can be aware of this, and asset events generated an be aware of this.

This also has a benefit of allowing folks to use the partitioning feature without having to adopt the asset decorator approach.

Proposal

We'll modify the ORM to add add partition_key to dag_run and asset_event.

We'll enhance the timetable interface so that periodic dags will be able to create a dag run for a partition instead of for a logical_date.  We think we'll need to add a mechanism to define a partition scheme.

Concerning how to define the scheduling and partitioning behavior, probably the most natural way to do it is to define the partition scheme and let the scheduling follow that.  So that would be something like the following:

DAG(schedule=PartitionTimetable(partition_scheme=MyPartitionScheme(...), ...))

But we can easily imagine that users will want to be able to use their standard timetables and have the new partitioning behavior.  And this, we imagine, will look something like this:

DAG(schedule=CronTriggerTimetable("*/5 * * * *", partitions=True)

Or something along these lines.

Currently we plan that logical date would be null when the DAG is driven by partitions.