Status
...
...
...
...
Abstract
This proposal aims to enhance Apache Airflow's DAG loading mechanism to support reading DAGs from ephemeral storage, superseding AIP-5 Remote DAG Fetcher. The objective is to generalize the DAG loader and processor to load DAGs from pathlib.Path rather than assuming direct access to the OS filesystem. This includes implementing a custom module loader that supports loading from ObjectStoragePath locations and other Path locations with caching capabilities provided by fsspec. Additionally, while this AIP does not implement DAG versioning directly, the proposed loader can be easily extended to support DAG versioning, laying the foundational layer for AIP-63.
...