Table of Contents
How to use this page?
This page aims to scope out and track major epics planned for Airflow 3.0, those that will be deferred to 3.1 and later, items that will be maintained from 2.10.x, and the work related to providers.
Each workstream should have one or more owners (identified by the Author field in the AIPs table or the Owner field in the "Other candidates" table) leading it from the implementation point of view. If no one volunteers to own a stream, it will be marked as needing community help, and a related GitHub issue will be created for it.
Stream/epics that require an architecture change or a change that impacts a large user base will need an AIP. The items in the "candidates" tables are epics that don't currently have an AIP. Some of these items have an early draft outside of cWiki, which will be fleshed out into AIPs next. At that point, it would be moved to the AIPs table automatically once the correct label is added.
3.0
AIPs targetting Airflow 3.0
| Theme / Workstream | Improvement Proposals | Responsible | Status (Github project link) |
|---|---|---|---|
| Run anywhere in any language | AIP-72 Task Execution Interface aka Task SDK | On-track, 50% functionality done Github project Packaging work to start in Jan. | |
| AIP-69 Edge Executor (Initial Name: Remote Executor) | In progress, but 3.0 related work waiting for Task Execution Interface | ||
| DAG versioning | AIP-65: Improve DAG history in UI | Back-end development now complete. Github project UI to be done as part of AIP-38 | |
| AIP-66: DAG Bundles & Parsing | In progress Github project | ||
| UI modernization | AIP-38 Modern Web Application | In progress Github project | |
| AIP-84 Internal API for UI Operations | Brent Bovenzi Pierre Jeambrun | On-track, 85% done Github project Could have a long tail because of UI driven changes | |
| AIP-68 Extended Plugin Interface for React Views | Not yet started | ||
AIP-79: Remove Flask AppBuilder as Core dependency | Back-end big rocks complete. Backwards compatibility work to be started | ||
| Data Assets, Event driven scheduling | AIP-74 Introducing Data Assets | Back-end development complete. UI to be done as part of AIP-38 Github project | |
| AIP-75 New Asset-Centric Syntax | In progress, to be done by mid-Dec Github project | ||
| Not yet started | |||
AIP-82 External event driven scheduling in Airflow | In progress Github project | ||
| Misc | Completed. Github project | ||
| Back-end development complete. UI to be done as part of AIP-38 and possible API work needed when UI is being done. Github project | |||
| Not yet started | |||
| In progress. Github project |
Other candidates
These are breaking changes or items that don't need an AIP (or will have an AIP in the coming weeks but don't have one yet) and are important to keep track of.
| Stream | Primary Theme(s) | Owner | Mailing List Thread / other links |
|---|---|---|---|
| Tech-debt | Will be obsoleted by AIP-86 Deadline Alerts Dennis Ferruzzi | |
| Monitoring | Deferred to post 3.0 | |
Scheduler Performance Improvements
| Performance | ||
Airflow 2 to 3 Upgrade Framework | Migration | Framework: Tzu-ping Chung Community effort for each rule/upgrade | |
Improve security of Airflow Supply Chain | Security | ||
Remove deprecated code
| Tech-debt | All | SubDAGs and DAG Pickling have been removed by Kaxil Naik https://lists.apache.org/thread/km375hokfs66hjgkz632ggwtnk96hkwj |
Simplify DB Migrations | Tech-debt | Done. As per Airflow 3.0 scope. | |
Code Cleanup
| Tech-debt | ||
| Tech-debt | Deferred to post Airflow 3, based on dev call on 7th Nov. | |
Observability of Callbacks on UI
| Observability | DAG level callbacks to be treated as Teardown tasks and to be run as part of Workers in Airflow 3. This will be done as part of AIP-72. | |
Improvements to the Timetable interface | Learning Curve Simplification | ||
Airflow Standalone Improvements | DevEx | ||
Sensor Improvements
| Performance | Kaxil Naik started thread on dev list | |
Remove StatsD and replace it with Open Telemetry as first-class citizen | Monitoring | ||
Avoid additional DAG run during catchup | Tech-debt / Simplification |
Other items that don't have a breaking change, aren't noteworthy or do not have a confirmed stakeholder are tracked as part of https://github.com/apache/airflow/issues/39593.
The AIPs tables below are auto-populated based on the labels on the AIP document:
| Category | Labels |
|---|---|
| Airflow >=3.1 | airflow-improvement-proposal/target=v3_1_plus |
| Airflow >=2.10, <3 | airflow-improvement-proposal/target=v2_10_plus |
3.1 & above
AIPs targetting Airflow >=3.1
| Title | State | Project | Authors |
|---|---|---|---|
| [WIP] AIP-70 Migrating to asynchronous programming | Draft | Hussein Awala | |
| [WIP] AIP-77 Asset Validations | Draft | ||
| AIP-57 Refactor SLA Feature | Accepted | Sung Yun (now taken over by Shubham Mehta Elad Kalif Rom Sharon ) | |
| AIP-67 Multi-team deployment of Airflow components | Accepted | Jarek Potiuk | |
| AIP-71 Generalizing DAG Loader and Processor for Ephemeral Storage | Draft | ||
| AIP-85 Extendable DAG parsing controls | Draft | ||
| AIP-86 Deadline Alerts (Formerly SLA) | Draft | Dennis Ferruzzi |
Other candidates
| Stream | Primary Theme(s) | Owner |
|---|---|---|
Additional Language Support (follow-up of AIP-72)
| Learning Curve Simplification | |
DAG & Task Group templates | Learning Curve Simplification | |
non-DAG Based Authoring:
| Persona-Expansion & Learning Curve Simplification | |
Airflow CLI interaction with remote Airflows | DevEx | |
Manual Task Approval | Modern Orchestrator | |
DAG Folders in UI | Observability | |
Define and visualize custom charts and metrics within the Airflow UI | Monitoring | |
Task Caching | Performance | |
Enforce Strong Typing | Tech-debt | |
Workflow-in-Workflow Make it possible for Airflow to easily integrate and orchestrate other workflows - such as: dbt, Databricks workflows, MLFlow, Kubeflow, LangChain, dlthub, dagworks etc. Allowing to externalize execution of whole "sub workflows" (likely denoted by a Task Group). Have a way to interact between Airflow (being monitoring an dmanagement for complete pipelines) where parts of the pipeline (whole task groups) would run elsewhere. This should work both ways:
A lot of inspiration can be taken from Cosmos and Databricks Workflow Plugin. GitHub issue for that is https://github.com/apache/airflow/issues/40587 to make sure the current "plugin" solution for it will be migrated to Airflow feature in due time. | Modern Orchestrator | |
Pluggable DAG bundle backend | Modern Orchestrator |
2.x
AIPs targetting Airflow 2.x
| Title | State | Project | Authors |
|---|---|---|---|
| AIP-44 Airflow Internal API | In Progress / Under Development. Replaced by AIP-72 for AF 3, and we can ship what we have in main for 2.10 as "experimental" and stated to change for AF 3 | Jarek Potiuk, Mateusz Henc | |
| AIP-49 OpenTelemetry Support for Apache Airflow | Completed | ||
| AIP-51 Removing Executor Coupling from Core Airflow | Complete | https://github.com/orgs/apache/projects/162 | |
| AIP-61 Hybrid Execution | Completed | Niko Oliveira | |
| AIP-62 Getting Lineage from Hook Instrumentation | Accepted | Maciej Obuchowski | |
| AIP-64: Keep TaskInstance try history | Completed |
Other candidates
| Stream | Owner | Mailing List Thread |
|---|---|---|
Critical Bug Fixes & Security Fixes |
Providers
Candidates
| Stream | Owner | Mailing List Thread |
|---|---|---|
Breakup monolithic providers (Google, k8s etc.) | ||
Longer minimum Airflow Support |