Table of Contents
How to use this page?
This page aims to scope out and track major epics planned for Airflow 3.0, those that will be deferred to 3.1 and later, items that will be maintained from 2.10.x, and the work related to providers.
Each workstream should have one or more owners (identified by the Author field in the AIPs table or the Owner field in the "Other candidates" table) leading it from the implementation point of view. If no one volunteers to own a stream, it will be marked as needing community help, and a related GitHub issue will be created for it.
Stream/epics that require an architecture change or a change that impacts a large user base will need an AIP. The items in the "candidates" tables are epics that don't currently have an AIP. Some of these items have an early draft outside of cWiki, which will be fleshed out into AIPs next. At that point, it would be moved to the AIPs table automatically once the correct label is added.
3.0
AIPs targetting Airflow 3.0
| Theme / Workstream | Improvement Proposals | Responsible | Status (Github project link) |
|---|---|---|---|
| Run anywhere in any language | AIP-72 Task Execution Interface aka Task SDK | Close to feature complete with support for Local, Celery and Kubernetes Executors. Main paths now working Github project Packaging changes still in progress | |
| AIP-69 Edge Executor (Initial Name: Remote Executor) | In progress, and incorporated most of Task Execution changes. | ||
| DAG versioning | AIP-65: Improve DAG history in UI | Feature complete. Github project UI done as part of AIP-38 | |
| AIP-66: DAG Bundles & Parsing | Feature complete and Tested. Github project | ||
| UI modernization | AIP-38 Modern Web Application | Excellent progress and close to feature complete Github project | |
| AIP-84 Internal API for UI Operations | Brent Bovenzi Pierre Jeambrun | Initial scope complete Github project Long tail changes being made because of UI driven changes | |
AIP-79: Remove Flask AppBuilder as Core dependency | Feature complete. | ||
| Data Assets, Event driven scheduling | AIP-74 Introducing Data Assets | Feature complete and Tested UI close to completion as part of AIP-38 Github project | |
| AIP-75 New Asset-Centric Syntax | Almost feature complete. Github project | ||
AIP-82 External event driven scheduling in Airflow | Original scope complete, additional scope based on streaming integration being worked on. Github project | ||
| Misc | Completed. Github project | ||
| Almost feature complete. | |||
| Feature complete and tested. Github project | |||
| In progress, will have a partial release. Github project | |||
| Packaging changes | Alpha and Beta based on UV | Multiple | UV also being used for easier Provider and TaskSDK development. Production packaging still WIP |
Other candidates
These are breaking changes or items that don't need an AIP (or will have an AIP in the coming weeks but don't have one yet) and are important to keep track of.
| Stream | Primary Theme(s) | Owner | Mailing List Thread / other links |
|---|---|---|---|
| Tech-debt | Will be obsoleted by AIP-86 Deadline Alerts Dennis Ferruzzi | |
| Monitoring | Deferred to post 3.0 | |
Scheduler Performance Improvements
| Performance | ||
Airflow 2 to 3 Upgrade Framework | Migration | Framework: Tzu-ping Chung Community effort for each rule/upgrade | |
Improve security of Airflow Supply Chain | Security | ||
Remove deprecated code
| Tech-debt | All | SubDAGs and DAG Pickling have been removed by Kaxil Naik https://lists.apache.org/thread/km375hokfs66hjgkz632ggwtnk96hkwj |
Simplify DB Migrations | Tech-debt | Done. As per Airflow 3.0 scope. | |
Code Cleanup
| Tech-debt | ||
| Tech-debt | Deferred to post Airflow 3, based on dev call on 7th Nov. | |
Observability of Callbacks on UI
| Observability | DAG level callbacks to be treated as Teardown tasks and to be run as part of Workers in Airflow 3. This will be done as part of AIP-72. | |
Improvements to the Timetable interface | Learning Curve Simplification | Likely to be deferred | |
Airflow Standalone Improvements | DevEx | Likely to be deferred | |
Sensor Improvements
| Performance | Kaxil Naik started thread on dev list | |
Remove StatsD and replace it with Open Telemetry as first-class citizen | Monitoring | ||
Avoid additional DAG run during catchup | Tech-debt / Simplification |
Other items that don't have a breaking change, aren't noteworthy or do not have a confirmed stakeholder are tracked as part of https://github.com/apache/airflow/issues/39593.
The AIPs tables below are auto-populated based on the labels on the AIP document:
| Category | Labels |
|---|---|
| Airflow >=3.1 | airflow-improvement-proposal/target=v3_1_plus |
| Airflow >=2.10, <3 | airflow-improvement-proposal/target=v2_10_plus |
3.1 & above
AIPs targetting Airflow >=3.1
| Title | State | Project | Authors |
|---|---|---|---|
| [WIP] AIP-70 Migrating to asynchronous programming | Draft | Hussein Awala | |
| [WIP] AIP-77 Asset Validations | Draft | ||
| AIP-57 Refactor SLA Feature | Superseded by AIP-86 | Sung Yun (now taken over by Shubham Mehta Elad Kalif Rom Sharon ) | |
| AIP-67 Multi-team deployment of Airflow components | Accepted | Jarek Potiuk | |
| AIP-68 Extended Plugin Interface for React Views | Accepted | ||
| AIP-71 Generalizing DAG Loader and Processor for Ephemeral Storage | Draft | ||
| AIP-76 Asset Partitions | Accepted | ||
| AIP-80 Explicit Template Fields in Operator Arguments | Accepted | Tzu-ping Chung | |
| AIP-85 Extendable DAG parsing controls | Draft | ||
| AIP-86 Deadline Alerts (Formerly SLA) | Accepted | Dennis Ferruzzi |
Other candidates
| Stream | Primary Theme(s) | Owner |
|---|---|---|
Additional Language Support (follow-up of AIP-72)
| Learning Curve Simplification | |
DAG & Task Group templates | Learning Curve Simplification | |
non-DAG Based Authoring:
| Persona-Expansion & Learning Curve Simplification | |
Airflow CLI interaction with remote Airflows | DevEx | |
Manual Task Approval | Modern Orchestrator | |
DAG Folders in UI | Observability | |
Define and visualize custom charts and metrics within the Airflow UI | Monitoring | |
Task Caching | Performance | |
Enforce Strong Typing | Tech-debt | |
Workflow-in-Workflow Make it possible for Airflow to easily integrate and orchestrate other workflows - such as: dbt, Databricks workflows, MLFlow, Kubeflow, LangChain, dlthub, dagworks etc. Allowing to externalize execution of whole "sub workflows" (likely denoted by a Task Group). Have a way to interact between Airflow (being monitoring an dmanagement for complete pipelines) where parts of the pipeline (whole task groups) would run elsewhere. This should work both ways:
A lot of inspiration can be taken from Cosmos and Databricks Workflow Plugin. GitHub issue for that is https://github.com/apache/airflow/issues/40587 to make sure the current "plugin" solution for it will be migrated to Airflow feature in due time. | Modern Orchestrator | |
Pluggable DAG bundle backend | Modern Orchestrator |
2.x
AIPs targetting Airflow 2.x
| Title | State | Project | Authors |
|---|---|---|---|
| AIP-44 Airflow Internal API | Completed ...but... Replaced by AIP-72 for AF 3, and we can ship what we have in main for 2.10 as "experimental" and stated to change for AF 3 | Jarek Potiuk, Mateusz Henc | |
| AIP-49 OpenTelemetry Support for Apache Airflow | Completed | ||
| AIP-51 Removing Executor Coupling from Core Airflow | Complete | https://github.com/orgs/apache/projects/162 | |
| AIP-61 Hybrid Execution | Completed | Niko Oliveira | |
| AIP-62 Getting Lineage from Hook Instrumentation | Accepted | Maciej Obuchowski | |
| AIP-64: Keep TaskInstance try history | Completed |
Other candidates
| Stream | Owner | Mailing List Thread |
|---|---|---|
Critical Bug Fixes & Security Fixes |
Providers
Candidates
| Stream | Owner | Mailing List Thread |
|---|---|---|
Breakup monolithic providers (Google, k8s etc.) | ||
Longer minimum Airflow Support |