This document outlines the essential elements for releasing Airflow 3.0. As a living document, it will be frequently updated based on ongoing progress and community discussions/decisions. The scope and timeline presented here serve as a “statement of intent,” ensuring transparency and visibility regarding the goals and progress of Airflow 3.0. The community discussions about Airflow 3 happen in the following places:
Principles of Approach for 3.0
The core principles and philosophies that will guide the development of Airflow 3.0:
- For the features that require breaking changes, ship Airflow 3 with the foundational code to allow for iterative development, optimizing for speed and a quicker feedback cycle.
- Ensure a smoother migration path between Airflow 2 and 3, particularly for DAG authors using the existing official Airflow providers.
- For AIPs, be explicit about what's for AF 3.0 and what's for the next minor releases (3.1, 3.2, ..)
- Build features that solidify Airflow as the modern Orchestrator that has state-of-the-art support for Data, AI & ML workloads.
- This includes enhancing scalability, performance, and enterprise-level security, adhering to the principle of least privilege.
- Making Airflow aware of what's happening in the task to provide better auditability, lineage & observability
- Set up the codebase for the next 3-5 years.
- Simplify the Learning Curve for new Airflow users
- Shift focus on Airflow 2 to stability: bug fixes + security fixes after AF 2.10 (~Aug). This should continue for a longer period of time after AF 3 release
- Target a shorter cycle to release Airflow 3 so users have enough time between Airflow 3 release and Airflow Summit 2025, so we can have talks about Successful migrations
The following Guidelines will help us decide if a feature should be in Airflow 3 or not:
- Alignment with Core Principles (mentioned above)
- Workstream Ownership (can be more than one). If no one is available to lead the workstream, the feature will be parked until a dedicated owner is found
- Community Demand and Feedback
- Impact on Scalability, Performance & Security
- Backward Compatibility and Migration Effort
- Implementation Complexity and Maintenance
- For big features, discussion on AIPs & a successful vote on the dev mailing list
Scope
The big items are listed below:
There are other items which aren't AIP-worthy but are part of Airflow 3.0 are documented in Airflow 3 Workstreams#Othercandidates
Timelines
The high level milestones are shown below. The detailed development milestones for Airflow 3.0 are documented in Airflow3.0 Development Milestones
Date | Milestone |
---|---|
| Discussion started on Dev list about Airflow 3 |
May - June 2024 | Scope definition for Airflow 3: AIP definitions and voting |
| The main branch becomes Airflow 3 as soon as Airflow 2.10 is released |
Aug - Dec 2024 | Development and feature iteration for Airflow 3. Details in Airflow3.0 Development Milestones |
Dec 2024 | Dev complete on breaking changes for Upgrade Utilities work |
Jan 2025 | Alpha releases |
Feb 2025 | Beta releases |
Week of | Airflow 3 release candidates to be published |
Development Calls and Meetings
- Airflow 3 Dev call: Meeting Notes
- Special focussed meeting for Special Interest Groups (SIGs)
Appendix
References:
- Airflow 3 Dev call: Meeting Notes
- Airflow 3 e-mail thread in the dev mailing list
- Airflow 3: A proposal (from Astronomer)
- Airflow 3 Work Streams and Challenges (from Google Cloud Composer)
- Airflow 3 Proposal (from MWAA)