This document outlines the essential elements for releasing Airflow 3.0. As a living document, it will be frequently updated based on ongoing progress and community discussions/decisions. The scope and timeline presented here serve as a “statement of intent,” ensuring transparency and visibility regarding the goals and progress of Airflow 3.0. The community discussions about Airflow 3 happen in the following places:

Principles of Approach for 3.0

The core principles and philosophies that will guide the development of Airflow 3.0:

  1. For the features that require breaking changes, ship Airflow 3 with the foundational code to allow for iterative development, optimizing for speed and a quicker feedback cycle.
  2. Ensure a smoother migration path between Airflow 2 and 3, particularly for DAG authors using the existing official Airflow providers.
  3. For AIPs, be explicit about what's for AF 3.0 and what's for the next minor releases (3.1, 3.2, ..)
  4. Build features that solidify Airflow as the modern Orchestrator that has state-of-the-art support for Data, AI & ML workloads.
    1. This includes enhancing scalability, performance, and enterprise-level security, adhering to the principle of least privilege.
    2. Making Airflow aware of what's happening in the task to provide better auditability, lineage & observability
  5. Set up the codebase for the next 3-5 years.
  6. Simplify the Learning Curve for new Airflow users
  7. Shift focus on Airflow 2 to stability: bug fixes + security fixes after AF 2.10 (~Aug). This should continue for a longer period of time after AF 3 release 
  8. Target a shorter cycle to release Airflow 3 si users have enough time between Airflow 3 release and Airflow Summit 2025, so we can have talks about Successful migrations


The following Guidelines will help us decide if a feature should be in Airflow 3 or not:

  1. Alignment with Core Principles (mentioned above)
  2. Workstream Ownership (can be more than one). If no one is available to lead the workstream, the feature will be parked until a dedicated owner is found
  3. Community Demand and Feedback
  4. Impact on Scalability, Performance & Security 
  5. Backward Compatibility and Migration Effort
  6. Implementation Complexity and Maintenance
  7. For big features, discussion on AIPs & a successful vote on the dev mailing list 

Scope

WIP in Airflow 3 Workstreams

Timelines

DateMilestone
August 2024The main branch becomes Airflow 3 as soon as Airflow 2.10 is released
......
......
Dec 2024Dev complete on breaking changes for Upgrade Utilities work
Jan 2025Alpha releases
Feb 2025Beta releases
March 2025Airflow 3 release

Development Calls and Meetings

Appendix

References:


  • No labels