Table of Contents

How to use this page?

This page aims to scope out and track major epics planned for Airflow 3.0, those that will be deferred to 3.1 and later, items that will be maintained from 2.10.x, and the work related to providers.

Each workstream should have one or more owners (identified by the Author field in the AIPs table or the Owner field in the "Other candidates" table) leading it from the implementation point of view. If no one volunteers to own a stream, it will be marked as needing community help, and a related GitHub issue will be created for it. 

Stream/epics that require an architecture change or a change that impacts a large user base will need an AIP. The items in the "candidates" tables are epics that don't currently have an AIP. Some of these items have an early draft outside of cWiki, which will be fleshed out into AIPs next. At that point, it would be moved to the AIPs table automatically once the correct label is added.

3.0

AIPs targetting Airflow 3.0

Theme / WorkstreamImprovement ProposalsResponsibleStatus (Github project link)
Run anywhere in any languageAIP-72 Task Execution Interface aka Task SDKOn-track, 50% functionality done Github project
Packaging work to start in Jan.
AIP-69 Edge Executor (Initial Name: Remote Executor)In progress, but 3.0 related work waiting for Task Execution Interface
DAG versioningAIP-65: Improve DAG history in UI Back-end development now complete. Github project
UI to be done as part of AIP-38
AIP-66: DAG Bundles & ParsingIn progress Github project
UI modernizationAIP-38 Modern Web ApplicationIn progress  Github project
AIP-84 Internal API for UI Operations 

Brent Bovenzi  Pierre Jeambrun

On-track, 85% done Github project
Could have a long tail because of UI driven changes
AIP-68 Extended Plugin Interface for React ViewsNot yet started

AIP-79: Remove Flask AppBuilder as Core dependency


Back-end big rocks complete. 
Backwards compatibility work to be started
Data Assets, Event driven scheduling

AIP-74 Introducing Data Assets


Back-end development complete. 
UI to be done as part of AIP-38 Github project
AIP-75 New Asset-Centric Syntax In progress, to be done by mid-Dec Github project

AIP-76 Asset Partitions


Not yet started

AIP-82 External event driven scheduling in Airflow


In progress Github project
Misc

AIP-83 Remove Execution Date Unique Constraint from DAG Run

Completed. Github project

AIP-78 Scheduler-managed backfill 

Back-end development complete.
UI to be done as part of AIP-38 and possible API work needed when UI is being done. Github project

AIP-80 Explicit Template Fields in Operator Arguments

Not yet started

AIP-81 Enhanced Security in CLI via Integration of API

In progress. Github project

Other candidates

These are breaking changes or items that don't need an AIP (or will have an AIP in the coming weeks but don't have one yet) and are important to keep track of.

StreamPrimary Theme(s)OwnerMailing List Thread / other links

Remove SLAs

They will be added back in 3.1 & tracked in AIP-57 Refactor SLA Feature

Tech-debt

Inspect & Revamp the cardinality of Metrics

  • Remove metrics that don't make sense
  • Add metrics in the hot path

Monitoring

Deferred to post 3.0

Scheduler Performance Improvements

  • Simplifying TI Deps
  • Reworking the heartbeat mechanism in Airflow
  • Inspect & improve queries on Scheduler:
    • Bulk fetch or rewrite

Performance

A joint effort among various Stakeholders

Jedidiah Cunningham Daniel Standish 

Michal Modras 


Airflow 2 to 3 Upgrade Framework

Migration

Framework: Tzu-ping Chung

Community effort for each rule/upgrade


Improve security of Airflow Supply Chain

Security


Remove deprecated code

  • SubDAGs
  • Experimental API
  • Remove "DAG pickle"

Tech-debt

All

Simplify DB Migrations

Tech-debt

Done. As per Airflow 3.0 scope. 

Code Cleanup

  • Clean Utils directory
  • Move non-db models from airflow models
  • Rethink public/internal user-facing classes

Tech-debt



Inspect & Simplify Airflow Configurations

Tech-debt

Deferred to post Airflow 3, based on dev call on 7th Nov.

Observability of Callbacks on UI

  • Running them on Workers
  • Logs on UI

Observability

DAG level callbacks to be treated as Teardown tasks and to be run as part of Workers in Airflow 3. 
This will be done as part of AIP-72.

Improvements to the Timetable interface

Learning Curve Simplification



Airflow Standalone Improvements

DevEx



Sensor Improvements

  • Change sensors to use "async" by default
  • Remove poke/reschedule mode from sensors

Performance

Kaxil Naik started thread on dev list

Remove StatsD and replace it with Open Telemetry as first-class citizen

Monitoring


Avoid additional DAG run during catchup

Tech-debt / Simplification



Other items that don't have a breaking change, aren't noteworthy or do not have a confirmed stakeholder are tracked as part of https://github.com/apache/airflow/issues/39593.


The AIPs tables below are auto-populated based on the labels on the AIP document:

CategoryLabels
Airflow >=3.1airflow-improvement-proposal/target=v3_1_plus 
Airflow >=2.10, <3airflow-improvement-proposal/target=v2_10_plus 


3.1 & above

AIPs targetting Airflow >=3.1

Other candidates

StreamPrimary Theme(s)Owner

Additional Language Support (follow-up of AIP-72)

  • Golang 

Learning Curve Simplification


DAG & Task Group templates

Learning Curve Simplification


non-DAG Based Authoring:

  • Native DAG Factory

Persona-Expansion &

Learning Curve Simplification

Airflow CLI interaction with remote Airflows

DevEx


Manual Task Approval

Modern Orchestrator


DAG Folders in UI

Observability


Define and visualize custom charts and metrics within the Airflow UI

Monitoring


Task Caching

1. Improve performance
2. Payload caching some workflows needs to run only if payload was changed
otherwise they should simply be skipped. ArgoCD has this functionality
and it's one of the main advantages of it.

Performance


Enforce Strong Typing

Tech-debt


Workflow-in-Workflow

  Make it possible for Airflow to easily integrate and orchestrate other workflows - such as:

       dbt, Databricks workflows, MLFlow, Kubeflow, LangChain, dlthub, dagworks etc.

  Allowing to externalize execution of whole "sub workflows" (likely denoted by a Task Group).

  Have a way to interact between Airflow (being monitoring an dmanagement for complete pipelines)

  where parts of the pipeline (whole task groups) would run elsewhere.

  This should work both ways:

  • mapping the external workflow definition into Airflow Task definition
  • ability to see logs, and interact with (clear/rerun/skip etc.) the individual tasks being executed externally.

  A lot of inspiration can be taken from Cosmos  and Databricks Workflow Plugin.

GitHub issue for that is https://github.com/apache/airflow/issues/40587 to make sure

the current "plugin" solution for it will be migrated to Airflow feature in due time.

Modern Orchestrator

Pluggable DAG bundle backend

Modern Orchestrator

2.x

AIPs targetting Airflow 2.x

Other candidates

StreamOwnerMailing List Thread

Critical Bug Fixes & Security Fixes



Providers

Candidates

StreamOwnerMailing List Thread

Breakup monolithic providers (Google, k8s etc.) 



Longer minimum Airflow Support






  • No labels