The purpose of an Airflow Improvement Proposal (AIP) is to introduce any major change to Apache Airflow. This is required in order to balance the need to support new features, use cases, while avoiding accidentally introducing half thought-out interfaces that cause needless problems when changed.
What is considered a major change that needs an AIP?
Any of the following should be considered a major change:
- Any major new feature, subsystem, or piece of functionality
- Any change that impacts the public interfaces of the project
What are the "public interfaces" of the project? All of the following are public interfaces that people build around:
- Airflow DAG DSL components: DAG, XCom, TaskInstance and context.
- Web UI
- Core operators (BaseOperator, SubDagOperator, SensorOperator)
- [Airflow 2.0] REST endpoints
- Command line tools and arguments
Proposal lifecycle
Discuss
Most proposals start with an idea. If you have an idea of how Airflow could improve, we encourage you to send an email to dev@airflow.apache.org with a subject starting with [DISCUSS]. This email thread will allow you to gather early feedback. We encourage you to start a Draft document or share a Google Document with the mailing list detailing your proposal. Use AIP template here to file details on your proposal.
Draft
Anyone is welcome to propose a new Improvement to Airflow. You will need to request edit access to the Confluence page in order to create your AIP document. You can do so by sending an email with your Wiki ID to dev@airflow.apache.org and request permission.
Vote
Once you or someone else feels like there’s a rough consensus on the idea and there’s no strong opposition, you can move your proposal to the Vote phase. For this you will send a new email to the dev list with the subject starting with [VOTE]. This phase follows a vote on code modifications for Apache projects as described here. Remember to link the discussion thread to your AIP and ideally also to this vote email. And don’t forget to link the AIP to the email as well.
The voting has to last at least 72 hours and it’s a good practice to extend this time if most of this period is a weekend. Whole community is encouraged to give +1 and -1 votes. There’s also fractional votes (0.5, -0.99) that allow you to express your view on the discussion without giving an explicit approval/block signal. However, only Airflow committers and PMC members have a binding vote. This means that any vote requires at least 3, +1 binding votes to be recognized as passed. In addition, any -1 binding vote will move the proposal back to the Discuss phase in order to address changes.
The vote is a consensus-seeking process, “the goal is not to *win* votes or come to a unanimous agreement, but rather to ensure that there’s a forum for people to raise and discuss their concerns, and that nobody feels strongly enough to block the group from moving forward. Consensus-seeking emphasizes discussion over enumeration: *Rough consensus is achieved when all issues are addressed, but not necessarily accommodated.*” (quote from Working in Public: The Making and Maintenance of Open Source Software by Nadia Eghbal)
If you want to introduce significant changes to your proposal after it has been accepted you can do so following the same procedure.
Accepted
Congratulations! Your proposal has been accepted. Next up is breaking down tasks. You should break down tasks on Apache Airflow GitHub issues page. You can optionally group the tasks together in a Github Project if it consists of multiple steps and you want to be able to track progress for its implementation or share the progress with others. Once issues are created, make sure to send them to the vote thread (optionally Airflow slack). Ask a committer to create a label for your AIP and link all your issues with it for easier searchability.
Now you can start contributing and - more importantly - you can, and probably should, encourage others to contribute to your project. However this is optional and up to your discretion. You can learn how to contribute and communicate by reading the Contributing guide in the GitHub repository. Use your imagination and various communication channels for ways to encourage people if you do not already have contributors following your idea. It is advisable that several people work on the AIP so that the knowledge is shared.
Completed
You should move your AIP to complete after you consider the main bulk of work has been merged to the repository. It is okay to leave open issues for minor follow up tasks like adding additional capabilities or similar.
Abandoned
AIP may be moved to abandoned if there has been no owner or intention of developing it further for over a year or if the owner decides to abandon it. The latter can happen if the creator decides to refactor AIP into a new AIP for example or if there is a proposal that supersedes the original AIP.
AIPs under discussion
Accepted AIPs
AIPs have been voted on and accepted for inclusion in Airflow but have not yet (fully) implemented.
Completed AIPs
Title | Discussion Thread | In Release | Created |
---|---|---|---|
AIP-3 Drop support for Python 2 | [DISCUSS] AIP-3: Drop Python2 support in Airflow 2.0 | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-4 Automation of System Tests [Deps: AIP-47] | Support for integration tests in Airflow | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-6 Enforce the usage of a code formatter & linter | AIP-6: Enforce the usage of a code formatter | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-7 Simplified development workflow | AIP-7 Simplified development workflow | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-8 Split Providers into Separate Packages for Airflow 2.0 | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-9 Automated Dependency Management | AIP-9 Automated Dependency Management | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-10 Multi-layered and multi-stage official Airflow CI image | Mutli-layered official image for Airflow | 1.10.4 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-11 Create a Landing Page for Apache Airflow | [PROPOSAL] Add a landing page for Apache Airflow | 1.10.7 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance | Multiple Schedulers - "scheduler_lock" A Naive Multi-Scheduler Architecture Experiment of Airflow [PROPOSAL][AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance] | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-16: Use nested commands in CLI | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor | [Discuss] Airflow sensor optimization | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-21: Changes in import paths | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-23 Migrate out of Travis CI | 1.10.10 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-24 DAG Persistence in DB using JSON for Airflow Webserver and (optional) Scheduler | Airflow DAG Serialisation | 1.10.7 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-26 Production-ready Airflow Docker Image | AIP-26 Discussion | ||
AIP-27 Migrate to pytest | 1.10.7 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-31: "TaskFlow API" for clearer/simpler DAG definition | 2.0.0 | 2020-02-21 | |
AIP-32: Airflow REST API | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-33 Secrets backend | [DISCUSS] AIP-33 secrets backend | 1.10.10 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator | 2.0.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-39 Richer scheduler_interval | Scoping out a new feature for 2.1: improving schedule_interval [DISCUSS][AIP-39] Richer (and pluggable) schedule_interval on DAGs | 2.2.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-40: Deferrable ("Async") Operators | [DISCUSS][AIP-40] Deferrable ("Async") Operators | 2.2.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-42 Dynamic Task Mapping | https://lists.apache.org/thread/xhq5o3b3h879cwsc93xwq0836mwc5k6p | 2.3.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-43 DAG Processor separation | https://lists.apache.org/thread/rgovdphjrxtcqfmcfdrbfntwrdr2nj71 | 2.4.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-43 DAG Processor separation | https://lists.apache.org/thread/rgovdphjrxtcqfmcfdrbfntwrdr2nj71 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-48 Data Dependency Management and Data Driven Scheduling | https://lists.apache.org/thread/vnjv32zv3fqqp92t5nqf0qqy03xphwm8 | 2.4.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-49 OpenTelemetry Support for Apache Airflow | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | ||
AIP-51 Removing Executor Coupling from Core Airflow | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | ||
AIP-53 OpenLineage in Airflow | 2.7.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-58 Airflow ObjectStore (AS) | 2.8.0 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
"Umbrella" AIPs
AIPs hat are too big to be treated as "deliverable" AIPs on its own - they are split into smaller AIPs.
Title | Discussion Thread | State | In Release | Created |
---|---|---|---|---|
AIP-1: Improve Airflow Security [Deps: AIP-43] | Superseded by AIP-43, AIP-44, AIP-67 & AIP-72 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | ||
AIP-4 Automation of System Tests [Deps: AIP-47] | Support for integration tests in Airflow | Completed | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-63: DAG Versioning | Draft | |||
AIP-73 Expanded Data Awareness | Accepted |
Abandoned AIPs
Title | State | Discussion Thread | Created |
---|---|---|---|
AIP-1: Improve Airflow Security [Deps: AIP-43] | Superseded by AIP-43, AIP-44, AIP-67 & AIP-72 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-2 Simplify process launching | Draft | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-5 Remote DAG Fetcher | Superseded by AIP-66 | ||
AIP-12 Persist DAG into DB | Abandoned in favour of AIP-24 | [DISCUSS] AIP-12 Persist DAG into DB | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-13: OpenAPI 3 based API definition | Abandoned in favour of AIP-32 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-14 Create composable operators using common interfaces | Superseded & Completed by AIP-58 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-18 Persist all information from DAG file in DB | Abandoned in favour of AIP-24 | TBD | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-19 Making the webserver stateless | Abandoned in favour of AIP-24 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-20 DAG manifest | Superseded by AIP-66 | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-22: Group ORM models by their logical usage instead of type | Superseded by AIP-72 due to Task Interface. If someone has time, it can be PRs, doesn't need AIP | https://lists.apache.org/thread.html/fc21e13413a4a11b2c490b0e2f59dd83239711433599fe8693b98d02@%3Cdev.airflow.apache.org%3E | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-25 The Knative Executor | Abandoned | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-28: Add AsyncExecutor option | Abandoned in favour of AIP-40: Deferrable ("Async") Operators | apache-airflow.slack.com - sig-async-operators | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-29: AWS Fargate Executor | Abandoned in favour of AIP-51 | ||
AIP-37 Virtualenv management inside Airflow | Draft | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) | |
AIP-46 Runtime isolation for airflow tasks and dag parsing | Replaced by AIP-72 and some parts of AIP-67 | https://lists.apache.org/thread/9kw13cxcg06p37shv57hsomnx6zognoc | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
AIP-55 Rule-based timetable with logical composition | Abandoned | ||
Deprecated- AIP-5 DagFetcher | Abandonded in favour of new https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher | $action.dateFormatter.formatGivenString("yyyy-MM-dd", $content.getCreationDate()) |
(The above list is automatically generated based on page labels. Do not edit manually)
5 Comments
Ash Berlin-Taylor
If you create the AIP from the button it should add the template automatically. (I thought)
We'll work "acceptance" process when our first AIP is voted on and accepted
Ash Berlin-Taylor
Ah, the template was missing the draft label. It has it now.
Michael Ghen
Great stuff here, I really like this AIP process, I don't have the karma to make edits to these pages but we have a similar process internally and the page properties macro is awesome for this: https://confluence.atlassian.com/doc/page-properties-macro-184550024.html, I'd be happy to make an edit here to demo how using this macro would look, I think if you read the link it'll make a lot of sense how the macro would work here
Ash Berlin-Taylor
Added permissions for you
Chen Tong
Since AIP-12 Persist DAG into DB has been abandoned, maybe create an abandoned section and move it there?