Airflow 3 Dev call: Meeting Notes

NothiThe dev calls are open for anyone in the community to join and help in the Airflow 3 efforts. 

Schedule: Fortnightly (once in 2 weeks) on Thursday at 4pm GMT/UTC

Calendar:

Table of Contents

4 June 2024

Agenda

Attendees

Summary

The following principles were agreed upon to drive Airflow 3 development:

• For the features that require breaking changes, ship Airflow 3 with the foundational code to allow for iterative development, optimizing for speed and a quicker feedback cycle.
  • Additional features that do not require breaking changes can be included in minor releases such as 3.1, 3.2, and beyond since we follow SemVer.
  • Discussion points:
    • Examples:
      • Add the hook points with a few references (fetcher for GCS, S3, Git) for DAG Versioning's Execution AIP (AIP-66)
      • Removing dependency on FAB in Core: The new plugin framework might support only a few functionalities, which are then built upon later.
      • For the multi-language AIP, start with only Python + one more language in AF 3.0 and then 3.1 and later minor versions can have more support for language like Typescript
    • Identify users who would be willing to give feedback during dev and beta snapshots. The Astronomer, MWAA, and GCC teams can help identify these.
• Ensure a smoother migration path between Airflow 2 and 3, particularly for DAG authors using the existing official Airflow providers.
  
  • Directionally the time required to update DAGs should be measured in hours, not days or months. 
  • Action Items:
    • Update the AIP template to ask for the level of effort (manual and automated) needed for the users to adapt to the breaking changes. This should include high-level details on what could go in the upgrade utilities. This will help the AIP authors consciously think through the migration efforts. (Kaxil Naik )
• For AIPs, be explicit about what's for AF 3.0 and what's for the next minor releases (3.1, 3.2, ..)
  • Action Items:
    • Complete the housekeeping of the AIPs (Kaxil Naik ):
      • Move the AIPs that we don't plan to work on in Abandoned state
      • Add labels if an AIP is for Airflow 2, 3.0 or >=3.1. These labels will be used via Macros to auto-populate the tables in Airflow Improvement Proposals, making it a good page for Roadmap items
• Build features that solidify Airflow as the modern Orchestrator that has state-of-the-art support for Data, AI & ML workloads.
  • This includes enhancing scalability, performance, and enterprise-level security, adhering to the principle of least privilege.
  • Making Airflow aware of what's happening in the task to provide better auditability, lineage & observability
• Set up the codebase for the next 3-5 years.
  • Reducing matrix of supported combinations for reducing complexity in testing & development. E.g Remove MySQL support to reduce the test matrix
  • Simplifying codebase & standardize architecture (e.g consolidating serialization methods)
  • Remove deprecations
  • Consider optimizing development workflows (core airflow vs. provider, chart development)
• Simplify the Learning Curve for new Airflow users
  • Decrease time from running the install command to first DAG
  • Decrease the boilerplate code needed to run first dag/task
  • Action Items:
    • Write a first draft of a doc on the different personas of Airflow users & current state. This will help tailor the learning curve via docs & tutorials as well as tailor features towards that persona. (Elad Kalif )
• Shift focus on Airflow 2 to stability: bug fixes + security fixes after AF 2.10. This should continue for a longer period of time after AF 3 release
  • The provider release will continue to happen independently of the core Airflow.
  • Discussion points:
    • After the AF 2.10 release (~Aug), the "main" branch will become Airflow 3, and the release manager will cherry-pick anything targeting Airflow 2 into the Airflow 2 release branch. 
    • The primary focus for AF 2 will shift to reliability. If certain features need to go in AF 2, they will be done by cherry-picking or opening PRs targeting the AF 2 branch.
• Target a shorter cycle to release Airflow 3
  • so that Airflow 2 branches for features don't diverge
  • users have enough time between Airflow 3 release and Airflow Summit 2025, so we can have talks about Successful migrations
  • Discussion point:
    • This means the realistic target is March-April 2025. This is to allow users enough time to migrate to AF 3 and use it so they can submit CFPs around May-July 2025.
    • The focus for AF Summit 2025 would be on Migration stories & features of Airflow 3

The following Guidelines were agreed upon that helps decide if a feature should be in Airflow 3 or not.

• Alignment with Core Principles (mentioned above)
• Workstream Ownership (can be more than one). If no one is available to lead the workstream, the feature will be parked until a dedicated owner is found
• Community Demand and Feedback
• Impact on Scalability, Performance & Security 
• Backward Compatibility and Migration Effort
• Implementation Complexity and Maintenance
• For big features, discussion on AIPs & a successful vote on the dev mailing list 

Next steps

• Create AIPs for the features targeted for AF 3 in the next few days to start technical discussions. 
• Finalize the agenda for the next call

13 June 2024

Agenda

Attendees

Summary

• Drafting proposals & getting feedback
  • The team discussed the challenges of Apache Confluence access and considered using Google Docs for initial feedback before moving to Confluence.
  • The general consensus was that developers could utilize the tool of their choice to get initial feedback (on their high-level draft) as long as the final AIP is published on Airflow's Confluence space for AIPs.
  • If someone does not have an account on Apache Confluence to comment on existing proposals or create a new proposal, reach out on #airflow-3-dev Slack channel.
• Personas for the Airflow User
  • Elad prepared a document on Airflow Actors that the group discussed to get initial feedback. Several good suggestions were made, so it was decided to take the discussion offline/async and continue the discussion on that page. 
• Task Context draft AIP
  • Ash presented the initial draft of the Task Execution InterfaceContext AIP (formerly known as Task Context SDK). 
  • The draft AIP is now published at AIP-72 Task Execution Interface and ready for review, although some technical details still need to be worked out.
• Frequency of the Next Dev calls
  
  • The general consensus was to make the dev calls weekly for the next 2-3 weeks
• Timeline to release Airflow 3
  • Based on the discussions on the first dev call, we also agreed that we should target March-April 2025 to release Airflow 3.
  • We will announce the dates and discuss some of the feature sets with the community at the Airflow Summit in September this year.
• Action items
  • Docs needing reviews:
    • AIP-72 Task Execution Interface
    • Enhanced Data Awareness proposal
    • Airflow Actors
  • Updating AIP template & Housekeeping of AIPs & label additions (Kaxil Naik )

20 June 2024

Agenda

Attendees

Summary

• The team reviewed the existing status of AIPs and marked them for AF 3.0, 3.1+ and AF 2 as needed. Here is the summary:

  AIP	Decision
  AIP-1	Superseded by AIP-44 & AIP-67. Close it
  AIP-5	Superseded by AIP-66
  AIP-14	Mark as completed by AIP-58
  AIP-20	Superseded by AIP-66
  AIP-22	Superseded by AIP-72 due to Task Interface. If someone has time, it can be PRs, doesn't need AIP
  AIP-30	Airflow >= 3.1
  AIP-35	Abandoned
  AIP-38	Keep it for 3.0. Brent will update this. There was an Ask about expanding the Plugin migration strategy.
  AIP-44	Replaced by AIP-72 for AF 3, and we can ship what we have in main for 2.10 as "experimental" and stated to change for AF 3
  AIP-46	Replaced by AIP-72 and some parts of AIP-67
  AIP-49	Phase 1 is in 2.10 and Phase 2 will be in 3.1 and above unless Howard Yoo  wants to lead and implement it for AF 3.0
  AIP-51	Marking it for Airflow 2.10. Niko Oliveira finishing up the last issue.
  AIP-55	Abandoned
  AIP-57	Airflow 3.0, Shubham owns it. There is a PR already with this comment contesting the proposed solution. We will review this again in few weeks
  AIP-59	Bartosz is working on it. Is OOO right now but will resume once back. Will be used in AF 2 & AF 3 to compare performance. Will target it for AF 2
  AIP-61	Marking it for Airflow 2.10. Niko Oliveira to get it done in the next few weeks
  AIP-62	Marking it for Airflow 3.0 for now. But we will review it again in a couple of weeks; Maciej Obuchowski owns it.
  AIP-64	Airflow 2.10
  AIP-65	Airflow 3.0
  AIP-66	Airflow 3.0
  AIP-67	Airflow 3.0
  AIP-68	Marking it for Airflow 3.0 for now, but it will be re-evaluated when Jens Scheffler creates a VOTE thread after discussing with Brent. Concern for keeping this for AF 2.10 was around adding a React Plugin interface that will need changing creating a migration pain.
  AIP-69	Tentatively marking it for 3.0. Jens Scheffler has a PR that needs review. The concern for keeping this for AF 2.10 even if most changes are in providers is that it builds on top of AIP-44 which will be experimental and replaced in AF 3. Will be reviewed in a few weeks
  AIP-70	Airflow 3.1+
  AIP-71	The general consensus was against this AIP and favouring the DAG bundles concept that is part of AIP-66.
  AIP-72	Airflow 3.0

• The team also decided to review the Airflow 3 Workstreams doc asynchronously. If the reviewers are interested in leading any workstream without an owner, they should add their names in the Owner column.
  • Other Action items:
    • Review SLA PR
    • Jens Scheffler to talk to Brent for AIP-68
    • Review PR from Jens for AIP-69 (Remote Executor)
    • Complete AIP-51 & AIP-61 for Airflow 2.10
    • Workstream owner should Start AIP or draft AIPs for items in Other Candidates items (under AF 3.0 section) in Airflow 3 Workstreams

27 June 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • Kaxil Naik updated the AIP template to include the "Migration effort" section. If the template is used to create a new AIP doc (click "Create" → "Airflow Improvement Proposal" on the wiki) it should pre-populate sections. The access to editing the template itself is limited to PMC members & ASF Confluence Admin group. Need to retroactively add that section in existing AIPs planned for 3.0
    
  • SLA PR: This is still pending review.
  • Complete AIP-51 & AIP-61 for Airflow 2.10
    • Marked AIP-51 as completed, and the pending UI issue will be covered in the UI refactor.
    • AIP-61: Niko Oliveira is still targeting 2.10 for completion. PRs are raised & pending reviews: #40472 (Backfill part) & #40017 (Scheduler part).
• Discuss: Workstreams & workstream owners (Airflow 3 Workstreams)
  • Updates on the existing AIPs for the 3.0 list
    • AIP-66 (DAG Versioning—Execution side): Jedidiah Cunningham will have the draft by the next dev call.
    • AIP-72 (Task Execution SDK) : No updates this week. Ash Berlin-Taylor aims to have the VOTE open for it by the next dev call.
    • AIP-38 (UI Modernization): Brent Bovenzi is focussing on Airflow 2.10 deliverables; post that, he will focus on replacing this AIP with a new Umbrella AIP. There was a consensus on the call that we should split this AIP into multiple AIPs (with an Umbrella AIP) as below:
      • React'ifying the UI & changing the UX required for a modern Webserver
      • FAB removal: The removal of Flask Appbuilder will have backend implications such as changes needed to Authentication (currently, it is done via FAB Auth backends.), changes to the Plugin interface, custom API endpoints etc. 
      • New Features like DAG Folders/DAG Groups
    • AIP-57 Refactor SLA Feature: The initial AIP was created with 2.x in mind, so it should be revised. Some things might be easier with 3.0, as we can make breaking changes. One of the contention points last time was around whether the SLA duration time should start from DAG/task start or should be based on absolute time, as reflected in this PR comment. Shubham Mehta and Kaxil Naik will reach out to Sungwon Yun to see if he is interested in being part of this effort.
    • AIP-65 (DAG Versioning—UI side): Before the next dev call, Jedidiah Cunningham will decide whether to keep this AIP or make it part of any existing DAG Versioning or UI Modernization AIP.
    • AIP-67 (Multi-team): Jarek Potiuk will revise this AIP based on all the discussions and new AIPs to present the overview in the next dev call. Shubham Mehta and his team are planning to co-own/contribute to this AIP with Jarek.
    • AIP-68 (Extending plugin interface): Jens Scheffler & Brent Bovenzi will create a POC PR to determine the direction to take. Jens will do it before the next dev call to then get the final feedback on 2.10.x vs 3.0. 
    • AIP-69 (Remote Executor): Jens Scheffler organized a call with interested contributors a day after the dev call to get feedback. A summary of it is posted here.
  • Updates on the "other candidates for 3.0" list:
    • Enhanced Data Awareness: Tzu-ping Chung and Constance Martineau created draft AIPs (AIP-74, AIP-75, AIP-76, AIP-77) for this workstream (under Umbrella AIP: AIP-73). These AIPs are marked for Airflow 3.0 but AIP-77 might move to Airflow 3.1, TBD for now. 
    • Scheduler-Managed Backills (aka Backfills at scale): Daniel Standish has an internal draft for it that will be published around next week.
    • Poll external Datasets to have event-based DAG scheduling: Vincent BECK volunteered to own this epic for 3.0. He and Shubham Mehta plan to publish a draft AIP in the next 2 weeks.
    • Dennis Ferruzzi has volunteered to own the "Inspect & Simplify Airflow Configurations" & "Inspect & Revamp the cardinality of Metrics" streams.
    • Synchronous DAG Execution: No update this week but targeting a proposal after the next dev call
    • Make Execution Date non-unique for a DAG: No update for now; TP is focussing on the Data Awareness AIPs.
    • Scheduler Performance Improvements: Some areas in the scheduler have been identified for improvements. If anyone has any specific areas of the Scheduler loop that they would like to own, please add it to the list on Airflow 3 Workstreams. The performance goals would be tied to the specific area of optimization rather than the scheduler as a whole.
    • Consider Developing Airflow and Providers separately. In the coming weeks, Kaxil Naik will draft a pro-cons list in a document to send to the mailing list around this epic.
    • Respect permissions in CLI: Buğra Öztürk has volunteered to own this stream and, ideally, will start working on a proposal next.
    • Improve security of Airflow Supply Chain: Once things are more defined, Jarek Potiuk will send a document about this stream of work describing its impact on dependencies in the coming month. 
    • Observability of Callbacks on UI: Some of this might be covered in AIP-72 & AIP-69 but we will keep this item in the table so we don't forget about it.
    • Remove StatsD and replace it with Prometheus as a first-class citizen: We agreed that we should replace StatsD with OpenTelemetry and not Prometheus. It needs an owner who could figure out the impact of breaking changes since StatsD metrics are utilized by all users, including Airflow service providers, to monitor and alert on the health of Airflow deployments.
    • Overhaul Operator Templating behaviour: Tzu-ping Chung mentioned he would be happy to write the design document & asked for someone to work on the implementation. Shahar Epstein expressed his interest in working on this effort. 
    • Shubham Mehta expressed interest in working on a native DAG Factory that is currently on the 3.1 list. We would keep this for 3.1 for now and reevaluate based on the progress of other epics.
    • There have been no noteworthy updates on any other streams. Kaxil Naik will create GitHub issues for items in that table with no owners and post them on the mailing list to see if anyone is interested in leading it. If there are no takers by the end of August, they will be moved to the 3.1+ list.
• The next dev call will be on 11 Jul 2024.

11 July 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • AIP-61 (Hybrid Executor): On track for Airflow 2.10.
  • Kaxil Naik  deferred Creating GitHub issues for items in the Workstream table with no owners to next week.
• Check-in: Workstreams & workstream owners (Airflow 3 Workstreams)
  • AIP-68 (Extending plugin interface): This is finalized for 3.0 instead of 2.10
  • AIP-38 (UI Modernization): Brent Bovenzi presented the progress of AIP-38 in the Airflow 2.3, 2.4, 2.7 & 2.8. An update on what's pending has been added to the AIP-38 page.
    • A new AIP was split from AIP-38 to replace FAB: AIP-79.
    • The new Plugin interface for UI will be defined by AIP-68
    • Brent Bovenzi  & Jens Scheffler : Could either of you add what you presented for AIP-68 in the AIP doc, around breaking changes to existing plugins & CLI to initialize a starter project?
    • Risk / Action Item: There were discussions around the impact on Auth due to the removal of FAB. KeyCloak Auth Manager was mentioned as a potential replacement for Enterprise installation and using FAB Airflow provider for local development. Fab auth manager via FAB provider might be a fallback until we get the Keycloak implementation. We should test this to make sure that it doesn’t become a blocker and that FAB provider is a reliable fallback. This should be noted in AIP-79.
  • AIP-67 (Multi-team): Jarek Potiuk presented the revised AIP. The AIP has been updated, and Jarek has answered the questions that were asked in the chat in this mailing list thread.
    • As a follow-up to the call, Jarek also created a mailing list thread about whether to finish AIP-44 for AF 2 or not.
  • AIP-62 (Lineage from Hooks): Maciej Obuchowski presented current progress and is targeting Airflow 2.10 for its completion from the Airflow core side, the rest will be in Providers.
  • AIP-57 Refactor SLA Feature: Kaxil & Shubham both heard back from Sungwon Yun. Unfortunately, he doesn't have the capacity to work on it, but he is going to meet Shubham Mehta and the team for knowledge transfer.
  • AIP-65 (DAG Versioning—UI side): Jedidiah Cunningham decided to keep this AIP as is instead of making it part of the UI Modernization AIP.
  • AIP-66 (DAG Versioning—Execution side): Jedidiah Cunningham marked it as ready for review. The email thread is here.
  • AIP-69 (Remote Executor): Jens Scheffler wants to start voting, but it is dependent on AIP-44, which is discussed separately here.
  • AIP-72 (Task Execution SDK): VOTE has succeeded as of July 16.
  • AIP-73 (Data Awareness):  Tzu-ping Chung & Constance Martineau plan to remove AIP-74 (Introducing Data Assets) & AIP-75 (New Asset-Centric Syntax) from the draft in a week and timelines for the rest to follow.
  • AIP-78 (Scheduler-managed backfill) was published by Daniel Standish and is under-going discussion.
  • "other candidates for 3.0" list:
    • Poll external Datasets to have event-based DAG scheduling:  – This will become an AIP, Vincent BECK / Shubham Mehta working on the draft AIP.
    • Respect permissions in CLI: Buğra Öztürk is working on the first draft, timelines TBD.
• The group collectively agreed that all the big AIPs targeting Airflow 3.0 should at least have the vote started before August 1. Anything after that does not have a realistic chance of completion. This is to not rush the AIPs but to prioritize the key items, especially the ones that contain breaking changes, get them vetted out properly, and make them better for the initial MVP launch.

25 July 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • Kaxil Naik created GitHub issues for workstream items with no owners; they have also been added as sub-items in this meta GitHub issue that tracks the 3.0 release. We had already had some interest from community members on some of these items, check individual issues for it.
• Discussions:
  • Branching Strategy:
    • As soon as 2.10.0rc1 is created, the Airflow main branch will become the Airflow 3 branch on 09 Aug 2024 .
    • Providers will continue to be developed on the main branch since we have the framework to run provider tests against older Airflow versions.
    • Bugfixes: If there is a PR with a bugfix to main, the committers should ask the PR author to raise a PR against v2-10-test  branch too. The team agreed to create a mailing list thread to discuss different approaches offline. The thread has been initiated here.
  • Which Python version should we support in Airflow 3?
    • The consensus was to keep things as-is i.e following the current Python-support policy where we would remove the support for a Python version when it is EOL.
  • Airflow 2 support policy including scope (feature vs bug fixes + security only) & support period
    • We might release 2.11 and mark it as a bridge release to add compatibility shims or deprecation warnings similar to 1.10.15. But we will decide on it as and when such a need arises.
    • For providers, we will need to increase the support window for the minimum Airflow version from the current 6 months. We will continue Airflow 2 provider support for a year or longer, exact period TBD. We will revisit this again in the coming months. If we cut 2.11 (or whatever the final 2.x version is), then after 6-12 months we could increase the minimum Airflow to 2.11. There were ideas around using a "compat" provider that might help with AF 2 and AF 3 stuff. This will be another thing we will discuss async at a later time to keep current focus on scope & AIPs.
• Workstream statuses for ones that does not have a vote started:
  
  • AIP-57 Refactor SLA Feature (Shubham Mehta):
    • It was agreed that for now, we will mark it for removal in Airflow 3.0 and adding it back in 3.1. We will assess again in the coming dev calls if something changes. Elad has also offered to help if needed.
  • AIP-67 Multi-team deployment of Airflow components (Jarek Potiuk ):
    • Jarek is waiting to hear back on his answers to some of the doubts raised in the discussion thread and AIP. This might also be moved to Airflow 3.1 since it depends on AIP-72 and AIP-66 and the implementation hasn't been started on those yet.
  • AIP-68 Extended Plugin Interface for React Views (Jens Scheffler / Brent Bovenzi ):
    • As of today, the vote has been started on the mailing list (link). During the meeting, a few things were discussed, including potentially renaming UI plugins to UI Extensions as the "Plugin" word has been overloaded. The other suggestion was to rename the AIP title to "New Interface" instead of "Extending the interface."
  • AIP-76 Asset Partitions (Tzu-ping Chung):
    • TP will send this out for VOTE in the next couple of days.
  • AIP-79: Remove Flask AppBuilder as Core dependency (Brent Bovenzi ):
    • Two big questions pending:
      1. Replacement for FAB Auth manager
      2. How does adding a FAB as Provider work? Can it allow users to make older plugins work?
    • Jed has been working on a POC on it. He is on vacation this week, but based on initial investigations, he was positive about the progress. We will check it back in the next dev call when he is back.
    • The pending item is also to update the AIP to add the decoupling of Providers's Connections metadata & form from FAB.
  • AIP-80 Explicit Template Fields in Operator Arguments (Tzu-ping Chung ):
    • VOTE has been kicked off here.
  • Poll external Datasets to have event-based DAG scheduling (Vincent BECK / Shubham Mehta): Draft AIP has been created as mentioned in this mailing list thread.
  • Respect permissions in CLI: Buğra Öztürk 
    • It now has an AIP-81 Enhanced Security in CLI via Integration of API and is awaiting feedback.
  • Returning DAG Results / Synchronous DAG Execution: Vikram Koka 
    • The first proposal (Make Execution DAG non-unique) from the original doc is separated out into a different workstream led by TP.
    • Language support has been excluded from the scope for Airflow 3.0. For now, the only change considered would be to add the support to the existing Trigger API to return an async result which doesn't need an AIP.
    • The other changes for GenAI will now be implemented as a new provider.
  • Make Execution Date to be non-unique for a DAG: Tzu-ping Chung 
    • In the next couple of days, a discussion thread with a draft AIP will be created by Tzu-ping Chung , followed by a vote.
• Splitting providers from the mono-repo (Kaxil Naik)
  • Kaxil Naik is not going to propose this anymore as he agrees that doing this now will increase the work and split the focus of folks. So we are going to NOT split providers from mono-repo to laser focus on Airflow 3 feature work.
  • We might think about it at a later date after Airflow 3.0 is released.
• AIP-49 OpenTelemetry Support for Apache Airflow:
  • The PR for the 2nd part got merged for 2.10 and the AIP is marked complete.
  • Howard Yoo 's comment: Initially we thought to collect logs in standard way (scraping from log files), but found a much better way to collect logs, by utilizing log api to get the task logs, and then attaching them as 'span events' which would be much more useful. The extraction of Task Logs and attaching them as span events is thus included in the AIP-49, thus it is not 'excluded', albeit not as a separate 'log' data as originally thought.
• StatsD vs Open Telemetry
  • There was a discussion on whether we should mark StatsD as removed in Airflow 2.10 itself in the docs here. The conclusion was that we should discuss this more as this has a wider impact including but not limited to the official Helm Chart which uses StatsD, the services providers (Astronomer, Composer, MWAA) and other long-time Airflow users. OTel also had limitation around character length limit and that you have to use tags with OTel but it didn't allow aggregating metrics with different tags etc. Shubham Mehta is going to check this out with his team and see if there are other concerns.
• Triage Help needed: There are 360 open feature requests. Some we might want to consider and seek help for in 3.0, and others we should close as not planned. Elad, Jens, Jarek, Vikram & Kaxil have volunteered to look at those issues. Elad is also going to check with Dosu team to see if we can re-label these old issues so they are more accurate. Categorizing these issues into "areas" label was one of the next steps too so that the volunteers can look at a particular area or some segment to divide & conquer. Elad Kalif could you take the one on for issue-triage?

8 August 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • AIP-79: Remove Flask AppBuilder as Core dependency: 
    • Jedidiah Cunningham  does not have a working POC yet. He will provide an update on the next call.
    • Jens Scheffler will draft a high-level plan for Decoupling Providers's Connections metadata from FAB after he is back from vacation in the next 2 weeks.
  • Airflow 2.10rc1 & 3.0 branching check:
    • The main branch became the Airflow 3 branch on 09 Aug 2024 as planned. 
    • Guidelines around it have been documented in this PR and here.
  • StatsD vs Open Telemetry:  Shubham Mehta mentioned that the AWS team has not encountered any issues so far, but the POC is yet to be completed. He will update about it on the next dev call.
• Discussions
  • Addressing Active/Paused DAG terminology for Airflow 3.0: The Consensus was to change "Active" to a better alternative. This GitHub issue should now cover it and is open for anyone to grab it.
  • Impact on Airflow 3 development while new UI is being built (Brent Bovenzi)
    • There will be two UIs: the existing UI and the new React UI. Brent Bovenzi  will write a "UI dev practices" in the Contributing guide for someone wanting to try out a new UI. The guide will also include details like ensuring that all new features go to the new React UI.
  • Should we split Webserver, SDK, CLI, Scheduler, and Triggerer into separately installable packages for the users, while the dev code is in same mono-repo? (Jarek Potiuk )
    • The consensus was to wait until there is decent progress on AIP-72 implementation, but there was a general agreement on splitting the packages out while keeping apache-airflow as a meta-package that brings it all together. The intention is not to have a separate version scheme or release these separately however unlike Providers.
  • Library choice for different Airflow APIs (API used internally by Webserver, AIP-72 and external API) (Jarek Potiuk)
    • The consensus was to use Fast API and generate spec from the code. This will mean we remove Gunicorn from the Webserver, which will be a breaking change, but most folks agree that it is worth it.
    • This will be tracked as part of AIP-84 UI REST API. Buğra Öztürk has also volunteered to help
  • AIP-80: Operator Templating 
    • Some of the attendees strongly opposed breaking Airflow 2's behaviour because they feared it would cause migration issues and that migration tooling wouldn't be sufficient. So, for Airflow 3, we will continue with the existing templating behaviour and add the new (and explicit) feature proposed in the AIP-80. After implementation, if TP finds a good 1-command migration for this, he will propose it on the mailing list with a demo to showcase the migration effort (with 100s of DAGs) to convince that we can break this behaviour but until then the default answer here is, for now, the decision is to keep the Airflow 2 templating behavior in Airflow 3.

22 August 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • AIP-79: Remove Flask AppBuilder as Core dependency: 
    • Jed made good progress on a working POC – he was able to pull out FAB from the core easily; he is going to verify if he didn't miss anything. He will provide an update on the next call.
    • Jed , Vincent  & Jarek  have also been discussing what Auth manager we should have for Core. Vincent's plan is to have a simple auth manager for local installs, and this will be driven via an Airflow config. He will build one for production using Casbin or Keycloak after the Airflow Summit. Currently, his team is doing research on which would be better.
      • Action Item: Vikram to get back to Vincent or Shubham on why the Astronomer team didn't use Casbin. This can also wait for an in-person chat during the Airflow Summit.
    • Elad raised a concern about this PR that separates FAB migrations from Core. His concern was that this was the first time we would be doing a migration from a provider, so we should think about the messaging to the users via docs (both Airflow & FAB provider) because up until now, we tell users to just upgrade provider dependency without thinking of DB migrations. Ash had an idea to maybe call the FAB thing as a Plugin instead of a provider, we decided to discuss this async.
      • Action Item: Ephraim  will send a mailing list thread about how we plan to handle messaging around DB migrations from FAB and if we need to discuss Ash's idea about calling FAB a plugin rather than a provider. 
  • Update on development practices/guide while the new UI is being built (Brent )
    • Brent proposed 3 guidelines to help committers accept changes to legacy UI until the new UI is ready to be merged.
      1. Bugfixes that can be cherry-picked for 2.10.x, 2.11.
         1. It is fine to accept these changes
      2. Airflow 3.0 features
         1. Add minimum changes necessary to unblock dev work without adding a ton of FAB views which will be deleted anyway once new UI is ready
      3. Fixes or Updates to existing react views 
         1. It is fine to accept those changes. The changes will be ported over to the new UI.
    • Brent proposed that the same rules should apply to non-public API endpoints for UI.
    • The PR to add this new airflow/ui  project has been created.
    • Pierre Jeambrun has joined forces with Brent to work on the UI pieces.
    • Action Item: Brent  will write up the guidelines in the docs and send an email with the link to PR on the dev mailing list.
  • Plan for Decoupling Providers's Connections metadata from FAB (Jens Scheffler)
    • Jens created this draft PR with the POC for it and presented it on the call.
    • Jarek proposed the idea of dumping the JSON/YAML with connection fields in the Database or loading it via package metadata so we don't load all the dependencies on the webserver.
    • We will need some plan for external providers on how they can define connections or register them.
    • The POC successfully proved that we can separate the connection metadata from FAB
    • Action Item: Jens to create a GitHub issue for decoupling the Connection metadata from FAB
  • Open Telemetry as a StatsD replacement POC Update (Shubham  / Dennis ) 
    • POC is working fine and the AWS team were able to use Open Telemetry with Cloud Watch. There were some duplicate metrics (based on name, length & limitation) that they will clean up as they decide to move.
    • Howard Yoo gave a presentation on OTel support in Airflow. Recording details below:
      • Zoom Recording Link | Passcode: OXzbxW+9
    • Action Item: This discussion will be continued on the dev mailing list. Kaxil to create a thread on the mailing list about OTel vs StatsD to discuss if we are ready to deprecate or remove StatsD support or just make it non-default in favour of OTel. 
• Discussions
  
  • Airflow 3 development guideline review / Airflow 2.11: Bridge release (Elad Kalif)
    • We discussed two open issues:
      1. How are we going to manage milestones? There will be two PRs: one against main and one against v2-10-test. In some cases, which milestone should we use on the original PR and the backport PR?
         1. We agreed that for now, we will only add milestones to the original PR (targeting the main); for example, if the original PR targets the main but is intended for 2.10.1, it should have a 2.10.1 milestone.
      2. How are we backporting/cherry-picking?
    • We need to remove features from 2.11 GitHub milestone 
    • Action Item: Kaxil to create a lazy consensus email as a follow-up to this discussion about Airflow 2.11 and delete all features from 2.11 GitHub milestone 
    • For PRs that just have deprecations changes, we should just merge them to release in the next 2.10 patch release so the users are made aware of it asap so they can start planning. When we have a change that is targeted for the 2.11 bridge release, we will discuss how to handle it since it won't be merged to v2-10-test. Possibly, we would create a v2-11-test  branch from v2-10-stable later this year when we are closer to have most of the AIPs on the finishing lines.
  • More help needed on our CI / infrastructure (Jarek Potiuk )
    • Hussein Awala  presented the Architecture of the current CI
      
    • Shahar Epstein & Buğra Öztürk volunteered to work on CI issues.
  • New code layout for core/providers/task-SDK (Ash Berlin-Taylor )
    • We ran out of time on this one but Ash has started a thread here for discussion.
• Consolidated Action items list:
  • Vikram to get back to Vincent or Shubham on why the Astronomer team didn't use Casbin. This can also wait for an in-person chat during the Airflow Summit.
  • Ephraim  will send a mailing list thread about how we plan to handle messaging around DB migrations from FAB and if we need to discuss Ash's idea about calling FAB a plugin rather than a provider. 
  • Brent  will write up the guidelines in the docs and send an email with the link to PR on the dev mailing list.
  • Jens to create a GitHub issue for decoupling the Connection metadata from FAB
  • Kaxil to do some homework on OpenTelemetry support so we can confirm an opinion to keep or drop StatsD support when we make OpenTelemetry the default
  • Kaxil to create a thread on the mailing list about OTel vs StatsD to discuss if we are ready to deprecate or remove StatsD support or just make it non-default in favour of OTel. 
  • Kaxil to create a lazy consensus email as a follow-up to this discussion about Airflow 2.11 and delete all features from 2.11 GitHub milestone 

5 September 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • Vikram to get back to Vincent or Shubham on why the Astronomer team didn't use Casbin. They will discuss it during the Airflow Summit.
  • Ephraim  created a thread here to discuss the messaging around DB migrations from FAB
  • Brentwrote up the guidelines in the docs and sent an email here to the dev mailing list.
  • Jens created a GitHub issue for decoupling the Connection metadata from FAB and potentially from provider dependencies
  • OTel & StatsD
    • Dennis has started a thread here about StatsD/OTel metric naming.
    • After the summit, Kaxil will need to create a thread on the mailing list about OTel vs StatsD to discuss whether we are ready to deprecate or remove StatsD support or just make it non-default in favour of OTel. 
  • Kaxil created a VOTE thread here about Airflow 2.11 being a bridge release and deleting all features from the 2.11 GitHub milestone 
• Presentations
  • Jarek  gave us a sneak peek of his Airflow Summit keynote talk on Security: "Airflow Beach Cleaning" project – Security work
  • Pierre presented the new UI Rest API to explain how it works, how to develop it, and how to migrate from Connexion to Fast API
  • Kaxilpresented the Survey around Debugging Improvements created as part of this GitHub Issue by Iliya Romm, Omkar & Amogh.
• Discussion about deprecation/removal of Core_operator functions if a provider package should be cut  (Elad Jens  Jarek Ash  )
  • We covered Airflow Standard Provider & deprecations (Rom Sharon ) as part of it.
  • The standard provider will contain "directly usable in DAGs" Operators, Hooks, Sensors that are currently in Airflow "core". Some of the base classes ( BaseOperator , Base Hooks, SkipMixins will remain in the airflow.sdk  as the "foundation" for all other operators)
  • Standard provider 1.0.0 will contain everything we want to extract there - which means that the provider will be in "not-ready" state until we migrate all the "standard" components - and we release it then as 1.0.0. Before the migration we remove all deprecations that were in those operators - so that the "standard" provider contains "clean Airflow 3 versions" of those operators.
  • We discussed a few paths of upgrade for users:
    1. The user is on Airflow 2; they fixed the deprecation warning and then installed the new standard provider. Then, switch the DAGs to use operators from the Standard provider instead of AF 2. They are now ready to upgrade to Airflow 3.
    2. The user on Airflow 2 fixes all the deprecations and upgrades to Airflow 3; the import paths ( airflow.operators.python ) work due to dynamic imports that Ash suggested on the mailing list. This will now throw warnings that a user needs to import PythonOperator (as an example) from the standard provider ( airflow.providers.standard ) instead of Airflow, but it will work as the "standard" provider will be preinstalled with Airflow. A user will now switch the import paths to using one from the standard provider.
  • The standard provider 1.0.0 will contain changelogs of breaking changes from the Airflow core. Where applicable, the relevant news fragment in the current core will be moved to the Airflow standard provider. The newsfragments (Updating guide) for Airflow 3 docs will say that the core operators have been moved to the standard provider and it will have a link to that Standard provider docs for detailed breaking changes. 
  • The Airflow 2 to 3 migration process will be documented in the Airflow docs (similar to this but better), with a recommended approach from each of the two approaches listed above. The news fragments are right now just used for tracking breaking changes, they will be collected & organized to create a detailed upgrade process and a list of breaking changes.
  • Jarekhas also started a lazy consensus email thread here about the above.
• The recording of this call was added to YouTube as an unlisted video as there were a lot of discussions & useful presentations on this call:
  

19 September 2024

Agenda

Attendees

Summary

• Catch-up on action items from last call
  • Kaxil Naik showed the GitHub project boards for some of the existing AIPs. Kaxil will follow up with all other AIP-leads to have the project board created.
  • Kaxil Naik discussed a proposal for how the intermediate milestones would be created to track progress. Kaxil will follow up with a draft with Vikram & Rajesh and then follow up with all the AIP leads to have the milestones defined. The general agreement was to have 3 intermediate milestones: Week of 24 Oct, 18 Nov and 16 Dec
  • Vikram Koka confirmed that Astronomer did not end up using Casbin itself but just the interfaces. Shubham Mehta mentioned Niko Oliveira is going to explore Casbin and Keycloak and start implementation in about a week. The Dex project was suggested as yet another option by Mathew Wicks at the Airflow Summit. This was used by them in the Kubeflow project. His suggestion was to utilize it for Authentication only and continue doing Authorization on the Airflow side. Shubham will reach out to him.
• Discussions:
  • Follow up of [LAZY CONSENSUS] "Edge" as the name for AIP-69 in devlist: Shall providers be split into Operators / Executors / Plugins for the purpose of deployment where needed? At the moment all features that can be contributed from the side are in the same structure (Jens Scheffler )
    • Diagram to show what functionalities we have in provider packages that does into which part of an Airflow deployment:

• • • The goal of the discussion was to split the providers into Executors, plugins, and what's needed for DAG Authors (i.e User Code) like Operators and hooks. This way, we can limit the Python dependencies needed for Webserver, Scheduler, Worker, etc. An example is an AWS provider that has an Executor as well as an Operator. If we split them, then Scheduler can use the Provider that has an Executor, and Worker (including Triggerer & DAG Parser) can use the AWS provider that has an Operator but not an Executor. Airflow 3 is the right time to split this out if we decide to go that way.
    • re External Operator Links: Some ideas were discussed, like the parser storing the links in the DB, which is then retrieved by the Webserver via an API, so the Webserver does not need to install a provider just for Operator Links.
    • Jens Scheffler  will redraw the above diagram to separate out the Scheduler and the DAG Parser as the Scheduler does not run User Code, but the Parser does. As a follow-up to it, Jens will have a proposal around what the separation of providers would like to discuss on the next call and/or the dev mailing list. Vikram Koka and Kaxil Naik have volunteered to work with Jens on it offline before the next call.
  • Airflow Standard Provider: Bash & Python Operator (Pavan Kumar)
    • Pavan brought up the concern that System tests will break if we change the example DAGs for Bash & Python from core to using Standard provider until it is released. This was because some of providers like Google use the Bash or Python Operators.
    • Ideally, it should work with Airflow 3 since the provider will be pre-installed (via required_deps) with Airflow core/SDK.
    • The team decided to take it offline. Reviews would be appreciated.
  • Potentially moving dev calls on Oct 3 to Oct 9 or Oct 10. And one on Oct 31 [Diwali] to Nov 1 (Kaxil Naik )
    • Decided to move 03 Oct 2024  call to 10 Oct 2024  and move the 01 Nov 2024  to 07 Nov 2024 

10 October 2024

Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Jens Scheffler walked through the draft diagram for splitting up the providers into Operators / Executors / Plugins, based on the long term (Airflow 3) view of where they would be used. Jens stated how a follow-up proposal could be to move the “Extra Links” to be stored in the Airflow meta-database, since they are shared across components. A similar issue is  for the Extra Fields for Connections since they are also shared across components such as DAG Parser and Webserver. Vikram Koka added that this separation of providers was primarily driven from the packaging and distribution strategy as well as the release strategy to be used for the providers going forward because of the Core vs. Task SDK split coming as part of Airflow 3. The consensus from the group was a proposal needed to be written as a follow-up and shared with the dev list. Jens Scheffler suggested that Vikram Koka take a stab at the draft of the proposal and Vikram Koka agreed to take it as an action item.
  • Vikram Koka walked through the monthly Development milestones for Airflow 3, across all AIPs. The feedback from the group was positive and it was felt that it was helpful not only for the contributors, but the larger community. Kaxil Naik suggested that this should be a page within the top-level Airflow 3.0 page on the wiki. Jens Scheffler mentioned that he needed feedback on this PR on AIP-69 based on the AIP-72 work and Ash Berlin-Taylor responded he would review that async. Vikram Koka asked the AIP owners was to update the milestones with details and status as appropriate, but also to notify him of changes for potential ripple effects.
• Presentations:
  
  • Rahul Vats walked through the high level Airflow 3 test plan which included all the AIPs targeted for the release as well as some initial end to end scenarios. Rahul created these based on the AIP documents and asked for the AIP authors to review these and add additional end to end scenarios as appropriate, so that any breaking issues could be identified as part of the development process itself. Rahul recommended that these be used as the source of truth and be a foundation for the system tests. He also referenced the use of the testing slack channel #sig-airflow-3-qa-testing and asked for help from the community in testing this major release. Rahul took the action-item to articulate “what should be covered in the AIP unit tests vs. here in the E2E tests” with examples. Rajesh Bishundeo asked about performance testing AIP-59 in the context of Airflow 3. Michal Modras took the action-item to follow up with the original author of AIP-59 and see if the author or an alternate Google developer could give an updated proposal in time for Airflow 3 testing. 
  • Brent Bovenzi shared how the team has been using the UI update opportunity (AIP-38) for a UI Design refresh and presented mockups including the new home and DAGs list page. This had also been shared as part of the Airflow Summit. Pierre Jeambrun presented a progress update on the FastAPI-based approach, including a quick demo (AIPs 84). Brent Bovenzi also presented a progress update on AIP-68, including a quick demo of the approach for custom react components (plug-ins) based on FastAPI. 
• Discussions:

  • Jens Scheffler brought up the topic of continued support for Python 3.8 in the upcoming Airflow 2.11 release. I.e. if it was required or if it could be dropped. The consensus was that there were no objections from the attendees, but the next step was for Jens Scheffler to send an email out to the dev list about this asking for a “lazy consensus”.

24 October 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Providers follow-up: Vikram Koka commented that he, Jens Schubert , Jarek Potiuk , and others have been collaborating on a proposal document and that it would be shared well before the next dev call. Vikram shared a split version of that document to dev list on Friday as a pre-read for the providers repackaging document.
  • Performance test plan: Michal Modras shared that one of his team members would be working on it. He said that the current situation is that they have an “elastic DAG” which can be used for benchmarking various scenarios. He said that they would be working on the benchmark scenarios and sharing that document by the end of Nov. After that, the initial data collection based on these scenarios would be done and shared in Dec.
• Development updates and presentations:
  
  • Tzu-ping Chung shared a recorded video detailing the progress of AIP-74 on AIP-74 Introducing Data Assets. This showed the current progress on the AIP which was close to completion from a back-end perspective. This included the renaming of the existing Datasets and adding a name attribute to the asset class, to enable the specification of a more user-friendly name for the asset. He also shared that this name attribute is not currently shown in the UI, but that they would be working with their UI team to reflect these changes in the Airflow UI. 
  • Ankit Chaurasia shared an update on AIP-83 Remove Execution Date Unique Constraint from DAG run. He mentioned how this follows up on the work initially started in Airflow 2.2 with the shift towards run_id as a unique identifier and the logical_date as a human usable non-unique reference. He walked through the massive (around 280 files touched) draft PR focused on removing the execution_date completely including the CLI and the API. This PR has already had several comments from reviewers. The pervasiveness of the “execution date” was reinforced in the discussion. He mentioned how one of the challenges was also with the backfill PR and was waiting for the backfill PR to be completed and merged. He showcased the test plan for testing this AIP and requested the group to review and provide feedback on the test plan. He concluded that the PR is in good shape logic-wise, but fixing tests. The discussion after that raised the point that we need to upgrade the API version from v1 to v2 or v3 as part of Airflow 3, possible name to be “stable v3 API”.
  • Ephraim Anierobi shared an update on AIP-79 Remove Flask AppBuilder as Core dependency. It has a lot of work streams including the auth manager. Ephraim mentioned how the part he was working on was the separation of the DB migration from Fab. He mentioned how apps could now use an alternative mechanism called external DB managers to run DB migrations as part of the upgrade. He mentioned how this AIP included other work such as Auth manager and UI. Rajesh added (on behalf of Vincent) that the Auth manager was close to being done. Ash had also done some work on this AIP regarding 5.0 RC, but was not yet complete with respect to testing. Vikram expressed concern that clearly multiple “necessary parts” were done as part of this AIP, but not clear if all the “sufficient parts” were done. Ephraim mentioned that “what is left is for the UI to switch”. This needs to be followed up in the next dev call. Jens suggested that we may need to migrate or reimplement the “Connection form UI”.
  • Ephraim Anierobi also shared an update on AIP-65 Improve DAG history in UI - backend changes. He said how he is working on DAG Versioning around the serialized DAG, so that older versions of the DAGruns could be visible through the UI. Ephraim mentioned that the serialized DAG hash is not always consistent today because of inconsistent sorting, which is the first thing he just fixed. He also added a user-specified version for the DAG and talked about how he is close to completion on the DAG history database changes. He said that the next steps were the API and UI work.
  • Kaxil Naik shared an update on AIP-72 Task Execution Interface aka Task SDK, specifically focussing on the API server. He showed the addition of the scaffolding for the API server, so that in the future, the Core API and the execution API servers could be run independently if needed for different scaling requirements. He mentioned how he and Ash are working on porting code from Core to TaskSDK. Ash Berlin-Taylor then followed up with sharing the main PR for the Task SDK, which covers the DAG object, the TaskGroup, the Base Operator and so on, which would be the heart of airflow.sdk, which is the core Task SDK. To make upgrades easier, DAG import would stay as “airflow.dag”, so people don’t even get any deprecation warnings. Ash mentioned how he was currently thinking of including this “airflow.sdk” in the Scheduler to avoid reimplementation, but was still thinking through the implications. He mentioned how a lot of the classes are now using “attrs” to save on boilerplate code. Ash mentioned how the next step is to start adding endpoints for the API server to start doing simple client-based execution.
  • Brent Bovenzi finished up the call with an update on designs for AIP-38 Modern Web Application by walking through mockups of the redesigned DAG details page, including a list of actions / filters to show only failed tasks, etc. Brent also walked through the mockups for DAGruns view for Gantt chart view vs. Graph view, etc. Brent showed how the modal view enables a quick view of the details of a DA run and to switch between different areas which a user may want to drill into. Brent mentioned how the motivation was to enable proper use of space and to zoom between a DAG and a DAGrun and TaskInstance, without having to show a giant Christmas tree of the grid view.
• Action items for next call:

  • Vikram Koka to update the development milestones doc based on the performance benchmark plan from Michal Modras  - DONE

  • Vikram Koka to create a subpage under Airflow 3.0 to track critical changes important for upgrades, including API changes such as those brought up in the Execution date removal discussion. API owners to update this page as they make user facing changes during development. 
  • Ephraim Anierobi , Vincent, and others interested to follow-up on "What's left for AIP-79" and present at the next dev call. 

7 November 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Update on AIP-79 with respect to what's left Jedidiah Cunningham, Vincent BECK, Jens Scheffler: Jed shared that it was unclear at this time if AIP-79 was blocked on AIP-38 (new UI).
    • Jed to report back by the next dev call if he could magic back the older Fab provider, so that the old Fab provider could exist independent of the new Fab provider, so that we are not blocked by the new UI work
    • The Connection form for the new UI is another item which needs to be done and is not blocked by anything else. Jens's intention is to wait  and build the Connection form based on the Trigger form in the new UI. Though to be clear, no one actively owns the Connection form work at this time, but Brent and Jens believe that this will naturally follow as part of the AIP-38 work.
    • Clarity from Jed and Vincent that the new Fab provider is needed for the Auth manager and more and is not in conflict with the work that Jed is proposing above. 
  • Breaking changes update Vikram Koka: Vikram walked the group through his thoughts around capturing all changes that users need to worry about as part of upgrading to Airflow 3. Specifically to structure the newsfragments with a template. Vikram proposed an initial template with the following elements: 

    • DAG code changes

      • Import changes

      • Other DAG code changes

      • e.g. SubDAGs - please note if it was already marked as “deprecated” or “experimental”

    • Stable API changes: both invocation and result

    • CLI changes:

    • Airflow configuration changes

      • Airflow.cfg changes

      • Integration changes

    • Behavior changes: Example Backfill or Execution date changes

    • Plugin changes

  • There was immediate agreement within the group for the above and the desire for a concrete template. Vikram Koka to publish the above and then it could be added as a pre-commit. Vikram to also create an initial Upgrade / Change overview page, which would have example results based on some of the initial newsfragments (around 10).
• Development updates and presentations:
  • Airflow 3 workstreams update Vikram Koka: Vikram walked the group through the updates to the Airflow 3 workstreams and Airflow 3 dev milestones pages to reflect the high level status of the Airflow 3 project as well as finer level monthly details. Positive feedback from Rajesh and the rest of the team on the overall summarization in the Workstreams page, with the more granular detail in the monthly dev milestones page. Vikram reiterated the ask for more help on the AIP-38 UI work. Brent said that we are now ready for others to pickup work now, since the back-end APIs are ready to use. Brent to create a Slack sub-channel for UI development specifically for AIP-38 work. 
• Discussion topics:
  
  • Update other candidates from the Workstreams list:
    • Remove SLAs and AIP-57 Refactor SLA Feature - obsoleted by AIP-86 Deadline alerts?  Dennis Ferruzzi  confirmed that this entry was now obsoleted by AIP-86 Deadline Alerts. 
    • Remove poke, reschedule mode from Sensors? Kaxil Naik said that there was a discussion in one of the early dev calls and that he felt that this should be removed to really simplify the code and choices for the users. Daniel Standish felt that there was a difference between support in Operators vs. Airflow Core. He felt that the support in Operators was very confusing.  Shubham Mehta felt that this capability was useful for very long running tasks (greater than 12 hours) in the past, but wasn't sure if it was still necessary after Airflow 2.8, and would get back to the team about the same. Jedidiah Cunningham felt that it would be much simpler to take it out. There was a good discussion with multiple members of the team and both Igor Kholopov and Shubham Mehta confirmed that they didn't have a strict "no" at this time, but would need to confirm. After the discussion, the action item was for Kaxil Naik to start an email discussion around this in the dev list. 
    • Inspect & Simplify Airflow Configurations: Shubham Mehta and Dennis Ferruzzi confirmed that we should just defer this to post Airflow 3.0. 
  • OTEL integration into Airflow Scheduler: Daniel Standish 
    • There was a long discussion about the task log reading introduction into the very performance intensive Scheduler loop and the challenges therein. 
    • Kaxil Naik to start an async discussion around what changes need to be done to either refactor or revert these changes for OTEL traces. 
• Action items for next dev call:
  • Jedidiah Cunningham to confirm back regarding the older Fab provider changes for compatibility. 
  • Vikram Koka to create a template for newsfragments for flagging upgrade issues. And to create an initial Airflow 3 upgrade considerations page based on some of the existing PRs. 
  • Brent Bovenzi to create an Airflow 3 UI slack channel to facilitate collaboration on Airflow 3 UI dev ("airflow-3-ui"). 
  • Kaxil Naik to start email discussion regarding removal of Poke and reschedule mode from sensors. 
  • Kaxil Naik  to send an email about the issues with OTEL integration for Traces into the Scheduler. 

21 November 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Update on the POC of Flask/FAB app into the Fast API Jedidiah Cunningham :
    • The POC worked and that means we will not need a separate webserver to maintain backward compatibility with existing plugins. 
    • In addition, this means that we don't need the new UI (AIP-38) to be far enough along to remove the old UI. The old UI can stay as-is, and support for plugins can be added in the new FastAPI webserver.
    • Taking this from POC to production level reality will take some work. This will be planned by Jed and Vincent. Anticipate seeing a Github project board with the items to be done in time for the next dev call. 
  • Breaking changes update Vikram Koka : 
    • Vikram walked the group through the draft of the Airflow 3 breaking changes document. 
    • Vikram mentioned that this is about 50% populated right now, with 50% of current known changes still to be populated. 
    • To reduce the manual effort going forward, we will have a newsfragment template to be filled out as part of the PR process
• Development updates and presentations:
  • AIP-72 update (Ash Berlin-Taylor  and Kaxil Naik ): 
    • Ash demonstrated the working state of AIP-72 with respect to the separation of the Task Execution code into the TaskSDK vs. Airflow Core. This was in the form of tasks created with the new Task SDK and being run on the Local Executor, using a supervisory process.
    • The demo included the code showing how the supervisory process launched the task process and the socket pair / dup mechanism for logging. Ash also mentioned that the logging providers may need changes to accommodate the new system.
    • The team also discussed how this would make it a lot easier to intelligently display logs in the UI, without the UI needing to reparse the logs. It was very encouraging to see this coming together. 
  • AIP-66 update (Jed):
    • Jed shared a quick update regarding the progress being made on AIP-66 including the focus on the refreshing bundles from an external source. Jed also mentioned that he would be ready to open a PR for this early next week and that the demo could happen in the next dev call. He also expressed hope that the connection form work could be leveraged for the UI support here.
  • AIP-38 update (Brent):
    • Brent presented the UI progress including the new react-based DAGs list page and the dashboard page with basic statistics all leveraging the new FastAPI work. Brent mentioned that they were working on the Dag Run and task instance pages.
  • AIP-84 update (Pierre):
    • Pierre and Brent expressed gratitude for the many contributions from the community on AIP-84. Pierre confirmed that they were almost done with the migration of all public endpoints and that there were only a few remaining private endpoints to be done. 
• Discussion topics:
  • Meta milestones (Vikram):
    • Vikram highlighted that from an overall standpoint, the project was generally on track with the published schedule. 
    • He emphasized that the backend features should ideally be completed by the end of December and must be integrated by the week of the 18th of January in order to be on schedule. 
    • He said that this was critical for the packaging and upgrade work to be done on a stable base and be completed by the end of March. He mentioned that the UI could continue to change till early March, but the Jan backend feature freeze was critical. He asked all the AIP owners to keep this in mind and be pragmatic about their AIP scope
• Action items for next dev call:
  • Update on Fab provider backward compatability project board Jedidiah Cunningham  and Vincent BECK 
  • Update on Performance benchmark scenarios Michal Modras (based on the call on 24th Oct)

5 December 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Update on the FAB provider for backwards compatibility project Jedidiah Cunningham  and Vincent BECK 
    • Jed and Vincent shared the progress to date including a PR that already implements plug-in backwards compatibility. 
    • Vikram Koka  expressed appreciation for the progress and asked about the expected timing of the remainder of the items to be done and their response was outside of the New UI completeness blocker, the other items could be done by mid-Jan. 
    • Jens Scheffler suggested that the PR to validate dependencies without the new UI be created as a draft and validated with the existing functionality of the new UI rather than waiting for the new UI to be completed.  
  • Update on Performance benchmark scenarios Michal Modras :
    • Augusto shared the thinking around performance benchmark scenarios and metrics, with a focus on DAG performance and resource consumption. 
    • Augusto shared that this was a follow up on the work already done on AIP-59 and would be based on the existing performance framework. 
    • There was a significant discussion around the task timings and if those were relevant for realistic performance benchmarks. 
    • Jens asked if this would cover different executors and Augusto responded that this would be Celery first and possibly Kubernetes executor later.
    • Jens and Vikram brought up comparing the performance of Airflow 2.10 vs. Airflow 3 to identify performance differences. Augusto confirmed that all the tests would be run on both Airflow 2 and 3 to confirm performance changes. 
• Development updates and presentations:
  • Update on AIP-75 New Asset-Centric Syntax Tzu-ping Chung 
    • TP shared a recording of the new syntax for asset creation.
    • TP also showed the demo of a new Airflow CLI command to list all the Data Assets and to show the details of a specified Data Asset. 
    • Finally, TP also introduced the "materialized" command for a data asset which ensures that the asset is created by running the DAG which outputs that asset. 
  • Update on AIP-66: DAG Bundles & Parsing Jedidiah Cunningham 
    • Jed demonstrated the process of defining DAG bundles and how DAG bundles would be parsed by the DAG processor
    • He mentioned how some of the changes are happening in conjunction with the changes being done in AIP-72. 
    • He also showed bundleIDs and bundle Versions. He then showed how a new version is parsed and reprocessed. 
    • He mentioned that there is much more work to be done, but the core of bundle definitions and DAGs being processed from those bundles is now in place.
    • In response to questions, he clarified how DAG Bundles currently pull down the entire Git clone into a temporary folder, so that all DAGs and their friends/dependencies could be processed. And that, more optimization is very feasible.
  • Update on AIP-78 Scheduler-managed backfill Daniel Standish 
    • Daniel said that all the back-end server work for this AIP as scoped has been completed for a bit. He added that the front-end UI work will be done as part of AIP-38. 
    • He however added that there is a Data completeness conversation to be had as a result, which led to the discussion below.
• Discussion topics:
  • Data completeness discussion  Daniel Standish 
    • Daniel brought up the concept of implicit data partitioning already in Airflow with the concept of execution date, when catchup is defined to be True. 
    • Daniel advocated making this implicit data partitioning an explicit concept in Airflow, arguing that the existing grid view is already an incarnation of the same. 
    • At a high level, users could declare that a DAG is partition-driven, based on the timetable. Going forward, Backfills or catchup would only be supported for partition-driven DAGs. 
    • For backwards compatibility, old DAGs would be assumed to be partition driven. 
    • The immediate reaction from the team is that this is a big change and there was significant discussion if this is absolutely required.
    • Daniel said that the trigger for this was AIP-78 Scheduler-managed backfill and AIP-83 Remove Execution Date Unique Constraint from DAG run, which left a bit of a vacuum between them.
    • The follow-up action item after the discussion was for Daniel to share thoughts async and everyone to think about the need for this. 
  • Milestone and scope update Vikram Koka 
    • Vikram shared that at a high level development was on track towards the plan shared earlier.
    • However, there would be one scope change with AIP-80 Explicit Template Fields in Operator Arguments being deferred from 3.0 to a future 3.x release. 
• Action items on/before next dev call:
  • Daniel Standish to post a document regarding explicit vs. implicit partitioning and its need as a result of the removal of execution date, especially with an eye towards backwards compatibility. Team to consider the introduction of a partition concept in Airflow. 

19 December 2024

(Proposed) Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Update on the Data completeness discussion Daniel Standish 
    • Daniel presented the problem around ambiguous behavior for backfills, when logical dates are no longer unique. 
    • He spoke about how the traditional Airflow behaviour using execution date based on the DAG schedule implicitly established a date-based partitioning scheme for DAGs.  
    • He then advocated making this behaviour explicit, by defining this to be a partition scheme of "implicit" and also supporting a partition scheme of "none" for non-date-based partitioned DAGs which could be run multiple times for the exact same logical date, which is a common use case both for model training as well as for inference execution. 
    • There was a significant level of debate around this topic, with two counter-points. The first counter-point was that the "partition" term was confusing, especially with "Data Asset Partitions" upcoming in AIP-76, and if this was the same, different, or related. The second counter-point was around whether a new DAG-level parameter should be added for this, as opposed to either one or more combinations of DAG API/UI invocation parameters and existing DAG parameters such as "catchup". 
    • Since a conclusion could not be reached in the call, the action item for Daniel, was to write up a document as an amendment to the existing AIP-83 Remove logical date uniqueness and share it with the mailing list. Daniel created and shared this doc on Friday.
• Development updates and presentations
  • Update on AIP-38 Modern Web Application Brent Bovenzi 
    • Brent demonstrated the progress on the UI specifically including the DagsList and DagDetails pages. 
    • This included the ability to trigger DAGs and view DagRuns.
    • Brent then drilled into viewing DAG Run details such as Task logs and XCom data. 
    • It was very encouraging to see the core DAG development and test workflows in action as part of Airflow 3. 
  • Update on AIP-72 Task Execution Interface aka Task SDK Amogh Desai , Kaxil Naik , Ash Berlin-Taylor 
    • Amogh started the update on AIP-72 by walking through how deferred tasks were being executed based on the Task SDK. 
    • The demo was based on a DAG including a date time sensor and the interaction between the supervisor process and the task runner for different task states.
    • Amogh also explained the task execution states which were currently implemented and the ones which were still to be done. 
    • Kaxil then walked through the implementation of the TaskContext, a key concept within the Task Execution Interface.
    • This included the interface and the implementation of Connections, and how that was fetched from the API server. 
    • Ash briefly shared a PR that removed the database dependency for DAG parsing and how that leveraged the supervisor-task runner communication interface. 
    • Because we were running out of time in the dev call, he said that would subsequently post a video of the demo. Ash posted this demo video to the dev list later. 
• Discussion topics:
  • PR strategy for 3.1+ feature work - Rajesh
    • Rajesh wanted to discuss how to handle PRs for post-3.0 feature work. Since the dev call was already running over, this was to be discussed async. 
  • Scheduling for next dev call
    • Since a lot of committers are out of holidays, the next dev call will be on the 9th of January, 2025 instead of the 2nd of January.

9 January 2025

Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • AIP-83 amendment aka Data completeness discussion follow-up Daniel Standish Tzu-ping Chung
    • The docs shared earlier were:  Amendment to AIP-83 and Option 2 clarification 
    • Daniel and TP walked through the detailed "option 2 clarification" document above to address the questions under discussion
    • At a high level, there was agreement about the overall approach. There were some details to be worked through.
    • Elad suggested a deadline to close the discussion, so that this could be done in time for release. Vikram agreed and recommended that Daniel / TP start an email voting thread, so this could be closed by Thursday of next week. 
    • The team also expressed appreciation to Daniel and TP for their perseverance in working through this change, since this had generated huge debate.
• Development Updates:
  • Update on AIP-66 DAG Bundles & Parsing Jedidiah Cunningham 
    • Jed demonstrated DAG bundles based on a Github repo, then DAG versioning and DAG fetching. 
    • He started a DAG run based on DAG version 1, updated the DAG, and then showed how the currently running DAG continued to execute subsequent tasks based on version 1 of the DAG, rather than the latest version.
    • Jed highlighted that this work had been a team effort with Ephraim and Daniel Standish.
    • DAG versionsing is a long requested feature within Airflow and it was great to see the working demo.
  • Update on AIP-81 Enhanced security of CLI by integration with API Buğra Öztürk
    • Bugra presented an update on the progress to date on the CLI
    • Bugra demonstrated the progress on making the CLI use the API for enhanced security, along with the command for connection configuration, as well as the authentication integration in flight. 
    • This AIP has been flying under the radar for a bit, so it was a treat to see the wonderful progress here.
• Discussion topics:
  • Doc build process Jarek Potiuk 
    • Jarek brought up the concern that the Airflow doc builds were taking 30 minutes now, from around 5 minutes a couple of years ago. He, Ash, Elad, Jed, and Kaxil all highlighted that this was a ticking time bomb. 
    • As a quick analysis, there seemed to be three different areas which needed to be addressed here:
      • The doc build technology which has not been updated in years
      • Process changes around Airflow user documentation including retention, versioning, and search engine optimization
      • Doc structure changes as needed for the upcoming Airflow 3 release.
    • Jarek also brought up the point that the typical folks who have been patching this are all very busy with Airflow 3 development and will be till release. Constance volunteered to take on the third item identified above. 
    • We could not finish the discussion because of time constraints, but the takeaway was to start an async discussion thread around this topic. 
  • When do we plan to release 2.11? And are all needed features available by then? Jens Scheffler 
    • Jens brought up the topic of Airflow 2.11, saying that we had discussed cutting the Airflow 2.11 release when we started working on Airflow 3 and if that was still in the plan.
    • The Airflow 2.11 discussion was initially triggered by the forward compatibility proposal with AIP-80. That approach had been changed before AIP approval and subsequently AIP-80 itself had been deferred to post-3.0. Since then, the Airflow 2.11 release faded away from the discussion.
    • Kaxil and Jens took the action item to find the issues tagged for Airflow 2.11 and to start an async discussion around the pros/cons of that release. 
• Action items on/before next dev call:
  • Daniel Standish  / Tzu-ping Chung  to start a voting thread for the AIP-83 amendment. 
  • Jarek Potiuk  to start an async discussion around the doc build issues
  • Jens Scheffler  / Kaxil Naik  to start an async discussion around the pros/cons for the 2.11 release

23 January 2025

Proposed Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • AIP-83 amendment aka Data completeness discussion follow-up Daniel Standish Tzu-ping Chung
    • Daniel Standish confirmed that the vote had been started on dev list and encouraged the team to respond. 
• Development Updates:
  • Update on AIP-82 External Event Driven Scheduling Vincent BECK 
    • Vincent demonstrated the implementation of the AssetWatcher concept built on top of FileOperator, which enables a DAG to be triggered based on the arrival of a file in an external blob storage. 
    • Vincent also did a second demo based on watching for two different files and that the DAG would be triggered based on either file arriving in blob storage. 
    • Vincent also showed a DAG which could be triggered based on a message arriving in an SQS queue. 
    • Vincent highlighted that DAG authors need to be careful on what triggers to use for invoking DAGs, since it is possible to have additional DAG runs because of persistent events. There was discussion around this and the agreement was to take that discussion offline. 
  • Update on AIP-72 Task Execution Interface aka Task SDK Ash Berlin-Taylor Kaxil Naik Amogh Desai 
    • Ash highlighted that the team was chipping away on the work. He highlighted that Mapped Tasks was close to completely working based on the TaskSDK. 
    • He said that once Mapped Tasks was done, Celery and Kubernetes executors would be moved over to use the TaskSDK. 
    • He also said that after Mapped Tasks was done, the Trigger changes was the last "big feature" to be done and then there was going to be the long tail of fixes. 
  • Update on AIP-38 Modern Web Application Pierre Jeambrun Brent Bovenzi 
    • Pierre demonstrated the triggering and clearing of DAG runs and Tasks. 
    • Pierre said that Brent is working on the Grid view and it isn't merged yet, but close
    • Pierre said that after the alpha, we would be able to use the new API exclusively and then we could delete the old UI and API. 
    • Pierre also said that they were making progress and no particular blockers. 
    • Brent demonstrated the GridView and said how they were looking forward to getting feedback as part of the Alphas. 
  • Update on Provider packages new structure Jarek Potiuk  
    • Jarek did a quick walk through about moving the providers to the new structure as a prelude to the new packaging work upcoming in February. 
    • He said that he would ask for help from the larger team on this work, since it could not be 100% automated. He said that his script automated 90+% of the work,  but needed human intervention for the last little bit. He walked the team through a quick run of this script to illustrate. 
• Discussion Topics
  • Milestone update and progress Vikram Koka  (actually Ash Berlin-Taylor ) 
    • Ash walked through the overall progress update, since Vikram was out sick and couldn't make the call. 
    • Ash gave a quick overview of where we are based on the dev milestone wiki page. He highlighted that we are shooting for getting the first alpha builds out next week.
    • Elad asked for clarification about the "feature freeze" beyond the alpha milestone. Ash took the action item to post a message to the dev list about it. 
    • Rajesh asked about where we are on the timeline and different AIP owners went through their progress updates as compared to the published timeline
    • Jed, Vincent, Pierre, Vincent, Bugra, and Daniel Standish all went through their AIPs and expressed confidence in the schedule as they saw it. 
    • Ash summarized the discussion to express that the first alpha would have rough edges, but that the team felt positive about the progress and where we are at this point.
  • AIP-81  CLI Discussion: Task Commands Remote vs. Local Buğra Öztürk  (Doc link)
    • Bugra wanted to have a discussion around some of the CLI Task commands which would not be possible through the Task API. 
    • Because of a lack of time, the proposal was to have that discussion async. 
    • Bugra suggested reviewing the doc also for a discussion around local vs. remote commands.
  • Airflow 2.11 release discussions Jens Scheffler (Jarek Potiuk stepped in because of bad Wifi connection for Jens)
    • Jed said that Utkarsh is planning to cut 2.10.5 next week.
    • Topics raised demanded/needed for 2.11
      • airflow config lint command (but still can be added in 2.10-line or independent as well, no big driver)
      • Migration tooling (which is still WIP)
      • No other things raised
    • Generation notion: Not critical at the moment. At least nobody raised a deadline or critical feature (yet)
• Action items on/before next dev call:
  • Ash to send out an email to the dev list regarding "feature freeze" guidelines, both to developers and committers about "merging PRs". 
  • Team to review Bugra's doc around CLI changes

6 February 2025

Proposed Agenda

Attendees

Summary

• Follow-up on action items from the last call
  • "Feature freeze" guidelines Ash Berlin-Taylor 
    • Ash confirmed that he had sent out the email to the dev list with his thoughts around how we as committers and PMC members need to be very risk-aware as we merge changes into main until the 3.0 release is shipped as GA. He also mentioned how fundamental the 3.0 changes were to the code and that they were much larger than even the 2.0 changes. Jarek and Vikram agreed with the risk-aware sentiment.
  • AIP-81 CLI changes Buğra Öztürk 
    • Bugra quickly walked through the doc already shared with the dev list (Doc link). 
    • Bugra walked through how local vs. remote commands now needed to be separated because of the TaskSDK changes in 3.0. There was general agreement about this.
    • There wasn't much discussion in the call around the detailed architecture and sequence diagrams in the document, and "the ask" was for the team to review the document and leave comments async. 
• Development updates
  • Update on AIP-38 Modern Web Application: New Trigger Form UI by Shubham Raj and Jens Scheffler 
    • Shubham walked through a demonstration of the new trigger form developed in the Airflow 3 UI. 
    • The demo went through flawlessly and there was general admiration for the work done. 
    • Shubham also confirmed that this work had been merged into main already. 
  • Test plan update Rahul Vats 
    • Rahul walked through the Airflow 3.0 Test plan which now includes the status of the formal testing effort in addition to the plan itself. 
    • Rahul described how the team was tracking development and test progress, while testing and reporting issues on the alpha builds. 
    • There was general appreciation of the work being done. Vikram urged Rahul to ask for help from the broader team as and when needed.
• Discussion topics
  • Milestone / workstream update Vikram Koka 
    • Vikram thanked the whole team for all the great work which had been done towards getting to the alpha builds. He mentioned how hard cutting the first alpha had been and appreciated Jed among others, who had almost a 24 hour day in order to make it happen. 
    • In order to be more predictable, Vikram asked the team to cut alphas (and betas in the future), at least every Thursday, so that we could have predictability. 
    • Jarek agreed, citing the heuristic "If it hurts, do it more often". Jed agreed, saying that the alpha2 being cut right then has been significantly smoother than alpha1. Jed also asked everyone on the team to be aware that the alpha builds would start roughly around noon India time every Thursday. 
    • By the time of writing, Airflow 3.0.0alpha2 had been shipped out. 

20 February 2025

Proposed Agenda

Attendees

Summary

• Follow-up on action items from the last call
• Development Updates and Presentations:
  • Update on AIP-72 Task Execution Interface aka Task SDK Ash Berlin-Taylor Amogh Desai
    
    • Ash walked through a demo of the Triggerer support with the new TaskSDK. He shared how this was largely in place, with the exception of connection access.
    • Ash mentioned that the Triggerer is now split into two processes, using the supervisor pattern from the Task SDK. 
    • Amogh gave an impromptu demo of the Kubernetes Executor now working with the TaskSDK. 
  • Update on AIP-66 DAG Bundles & Parsing Daniel Standish  
    • Daniel Standish shared a recording of the demo covering the stale DAG bundle cleanup process.
    • Since Workers now pull code from remote sources to run tasks and since this is stored locally to reduce latency, there is cleanup required to avoid these piling up on the worker nodes. 
    • He said that "old / stale" could be defined by time or number of older versions.
  • Test plan update Rahul Vats 
    • Rahul covered the testing status as documented in the Airflow 3 Test Plan
    • Rahul covered the testing status of Alpha 3 and the confidence levels of the different areas including the UI and CLI. He mentioned that these are still in progress, since development is still ongoing. 
    • He mentioned that they have started testing with focus on specific AIPs and are now reporting status by AIPs as well.
    • He asked the AIP owners to review the test cases and validate them as they make changes. And for the AIP owners to ensure that there was at least one run on each before the beta releases as well. 
  • AIP-81 CLI Update Buğra Öztürk
    • Bugra covered the updated design for the Authentication work for the CLI. 
    • Bugra asked for help from the committers on the call to review the PRs coming in for the CLI AIP, since some of the PRs were sitting for a while in review status.
    • Bugra commented that some of the authentication work would probably need to come in Airflow 3.1, rather than in 3.0. 
  • Update on AIP-38 Modern Web Application Brent Bovenzi 
    • Brent demonstrated an updated UX in the Airflow UI, with the Grid View or Graph View (based on the user preference) on the left hand of the page to give a consistent view with the modal views on the right hand side. 
    • The updated UX drew a very positive response from the team on the call. 
    • Brent showed how the new UX addressed color-blindness issues with icons and was more accessible. 
• Discussion topics:
  • Python 3.13 (not only) -driven dependency issues Jarek Potiuk 
    • Jarek highlighted these three issues with 3.13.
    • Pendulum: He said that the Pendulum maintainer got back to him and we don't have to do anything in this area. 
    • google-re2 → can we replace "root" usage in UI: Jarek said that Google-re2 issue is a bit more pervasive in Airflow and that after speaking with Bugra is solvable in the CLI. He was concerned about its use in the UI and API. Brent and Pierre said that this was no longer being used, so it is not needed anymore.
    • FAB (and sqlalchemy and pandas) → to FAB or not to FAB: The dependency conflict between SqlAlchemy, Pandas and FAB requires us to probably take out FAB as quickly as possible from Core Airflow. 
  • Removing the old UI Jedidiah Cunningham 
    • Jed said that he is targeting next week to remove the old UI, before we ship Beta 1. 
    • Jed also proposed moving that branch to the main repo for easier collaboration from the larger team and there was general agreement.  
  • Bump minimum version for Airflow 3.0 to Python 3.10 (~6months ahead of EoL schedule) Ash Berlin-Taylor
    • Ash proposed this and asked if there were any strong feelings against it. 
    • There was significant discussion about what this meant for Airflow 2.x vs. 3.x including the implications for Provider releases going forward. 
    • The decision was to take this to the mailing list. 

6 March 2025

Proposed Agenda

Attendees

Summary

• Follow-up on action items from the last call:
  • Airflow 3.0 minimum version to be Python 3.10 Ash Berlin-Taylor 
    • Moving to Python 3.10 is not a blocker.
    • Ash said that he didn't have time to make this change in time for the 3.0 release. 
    • Therefore, decided to bump to after the 3.0 release and do this as part of the standard Python support cadence.
• Development Updates and Presentations:
  • Update on AIP-72 Task Execution Interface aka Task SDK 
    
    • Short circuit and Branch operators Shahar Epstein 
    • Shahar walked through the changes required for these operators.
    • Specifically, the change from direct DB access to using the TaskSDK for updating XComs and the state of the task. 
  • Test plan update Rahul Vats
    • Rahul walked through testing status with Beta1, including testing migrations (upgrades and downgrades). 
    • Rahul said that confidence level with respect to the various AIPs was increasing and that he expected the Grid of confidence by AIP to get a lot more colorful over the next week.  
    • He said that since the "DB isolation will be enforced starting Beta2", he expected more test DAGs to start failing. 
    • When asked about UI test automation, he said that the UI testing was manual and could not be automated yet because it was changing every day. 
    • The team was very appreciative of Rahul's update.
  • Milestone update Vikram Koka 
    • Feature completeness and road to RCs
    • Vikram shared the updated Airflow 3 dev milestones and said that he wanted the beta 4 release targeted for the week of March 17th, to be feature complete and the final beta release. 
    • Vikram asked for all open PRs to be completed, tested, and merged before beta 4. He urged contributors to ask for help if needed to meet this milestone.
    • Vikram asked for patience for the community over the next few weeks, since open issues could not be allowed to linger and would be reassigned if the assignee was unresponsive.
    • There was general consensus from the team. 
    • Vikram also brought up AIP-68 Extended Plugin interface for React Views, saying that this would need to be deferred to 3.1 because of timing. There was general agreement on this as well. 
  • DevEx improvements as result of provider's distribution separation and devel-common Jarek Potiuk
    • Jarek shared a quick update on the recent DevEx improvements for Airflow development based on UV.
    • He shared how this would make Provider development a lot easier than previously the case as well as benefiting TaskSDK development going forward.
• Discussion topics:
  • Packaging update - open items (Ash Berlin-Taylor , Jedidiah Cunningham Jarek Potiuk )
    • A quick update shared was about making the Airflow repo a meta package, with Core Airflow becoming a project within the Airflow repo. 
  • Edge provider naming issue (and PEP 752 as long term solution) Jarek Potiuk 
    • Jarek brought up the issue that this name is already taken on the PyPI registry and that this is related to the PEP 752 discussion. 
    • When this PEP is implemented, ASF will take the "Apache" name, but this is still in progress with the POC to be implemented this week. 
    • The immediate action is to hopefully get this name (Edge) from the PyPI registry. 
  • FAB next steps - Jarek Potiuk 
    • Jarek confirmed that he is working on enabling Airflow to use FAB 5.0, since this is holding us back from upgrading SqlAlchemy. 
    • Jarek outlined multiple options: 
      • Option 1: Wait for 5.0, but that is 2 to 3 months
      • Option 2: Take the needed parts of FAB and incorporate into the FAB provider; Then upgrade to SqlAlchemy 2.
      • Option 3: Stay with SqlAlchemy 1 for now, but the security issues with it are problematic
    • The team agreed that option 1 was a non-starter, and that we should on option 2, but be prepared to switch to option 3, if we run into issues with the SqlAlchemy upgrade being so close to release. 

20 March 2025

Proposed Agenda

Attendees

Summary

• Development Updates and Presentations
  • AIP-81 CLI Authentication Design Update Buğra Öztürk Jedidiah Cunningham 
    • Bugra and Jed explained how AIP-81 would create a completely separate remote CLI package, which will be "airflow-ctl".
    • For local execution, all of the CLI commands (which are DB backed) will just work. 
    • All of the merged code for remote CLI execution will be moved into airflow-ctl as part of beta4 and will also simplify packaging and release timelines. 
  • AIP-72 Task Execution Interface aka Task SDK Update Ash Berlin-Taylor Kaxil Naik Amogh Desai 
    • Kaxil gave an update on the AIP-72 detailing that JWT Integration, Custom XCom backend, and Secrets backend work is all completed. 
    • He added that the Triggerer backend now works without going through the public API, to avoid all the authentication passing complexity.
    • Kaxil said that the Task Notifiers is the one open item, which TP is currently working on to finish.
    • Amogh added that the Secrets backend also enables a separate secrets backend for Workers as compared to the Secrets backend configured for the Scheduler. 
    • Ash added that each Task now has a JWT token that is associated with it, bound to a Task Try ID. 
    • Ash also added that the Execution API versioning was now merged for the Task Execution Interface and implemented Calendar versioning, based on Cadwyn.
    • Ash added that this was only for the Task Execution Interface, not for the public API, nor for UI internal API. 
    • Shahar added that he had heard that some AWS provider system tests were breaking right and that he would be investigating. 
  • Test plan update Rahul Vats
    • Rahul shared the updated Airflow 3 Test plan and walked through a quick progress update by AIP. 
    • Rahul asked everyone on the team to put on their testing hat, treat beta4 as being the internal testing vehicle, and to test it out, so that we can find more bugs before release.
    • Rahul also added that AIP-81 needed to be tested for CLI regressions. 
• Discussion Topics:
  • Packaging Update Jarek 
    • Jarek highlighted the big PR which introduces the "airflow-core" package within the main airflow repo.
    • He highlighted how there is no longer an "airflow" folder in the main repo.
    • Instead, there is "airflow-core" which has all the key items inside it including the pyproject home.
    • The second thing we have is an airflow meta package which has all the configuration for builds and additional elements for packaging and distribution. 
    • Jarek said that this was still work in progress, with a few things to be fixed, but was mostly done.
    • Jarek also spoke about how UV works with the new packaging structure and stressed how this simplifies Airflow development.
  • Cleanup of Airflow 3 open items in Github Elad Kalif 
    • Elad asked all the AIP owners to review the list of Github issues originally targeted for Airflow 3 and still open
    • Specifically, these are airflow3.0:candidate and airflow3.0:breaking
    • Elad made the point that these should either be closed as having been done or moved to a subsequent release tag as part of the Airflow 3 RCs. 
• Action items
  • What's critical to finish up
    • Jens brought this up in the context of the UI, if there are open critical items to be completed before 3.0 release
    • Brent took the action item to highlight this async and shout out if anything needed help
  • Change dev calls to be weekly until release
    • Elad suggested that we make the dev calls weekly, since we are very close to release.
    • Vikram to update the calendar invite to be weekly and to send out a reminder if a meeting is required that week. 

27 March 2025

Proposed Agenda

Attendees

Summary

• Follow-up on action items from last call
  • Cleanup of Airflow 3 open items in Github - specifically airflow3.0:candidate and airflow3.0:breaking
  • There was consensus already either closing the "airflow3.0:candidate" issues or tagging them to be "airflow 3.1+" as appropriate based on where we are today
  • The harder discussion was around "airflow3.0:breaking" issues: 
    • Some of these were actually done and need to be just closed. Ash, Jens, and Kaxil had identified at least one each. 
    • There was a discussion about others which could not be done in the time left because of ripple effects and migration challenges. The team decided that we would discuss a mechanism for introducing deprecations / breaking changes without having to be as ultra-strict on semantic versioning as we are today. To avoid cognitive overload, this discussion will be brought to the dev list in May. Jarek took the action item for this. 
• Development Updates and Presentations
  • Test Plan Update Rahul Vats 
    • Rahul walked through the Airflow 3 Test plan and showed how it was now updated to also reflect the status of AIPs 81 and 82, thanks for Bugra and Vincent. 
    • He also spoke about the integration testing with Providers under way. 
    • For AWS providers, Rahul mentioned that he had been working with Dennis and that the AWS providers were mostly working, with only a couple of issues to be resolved.
    • For Google Providers, he mentioned that the Google Provider dashboard was broken and that the Google team was working on fixing the dashboard. Eugene from the Google team confirmed the same. 
  • Open issues to be resolved 
    
    • With respect to the Branch Python Operator raised by Shahar last week, Pavan said that he had fixed that issue. There was some discussion if there were any other issues regarding the BranchPythonOperator, but there was nothing on the board at this time. 
    • Kaxil mentioned that there is a high priority issue with BaseSensor, which he was working on fixing right now. 
    • There is also a critical issue with Triggerer , which Pierre is fixing and close to merging at this time. Pavan to fix the Deferrable Operator issues based on this. 
    • A high priority DAG Versioning UI / Grid View issue was also discussed and Jed said that he had a fix for that on his dev branch based on work with Brent, which he would be merging shortly.
    • Rahul mentioned that he and the team were also looking at UX issues between Airflow 3 and Airflow 2.x, and would raise them as appropriate.

3 April 2025

Summary

• Excellent progress being made on the testing front. Rahul making async updates to the Airflow 3 Test plan
• Excellent progress being made on reporting and resolving open issues - great dialogue on Github
• Skipped dev call, since there was no compelling need for a meeting. 
• Using that time for cutting RC1.

10 April 2025

Summary

• Release Candidate cut late last week (on the 3rd) - Airflow 3.0 RC1
• Excellent progress being made on the testing front. Rahul making async updates to the Airflow 3 Test plan
• Excellent progress being made on reporting and resolving open issues - great dialogue on Github
• Skipped dev call, since there was no compelling need for a meeting. 
• Using the time for working towards RC2.

17 April 2025

Proposed Agenda

Attendees

Summary

• Follow-up on action items from last call:
  • Edge Provider naming Elad Kalif  Jarek Potiuk:
    • Elad explained that the Edge provider will be called "edge3" for many reasons.
    • It's ready to be used and tested.
    • Once the current Provider vote is closed, Elad will cut a new wave including Git and Edge3 for an extra 24 hours voting period and then these would be released. 
• Development Updates
  • Test plan and Issue statistics update Rahul Vats and vikram 
    • Rahul shared the current state of testing based on RC3 and said the regression testing on RC3 is ongoing.
    • Rahul felt that the current confidence level is high.
    • Rahul also shared the issue statistics for Airflow 3 with:
      • Alpha releases: 40 issues reported and closed.
      • Beta releases: 129 issues reported and closed.
      • Release candidates: 102 issues reported, with 64 of them already closed. 
    • Rahul also said that there are no currently open blocking issues with the release.
    • He also expressed how many issues were fixed directly in PRs without the issues even being reported.
    • Vikram added that he had a target of at least 200 issues to be reported in the combination of Alpha, Beta, and RCs.
    • He said that seeing a total of 271 issues reported was great to see and gave confidence in testing of the release. 
• Discussion Topics
  • Dev workflow for 3.0+ Kaxil Naik 
    • Kaxil explained that we now have project milestones for 3.0.1, Airflow 3.1.0, and Airflow 3.1+. 
    • He articulated how these should be used for new work going forward with issue resolutions for 3.0.1, vs. new feature work being targeted for 3.1.0 and 3.1+ 
  • Docs approach towards AF 3 users Elad Kalif 
    • Elad reminded everyone that our docs are versioned and asked everyone to write docs assuming that the users coming are using Airflow 3.0. 
    • Elad asked doc writers to avoid "if Airflow 2, do this" and "if Airflow 3, do this" within the docs, since that would be confusing to the reader. 
  • Ongoing support for Airflow 2 Kaxil Naik 
    • Kaxil brought up the topic of "maintenance support for Airflow 2", once Airflow 3 is released.
    • Kaxil showed that we had supported Core Airflow 1.10.x maintenance releases for 6 months after the release of Airflow 2. 
    • There was significant discussion around this including the separation of maintenance of bug fixes vs. security fixes. 
    • Elad also brought up the separation of maintenance updates for Core Airflow OSS vs. Providers
    • The decision from the team was to take this discussion to the devlist. Kaxil to start a mailing list thread.

24 April 2025

Summary

Celebratory team call after the Airflow 3.0 release.

8 May 2025

Proposed Agenda

Attendees

Summary

• Development Updates:
  • Update on the 3.0.1 release Kaxil Naik 
    • Kaxil mentioned that over 200 commits are part of 3.0.1 RC1 including many usability improvements.
    • He said that the voting for RC1 ends on Monday, the 12th of May. 
    • Jarek added that we would probably need an RC2 because of the inclusion of an older version of an Amazon provider. 
  • 2.11 release scope / timing Kaxil Naik 
    • Kaxil mentioned that the decision between 2.11 vs. 2.10.x was made because of several features and changes in the 2.x branch, which meant beyond a patch. 
    • Specifically, he mentioned the new Timetable addition as well as the PR for making sure that the metrics are aligned for consistent reporting. 
    • Another reason for the feature release was also to add support for "airflow config lint" which enables validation of the airflow configuration before upgrading to Airflow 3. 
• Discussion Topics:
  • 3.0 release feedback / discussion topics (All)
    • Vikram shared a quick update on the Airflow downloads by version over the last two weeks since Airflow 3 was released. 
    • Vikram highlighted that Airflow 3 has seen significant adoption with about 7.5% of the total Airflow downloads. The highest downloaded version continues to be 2.10, which is the last release on the 2.x release stream, with about a third of all downloads. Of course, these metrics should be taken with the usual grain of salt, because they are susceptible to image caching and CI processes.
    • Elad asked where features "missing" in Airflow 3 were documented. Vikram said that these were documented as breaking changes within the Upgrading to Airflow 3 doc. The team identified a couple of missing elements, which Elad volunteered to add. 
    • Jarek said that there was significant user confusion around upgrading to 3.0 with respect to DAG writing practices, specifically around imports. After much discussion, the conclusion was that one of the core problems here was the UX around doc versions on the website, being too subtle and not obvious to users about the differences between 2.x and 3.0 syntax. The action item for Jarek and Elad was to look at the docs UX to see if this could be made more obvious.
  • 3.1 release scope / timing discussion Vikram Koka  
    • In the follow-up discussion about the scope of 3.1, Vikram explained that the Airflow 3.1 scope was largely based on user feedback of 3.0 including the prior discussion about missing features.
    • The early draft of Airflow 3.1 scope is documented on the wiki for discussion. 
    • Vikram mentioned that Dennis couldn't make the call today, but asked for reviews on his PR which was early work towards Deadline Alerts, which replaces SLA alerts in Airflow 2.x and is currently scoped for Airflow 3.1. Rajesh added that he was confident with the timing, since Dennis and others were actively working on this and would be ready to update the team in two weeks' time.  
    • Jarek raised the question a
    • Other language support
    • Ash shared an update on the Go SDK. He shared a link to the branch with the draft incarnation and said that a PR would be coming soon. 
    • Vikram shared an update with the early Java SDK branch. Jens chimed in referencing the offline discussion with Vikram around leveraging the Edge API for these other languages.
    • The action item was for Jens, Ash, Kaxil, and Vikram to discuss async on any changes needed to the Edge provider. 
    • Ash raised a concern that we may not see significant adoption until the DAGs were in other languages as well. Elad and Jarek argued against the viewpoint feeling that a single task in a different language would be very powerful for application integration. Elad said that "The notion of executing a task directly without a wrapper DAG is a request I hear from time to time (already)". 
  • FAB upgrades (4.6.2 first, 5.0.0 later) Jarek Potiuk 
    • Jarek raised the FAB upgrade issue which we had deferred a couple of months ago to post 3.0. He said that we are in a much better situation now, but that he needed help with the SqlAlchemy migrations and would follow up on the dev list. 
    • As part of this conversation, there was a discussion about dropping support for Python 3.9 early. However, there was no appetite for that from the attendees because of all the complexity around doing this selectively for Core Airflow vs. Providers. This was going to be conveyed back to the dev list discussion by Jens.
  • Example DAGS Jens Scheffler  Jarek Potiuk 
    • Jens and Jarek brought up the issue with the Example DAGs which have grown to the point of having become too confusing, because they are used for two different purposes: One for developers testing Airflow before release, and second for users to learn how to use Airflow. 
    • There was clear agreement on separating the examples for those two use cases and Jarek was going to take the lead on that. 
    • Jens was going to take the lead on making the true end user example DAGs more useful with a story telling concept for a broad use case.  
      • UPDATE:
        • Planning Page: Examples Refurbish
        • Discussion in Slack: #sig-examples 
  • Continuing dev calls
    • Vikram suggested that we make these dev calls every two weeks going forward, rather than weekly.
    • There was immediate agreement on this. 
• Action items:
  • Airflow docs UX update to make versions more obvious (Elad, Jarek)
  • Update on Deadline Alerts AIP progress and Github board (Dennis, Rajesh)
  • Edge executor for other languages (Jens, Ash, Kaxil, Vikram)
  • Update devlist about no-go on Python 3.9 early drop support (Jens)
  • Get help on SqlAlchemy upgrade changes (Jarek)
  • Example Dags update (Jens, Jarek)
  • Make dev calls to be every two weeks (Vikram)

22 May 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • Docs site update (Elad, Jarek)
    • Elad said that they could not make much progress on the Docs site update over the last couple of weeks, but would have an update by the next dev call. 
    • Jarek added that there has been significant improvements to the Doc publishing process, led by Pavan.
    • The team agreed with that and gave a shoutout to Pavan, Jarek, and everyone who had contributed to these improvements. 
  • Example Dags update (Jens, Jarek)
    • Jens talked about the small team who have gathered together to start working on improvements to the Example DAGs.
    • Jens shared the wiki page which was being used to track these ideas. 
    • There was a discussion around whether "testing DAGs" are useful to end users and if they should be tagged as "dual use" to reflect that they can be used as an example.
• Development Updates:
  • Deadline alerts AIP-85 (Dennis Ferruzzi, Rajesh B)
    • Dennis and Ramit gave an update on Deadline alerts which included a long, in-depth technical discussion around the implementation options. 
    • There was significant discussion around where the loop to check for alerts should be run as well as where the dag-level callbacks should be run. 
    • The view from the team was that system monitoring i.e. if the scheduler is down is out of scope for this effort and should be done outside of Airflow, i.e. if Schedulers (or Workers) are down, no SLAs would be fired. 
    • After the extended discussion, the summary was that we aligned on a variation of option 1 from the email, with the following elements:
      • Scheduler loop would be used for checking SLA misses
      • If an async notifier is specified, the callback would be run in the triggerer
      • If a sync notifier is specified, the callback would be run on a worker
      • We recommend async callbacks using the builtin Airflow async notifiers
  • Update on the 3.0.x and 2.11 releases Kaxil Naik 
    • Kaxil mentioned that we released 3.0.1 and 2.11 already and that we would be cutting 3.0.2 in about two weeks.
• Discussion Topics:
  • UI Internationalization
    • Brent shared that we had just merged a PR to support internationalization in the UI.
    • There was a lot of excitement around this. 
    • There was an early discussion on what terms should NOT be translated such as "DAG" or "Xcom".
    • This was to be followed up offline.
  • Golang SDK
    • Ash shared that he had raised a PR for an early incarnation of the Golang Task SDK and requested feedback. 
  • Airflow 3 feedback:
    • Pavan shared that he was at the Astronomer roadshow in London the day before and the feedback from people was very positive. 
    • Quote: "People are amazed with Airflow 3 features, dag versioning, Event driven scheduling. everyone looking to upgrade AF3, that's what i hear at least from the people i met :)"

5 June 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • Docs site update (Jarek)
    • Jarek shared a quick, AI generated view of what an updated version selection UI could look like on the docs site. 
    • Jarek also articulated an idea of showing a banner (either left or top) to highlight which Airflow version was being viewed at the moment. 
    • Vikram brought up a point of possibly only showing the major.minor versions of docs, not showing the full list of patch versions as well, to avoid an overwhelming list of choices.
    • Vikram also brought up the issue of the Airflow Core version vs. Task SDK Version. Ash said that most of the docs would probably end up associated with the TaskSDK version, but we weren't there yet. 
    • Jens highlighted that we need to be careful to not overwhelm an initial user with the internal Airflow complexity. 
    • Ended the discussion with a desire to get feedback from UX perspectives, before proceeding forward.
• Development Updates and Presentations
  • UI plugins - Pierre
    • Pierre gave a quick update regarding the status of the UI plugins AIP. 
    • The first kind of plugins being worked on are the iframe plugins and the front-end will take care of rendering them. Templates can be used here with attributes including elements as TaskID and DagID, so that the display can show the context. 
    • The second kind of plugins will be to support React plugins. Additionally, Brent is thinking about dev tools to facilitate the creation of these plugins. 
    • All of this is aimed for 3.1 and quite confident on timing. 
    • Jens raised the question of whether these should initially be marked as "experimental", so that we have the flexibility to change the interface based on initial feedback. 
  • Deadline Alerts - Dennis
    • Dennis raised a database design question for the team encountered during the development of the AIP.
    • Specifically whether to only keep the list of deadline alerts to be checked in the database table, or to keep historical records (such as deadlines met or not met) as well. The point was regarding the performance impact of keeping more data vs. the benefits of keeping this information. 
    • Pierre raised the point that keeping more than "currently in flight" deadlines would be useful from a UI/UX perspective.
    • Dennis was wondering if an index would be sufficient to mitigate performance. Ramit suggested using a view and Vikram suggested that in the worst case, you could have a separate table.  
    • After some discussion, the action item was for Dennis to make the decision using tests / benchmarks based on expected sizes. 
  • Internationalization - Shahar
    • Shahar presented the proposed internationalization policy to address challenges with maintaining translations across 300 languages with only 70 committers. 
    • Shahar mentioned that German, Polish, Dutch, and Mandarin translations have already been merged.
    • And that, Hebrew and Arabic translations are pending review.
    • The presentation covered details such as assigning two roles for each locale: a translation owner responsible for language aspects and a code owner handling technical aspects, with translation sponsors to potentially be non-committers. 
    • It also referenced a process for release managers to notify code owners about missing translation items as part of cutting the release. 
    • The release process impact and timing prompted significant discussion with the team expressing concerns about increasing the burden on release managers. 
    • Jarek suggested starting the release process a week earlier to help accomplish this.
    • Shahar explained that the fallback policy was to use English when terms are not translated, but this raised UX concerns. 
    • The team agreed to limit the number of languages to start with to reduce overhead and implement a system for periodic reminders about missing translations. 
    • Shahar took the action item to follow up with potential refinements to reduce release friction.

26 June 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • Docs site update (Jarek)
    • Jarek mentioned that there is an issue created for the work here.
    • The docs build process is a lot better now because of the work led by Pavan, but the actual result still needs work.
    • Jarek also spoke with Kamil about the original work done on the docs site and his availability to help. 
  • Update regarding internationalization impact on release process (Shahar)
    • Shahar said there isn't any change to the process from prior discussions. 
    • The expectation is for the Release Manager to announce that we will cut a new release and that serves as a notification for the code owner to finish translation by a particular date. 
    • The medium term intent is to publish translations outside of "Core Airflow". The initial concept here is to have translations released similar to provider releases and not to be tied to the core Airflow software release. This could be just a file release or even a data update to the database, to be figured out. 
    • There was a lot of enthusiasm from the team about how this project has happened so quickly and all the community involvement here. 
• Development Updates and Presentations (Please keep the updates to less than 10 minutes):
  • UI Plugins Update (Brent, Pierre)
    • Brent and Pierre did a quick demo of the UI plugins work and shared the progress to date. 
    • This included the common abstraction for the UI plugin regardless if the UI should be in a new tab or an iFrame rendering
    • The team was enthusiastic about the great progress here.
  • Deadline Alerts: Update on conclusion for database tradeoff (Dennis)
    • Ramit shared that the development completion target for Deadline alerts is the end of July
    • Ramit also shared that right now when a DAG is created with a deadline alert, the deadline would be created and stored in the database
    • The scheduler loop change work is upcoming and the database change approach will be figured out as part of that. 
    • Brent asked about needing to sync on the API endpoints for the same. Dennis to sync with Brent over the course of the next week.
• Discussion topics:
  • Multi-team discussion from dev list - (Jarek)
    • There was long spirited discussion around the multi-team effort. Jarek opened by asking for comments. 
    • Vikram went first, saying that he wanted to make sure that we were not trading off the simplicity of Airflow starter experience for individual data engineers with the complexity needed for large organizations wanting granular operational procedures. Vikram asserted that as an open source project, viral adoption has been the key to Airflow's success and that simplicity was important to maintain and in fact make better. 
    • Jarek immediately agreed with that assertion and said that was a key point in his desire to also simplify this AIP to multi-team, from the original, far more complex, multi-tenancy proposal. 
    • Vikram also said that he was surprised by the lack of reference to AIP-82 Event driven scheduling, since this was such a commonly used mechanism for segmenting Airflow deployments across teams from a ownership perspective, following classic distributed system principles. 
    • Jens also chimed in saying that we should perhaps articulate this better in the context of how to set up Airflow for large enterprises. 
    • There was a significant level of debate around the complexity vs. simplicity tradeoff, with Rajesh stating this is probably the most common requirement that MWAA hears from their customers. Niko also chimed in saying that we should be willing to accept some tradeoffs to simplicity. 
    • There was a further discussion on what needs in priority order needed to addressed here, with Kaxil raising the point that this raised significant operational considerations and he wanted to make sure that we were not getting in over our skis in complexity with quota management, etc. 
    • This ran out of time without an explicit decision in the meeting itself.
    • Jarek concluded by asking people to comment back in the AIP soon, so that feedback could be addressed in writing. 
  • Call for actions ...ah.. review! Example Dags! (Jens)
    • Jens raised a "call for action", asking people to help review the work on the Example DAGs and to contribute

3 July 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • None
• Development updates and presentations: 
  • Airflow 3 adoption (Vikram)
    • Vikram shared that the download numbers show encouraging signs of 3.0 adoption, with about 12% of all downloads for the last month being for 3.0 placing it in third place, with 2.10 in the lead with 29% and 2.6 (still puzzling) in second place with 16%.  
    • Vikram also noted that 2.11 which was released after 3.0 is at 5% of downloads, which showed that more people were willing to go to 3.0 directly from earlier 2.x versions. 
    • Jens noted that it was still a very good thing that we had released 2.11, since 5% of people had upgraded to it from earlier 2.x versions, presumably because they were not yet ready to upgrade to 3.0. 
    • The team noted that for several organizations there were a couple of functional gaps, notably UI Plugins and "run as user" which blocked the upgrade to 3.0. Amogh responded that "run as user" is available as part of Airflow 3.0.3.
    • Vikram noted that the two big known gaps i.e. UI plugins and Deadline Alerts are targeted for the 3.1 release.
    • The team noted that we should make an effort to clean up any such small blockers in the 3.1 release. 
    • The ask for the community is to please highlight such gaps encountered in the dev list or the Airflow 3 dev slack channel. 
  • CLI update (Bugra)
    • Bugra walked through the Github project for this AIP as well as the current status of the work. 
    • Bugra stated that the project would be ready to be integrated into the release process in a couple of weeks.
    • Bugra also volunteered to help in the release process as needed. Jarek pointed out that some additional work may be needed around tests as part of getting ready for release. 
  • Airflow 3.1 update (Vikram)
    • Vikram shared the overall scope and timing of the Airflow 3.1 release as documented on the wiki page. 
    • The one area of significant discussion was the Human in the loop as detailed below.
• Discussion topics:
  • Human in the loop
    • Jens started the discussion here by pushing for the Human in the loop feature to be built and released as a separate provider, not within the standard provider package.
    • Kaxil vocally advocated for this to be in Core Airflow for simplicity and walked through the initial PR created by Wei to illustrate his rationale. 
    • Jarek vehemently agreed with Kaxil. 
    • Pierre also agreed with Kaxil, saying that he would prefer UI plugins to be only used by users, not by us as Airflow contributors. 
    • This then led to the discussion that we have gone too far with the Provider separation from Core Airflow. Jed brought up the point that he would like to get the Executors out of the Providers into a different package. 
    • The discussion ended with the decision to make the "core" Human in the loop functionality as part of Core Airflow. 
  • Code sharing discussion (Ash)
    • The prior discussion also led to the code sharing discussion between Task SDK and Core Airflow, following the thread brought up by Ash in the dev list. 
    • This was a great discussion with a lot of participation from everyone on the call. 
    • Ash to summarize this.

17 July 2025

Proposed Agenda

Attendees

Summary

• Catch-up on action items from last call:
  • None
• Development Updates
  • UI Plugins Update (Pierre and Brent)
    • Pierre shared that the React plugins implementation is nearly complete, including embedded web applications now working inside the Airflow UI. 
    • Pierre also said that key technical challenges have been resolved including react library sharing between plugins and Airflow Core.
    • Pierre raised one issue which had been open about where to keep the CLI/Python scripts to bootstrap React project. Based on team feedback, it was resolved to have those live within dev-tools. 
    • Brent gave a quick demo of the React plugins working in the Airflow UI
    • Outstanding work on this stream before the 3.1 release includes documentation and examples to demonstrate both iFrame and React app integration, as well as CSS fine-tuning for wider screen layouts. 
  • Task SDK Update (Kaxil, Ash, and Amogh)
    • Kaxil shared that multiple parallel work streams are active right now for this AIP
    • Ash is prototyping the common object sharing approaches (symlink vs. rendering). 
    • Utility code migration is happening based on that to appropriate locations (task SDK vs. shared). Serialization structure changes are also being done to support this. 
    • The Core Operator cleanup is happening, largely based on TP's work. Base Operator is now removed from Airflow models. Similar removal planned for Map operator and other core Operators. 
    • Branch Operator bug identified and is being addressed. 
  • Python 3.13 (Jarek)
    • Jarek shared that Python 3.13 is now available for all users and gave a quick demo of the same. 
    • Jarek also said that the Flask Application Builder (FAB) dependency has now been moved and a truly FAB-less Python 3.13 deployment is possible. 
    • Jarek also shared that Python 3.13 fixed a long-standing file description duplication bug. This bug could explain the recent "too many files open" issues in supervisor tasks. 
  • Deadline Alerts Update (Ramit and Rajesh)
    • Ramit shared that he had done performance testing on different approaches using a standalone Python script on Postgres, with 10 million rows. 
    • He tested 3 approaches: Two separate tables, view-based single table, and indexed single table. 
    • The index approach (state + deadline time) had the best performance and therefore led to the implementation approach of adding a state column to the deadlines tables. 
    • Ramit also shared that async callbacks would be run on the triggerer and synchronous callbacks on the workers. 
    • Rajesh shared that the deadline detection code is nearly complete, with the Slack and SMTP notifiers being implemented as async functions. Documentation is also in progress, with an estimated dev completion in about 2 weeks. 
    • There were questions around performance considerations and running under scale. 
• Discussion Topics
  • Airflow 3 adoption blockers
    • Vikram started the discussion with the commonly raised blockers that he had seen or been asked about:
    • Changes in imports: Vikram said that there seemed to be a significant lack of awareness of the upgrade utilities and the upgrade guide. Pavan chimed in saying that he had seen this too. There was some discussion about how to highlight this better on the docs site or in the UI, without a clear resolution. 
    • Missing UI elements such as UI filters: Brent chimed in saying that these have been identified as Github issues and are in the process of getting resolved. Brent also asked for community help here. 
    • Database size causing long migration times, due to long database upgrade times: Again, something covered in the upgrade guide, with the recommendation to clean up the database before upgrade. 
    • Vikram also said that he has been using the "affected_version:3.0" issue tag to track adoption issues and then with priority. 
    • The ask for the community is to "self-assign" adoption issues before the next meeting.
• Action Items
  • Pierre: Continue React plugin dev tools development
  • Brent: React package requirements optimization after vacation
  • Jarek: Investigate file description issue workarounds for pre-3.13
  • Dennis and Ramit: Finalize deadline alerts implementation
  • Kaxil, et al: Continue TaskSDK / shared code changes
  • Vikram: Clean up the Airflow 3.1 wiki structure and add Augusto Hidalgo to discuss Benchmark updates to next dev call. 
  • Community: Create, tag, and self-assign (if possible) Airflow 3 adoption issues before next dev call

31 July 2025

Proposed Agenda