Page History

...

Schedule: Fortnightly (once in 2 weeks) on Thursday at 4pm GMT/UTC

Calendar:

Table of Contents

...

Attendees

Summary

• Follow-up on action items from the last call:
  • AIP-83 amendment aka Data completeness discussion follow-up Daniel Standish Tzu-ping Chung
    • Daniel Standish confirmed that the vote had been started on dev list and encouraged the team to respond. 
• Development Updates:
  • Update on AIP-82 External Event Driven Scheduling Vincent BECK 
    • Vincent demonstrated the implementation of the AssetWatcher concept built on top of FileOperator, which enables a DAG to be triggered based on the arrival of a file in an external blob storage. 
    • Vincent also did a second demo based on watching for two different files and that the DAG would be triggered based on either file arriving in blob storage. 
    • Vincent also showed a DAG which could be triggered based on a message arriving in an SQS queue. 
    • Vincent highlighted that DAG authors need to be careful on what triggers to use for invoking DAGs, since it is possible to have additional DAG runs because of persistent events. There was discussion around this and the agreement was to take that discussion offline. 
  • Update on AIP-72 Task Execution Interface aka Task SDK Ash Berlin-Taylor Kaxil Naik Amogh Desai 
    • Ash highlighted that the team was chipping away on the work. He highlighted that Mapped Tasks was close to completely working based on the TaskSDK. 
    • He said that once Mapped Tasks was done, Celery and Kubernetes executors would be moved over to use the TaskSDK. 
    • He also said that after Mapped Tasks was done, the Trigger changes was the last "big feature" to be done and then there was going to be the long tail of fixes. 
  • Update on AIP-38 Modern Web Application Pierre Jeambrun Brent Bovenzi 
    • Pierre demonstrated the triggering and clearing of DAG runs and Tasks. 
    • Pierre said that Brent is working on the Grid view and it isn't merged yet, but close
    • Pierre said that after the alpha, we would be able to use the new API exclusively and then we could delete the old UI and API. 
    • Pierre also said that they were making progress and no particular blockers. 
    • Brent demonstrated the GridView and said how they were looking forward to getting feedback as part of the Alphas. 
  • Update on Provider packages new structure Jarek Potiuk  
    • Jarek did a quick walk through about moving the providers to the new structure as a prelude to the new packaging work upcoming in February. 
    • He said that he would ask for help from the larger team on this work, since it could not be 100% automated. He said that his script automated 90+% of the work,  but needed human intervention for the last little bit. He walked the team through a quick run of this script to illustrate. 
• Discussion Topics
  • Milestone update and progress Vikram Koka  (actually Ash Berlin-Taylor ) 
    • Ash walked through the overall progress update, since Vikram was out sick and couldn't make the call. 
    • Ash gave a quick overview of where we are based on the dev milestone wiki page. He highlighted that we are shooting for getting the first alpha builds out next week.
    • Elad asked for clarification about the "feature freeze" beyond the alpha milestone. Ash took the action item to post a message to the dev list about it. 
    • Rajesh asked about where we are on the timeline and different AIP owners went through their progress updates as compared to the published timeline
    • Jed, Vincent, Pierre, Vincent, Bugra, and Daniel Standish all went through their AIPs and expressed confidence in the schedule as they saw it. 
    • Ash summarized the discussion to express that the first alpha would have rough edges, but that the team felt positive about the progress and where we are at this point.
  • AIP-81  CLI Discussion: Task Commands Remote vs. Local Buğra Öztürk  (Doc link)
    • Bugra wanted to have a discussion around some of the CLI Task commands which would not be possible through the Task API. 
    • Because of a lack of time, the proposal was to have that discussion async. 
    • Bugra suggested reviewing the doc also for a discussion around local vs. remote commands.
  • Airflow 2.11 release discussions Jens Scheffler (Jarek Potiuk stepped in because of bad Wifi connection for Jens)
    • Jed said that Utkarsh is planning to cut 2.10.5 next week.
    • Topics raised demanded/needed for 2.11
      • airflow config lint command (but still can be added in 2.10-line or independent as well, no big driver)
      • Migration tooling (which is still WIP)
      • No other things raised
    • Generation notion: Not critical at the moment. At least nobody raised a deadline or critical feature (yet)
• Action items on/before next dev call:
  • Ash to send out an email to the dev list regarding "feature freeze" guidelines, both to developers and committers about "merging PRs". 
  • Team to review Bugra's doc around CLI changes

...

Attendees

Summary

• Follow-up on action items from the last call
  • "Feature freeze" guidelines Ash Berlin-Taylor 
    • Ash confirmed that he had sent out the email to the dev list with his thoughts around how we as committers and PMC members need to be very risk-aware as we merge changes into main until the 3.0 release is shipped as GA. He also mentioned how fundamental the 3.0 changes were to the code and that they were much larger than even the 2.0 changes. Jarek and Vikram agreed with the risk-aware sentiment.
  • AIP-81 CLI changes Buğra Öztürk 
    • Bugra quickly walked through the doc already shared with the dev list (Doc link). 
    • Bugra walked through how local vs. remote commands now needed to be separated because of the TaskSDK changes in 3.0. There was general agreement about this.
    • There wasn't much discussion in the call around the detailed architecture and sequence diagrams in the document, and "the ask" was for the team to review the document and leave comments async. 
• Development updates
  • Update on AIP-38 Modern Web Application: New Trigger Form UI by Shubham Raj and Jens Scheffler 
    • Shubham walked through a demonstration of the new trigger form developed in the Airflow 3 UI. 
    • The demo went through flawlessly and there was general admiration for the work done. 
    • Shubham also confirmed that this work had been merged into main already. 
  • Test plan update Rahul Vats 
    • Rahul walked through the Airflow 3.0 Test plan which now includes the status of the formal testing effort in addition to the plan itself. 
    • Rahul described how the team was tracking development and test progress, while testing and reporting issues on the alpha builds. 
    • There was general appreciation of the work being done. Vikram urged Rahul to ask for help from the broader team as and when needed.
• Discussion topics
  • Milestone / workstream update Vikram Koka 
    • Vikram thanked the whole team for all the great work which had been done towards getting to the alpha builds. He mentioned how hard cutting the first alpha had been and appreciated Jed among others, who had almost a 24 hour day in order to make it happen. 
    • In order to be more predictable, Vikram asked the team to cut alphas (and betas in the future), at least every Thursday, so that we could have predictability. 
    • Jarek agreed, citing the heuristic "If it hurts, do it more often". Jed agreed, saying that the alpha2 being cut right then has been significantly smoother than alpha1. Jed also asked everyone on the team to be aware that the alpha builds would start roughly around noon India time every Thursday. 
    • By the time of writing, Airflow 3.0.0alpha2 had been shipped out. 

...

Attendees

Summary

• Follow-up on action items from the last call
• Development Updates and Presentations:
  • Update on AIP-72 Task Execution Interface aka Task SDK Ash Berlin-Taylor Amogh Desai
    
    • Ash walked through a demo of the Triggerer support with the new TaskSDK. He shared how this was largely in place, with the exception of connection access.
    • Ash mentioned that the Triggerer is now split into two processes, using the supervisor pattern from the Task SDK. 
    • Amogh gave an impromptu demo of the Kubernetes Executor now working with the TaskSDK. 
  • Update on AIP-66 DAG Bundles & Parsing Daniel Standish  
    • Daniel Standish shared a recording of the demo covering the stale DAG bundle cleanup process.
    • Since Workers now pull code from remote sources to run tasks and since this is stored locally to reduce latency, there is cleanup required to avoid these piling up on the worker nodes. 
    • He said that "old / stale" could be defined by time or number of older versions.
  • Test plan update Rahul Vats 
    • Rahul covered the testing status as documented in the Airflow 3 Test Plan
    • Rahul covered the testing status of Alpha 3 and the confidence levels of the different areas including the UI and CLI. He mentioned that these are still in progress, since development is still ongoing. 
    • He mentioned that they have started testing with focus on specific AIPs and are now reporting status by AIPs as well.
    • He asked the AIP owners to review the test cases and validate them as they make changes. And for the AIP owners to ensure that there was at least one run on each before the beta releases as well. 
  • AIP-81 CLI Update Buğra Öztürk
    • Bugra covered the updated design for the Authentication work for the CLI. 
    • Bugra asked for help from the committers on the call to review the PRs coming in for the CLI AIP, since some of the PRs were sitting for a while in review status.
    • Bugra commented that some of the authentication work would probably need to come in Airflow 3.1, rather than in 3.0. 
  • Update on AIP-38 Modern Web Application Brent Bovenzi 
    • Brent demonstrated an updated UX in the Airflow UI, with the Grid View or Graph View (based on the user preference) on the left hand of the page to give a consistent view with the modal views on the right hand side. 
    • The updated UX drew a very positive response from the team on the call. 
    • Brent showed how the new UX addressed color-blindness issues with icons and was more accessible. 
• Discussion topics:
  • Python 3.13 (not only) -driven dependency issues Jarek Potiuk 
    • Jarek highlighted these three issues with 3.13.
    • Pendulum: He said that the Pendulum maintainer got back to him and we don't have to do anything in this area. 
    • google-re2 → can we replace "root" usage in UI: Jarek said that Google-re2 issue is a bit more pervasive in Airflow and that after speaking with Bugra is solvable in the CLI. He was concerned about its use in the UI and API. Brent and Pierre said that this was no longer being used, so it is not needed anymore.
    • FAB (and sqlalchemy and pandas) → to FAB or not to FAB: The dependency conflict between SqlAlchemy, Pandas and FAB requires us to probably take out FAB as quickly as possible from Core Airflow. 
  • Removing the old UI Jedidiah Cunningham 
    • Jed said that he is targeting next week to remove the old UI, before we ship Beta 1. 
    • Jed also proposed moving that branch to the main repo for easier collaboration from the larger team and there was general agreement.  
  • Bump minimum version for Airflow 3.0 to Python 3.10 (~6months ahead of EoL schedule) Ash Berlin-Taylor
    • Ash proposed this and asked if there were any strong feelings against it. 
    • There was significant discussion about what this meant for Airflow 2.x vs. 3.x including the implications for Provider releases going forward. 
    • The decision was to take this to the mailing list. 

...

Attendees

Summary

• Follow-up on action items from the last call:
  • Airflow 3.0 minimum version to be Python 3.10 Ash Berlin-Taylor 
    • Moving to Python 3.10 is not a blocker.
    • Ash said that he didn't have time to make this change in time for the 3.0 release. 
    • Therefore, decided to bump to after the 3.0 release and do this as part of the standard Python support cadence.
• Development Updates and Presentations:
  • Update on AIP-72 Task Execution Interface aka Task SDK 
    
    • Short circuit and Branch operators Shahar Epstein 
    • Shahar walked through the changes required for these operators.
    • Specifically, the change from direct DB access to using the TaskSDK for updating XComs and the state of the task. 
  • Test plan update Rahul Vats
    • Rahul walked through testing status with Beta1, including testing migrations (upgrades and downgrades). 
    • Rahul said that confidence level with respect to the various AIPs was increasing and that he expected the Grid of confidence by AIP to get a lot more colorful over the next week.  
    • He said that since the "DB isolation will be enforced starting Beta2", he expected more test DAGs to start failing. 
    • When asked about UI test automation, he said that the UI testing was manual and could not be automated yet because it was changing every day. 
    • The team was very appreciative of Rahul's update.
  • Milestone update Vikram Koka 
    • Feature completeness and road to RCs
    • Vikram shared the updated Airflow 3 dev milestones and said that he wanted the beta 4 release targeted for the week of March 17th, to be feature complete and the final beta release. 
    • Vikram asked for all open PRs to be completed, tested, and merged before beta 4. He urged contributors to ask for help if needed to meet this milestone.
    • Vikram asked for patience for the community over the next few weeks, since open issues could not be allowed to linger and would be reassigned if the assignee was unresponsive.
    • There was general consensus from the team. 
    • Vikram also brought up AIP-68 Extended Plugin interface for React Views, saying that this would need to be deferred to 3.1 because of timing. There was general agreement on this as well. 
  • DevEx improvements as result of provider's distribution separation and devel-common Jarek Potiuk
    • Jarek shared a quick update on the recent DevEx improvements for Airflow development based on UV.
    • He shared how this would make Provider development a lot easier than previously the case as well as benefiting TaskSDK development going forward.
• Discussion topics:
  • Packaging update - open items (Ash Berlin-Taylor , Jedidiah Cunningham Jarek Potiuk )
    • A quick update shared was about making the Airflow repo a meta package, with Core Airflow becoming a project within the Airflow repo. 
  • Edge provider naming issue (and PEP 752 as long term solution) Jarek Potiuk 
    • Jarek brought up the issue that this name is already taken on the PyPI registry and that this is related to the PEP 752 discussion. 
    • When this PEP is implemented, ASF will take the "Apache" name, but this is still in progress with the POC to be implemented this week. 
    • The immediate action is to hopefully get this name (Edge) from the PyPI registry. 
  • FAB next steps - Jarek Potiuk 
    • Jarek confirmed that he is working on enabling Airflow to use FAB 5.0, since this is holding us back from upgrading SqlAlchemy. 
    • Jarek outlined multiple options: 
      • Option 1: Wait for 5.0, but that is 2 to 3 months
      • Option 2: Take the needed parts of FAB and incorporate into the FAB provider; Then upgrade to SqlAlchemy 2.
      • Option 3: Stay with SqlAlchemy 1 for now, but the security issues with it are problematic
    • The team agreed that option 1 was a non-starter, and that we should on option 2, but be prepared to switch to option 3, if we run into issues with the SqlAlchemy upgrade being so close to release. 

...

Attendees

Summary

• Development Updates and Presentations
  • AIP-81 CLI Authentication Design Update Buğra Öztürk Jedidiah Cunningham 
    • Bugra and Jed explained how AIP-81 would create a completely separate remote CLI package, which will be "airflow-ctl".
    • For local execution, all of the CLI commands (which are DB backed) will just work. 
    • All of the merged code for remote CLI execution will be moved into airflow-ctl as part of beta4 and will also simplify packaging and release timelines. 
  • AIP-72 Task Execution Interface aka Task SDK Update Ash Berlin-Taylor Kaxil Naik Amogh Desai 
    • Kaxil gave an update on the AIP-72 detailing that JWT Integration, Custom XCom backend, and Secrets backend work is all completed. 
    • He added that the Triggerer backend now works without going through the public API, to avoid all the authentication passing complexity.
    • Kaxil said that the Task Notifiers is the one open item, which TP is currently working on to finish.
    • Amogh added that the Secrets backend also enables a separate secrets backend for Workers as compared to the Secrets backend configured for the Scheduler. 
    • Ash added that each Task now has a JWT token that is associated with it, bound to a Task Try ID. 
    • Ash also added that the Execution API versioning was now merged for the Task Execution Interface and implemented Calendar versioning, based on Cadwyn.
    • Ash added that this was only for the Task Execution Interface, not for the public API, nor for UI internal API. 
    • Shahar added that he had heard that some AWS provider system tests were breaking right and that he would be investigating. 
  • Test plan update Rahul Vats
    • Rahul shared the updated Airflow 3 Test plan and walked through a quick progress update by AIP. 
    • Rahul asked everyone on the team to put on their testing hat, treat beta4 as being the internal testing vehicle, and to test it out, so that we can find more bugs before release.
    • Rahul also added that AIP-81 needed to be tested for CLI regressions. 
• Discussion Topics:
  • Packaging Update Jarek 
    • Jarek highlighted the big PR which introduces the "airflow-core" package within the main airflow repo.
    • He highlighted how there is no longer an "airflow" folder in the main repo.
    • Instead, there is "airflow-core" which has all the key items inside it including the pyproject home.
    • The second thing we have is an airflow meta package which has all the configuration for builds and additional elements for packaging and distribution. 
    • Jarek said that this was still work in progress, with a few things to be fixed, but was mostly done.
    • Jarek also spoke about how UV works with the new packaging structure and stressed how this simplifies Airflow development.
  • Cleanup of Airflow 3 open items in Github Elad Kalif 
    • Elad asked all the AIP owners to review the list of Github issues originally targeted for Airflow 3 and still open
    • Specifically, these are airflow3.0:candidate and airflow3.0:breaking
    • Elad made the point that these should either be closed as having been done or moved to a subsequent release tag as part of the Airflow 3 RCs. 
• Action items
  • What's critical to finish up
    • Jens brought this up in the context of the UI, if there are open critical items to be completed before 3.0 release
    • Brent took the action item to highlight this async and shout out if anything needed help
  • Change dev calls to be weekly until release
    • Elad suggested that we make the dev calls weekly, since we are very close to release.
    • Vikram to update the calendar invite to be weekly and to send out a reminder if a meeting is required that week. 

...

Attendees

Summary

• Follow-up on action items from last call
  • Cleanup of Airflow 3 open items in Github - specifically airflow3.0:candidate and airflow3.0:breaking
  • There was consensus already either closing the "airflow3.0:candidate" issues or tagging them to be "airflow 3.1+" as appropriate based on where we are today
  • The harder discussion was around "airflow3.0:breaking" issues: 
    • Some of these were actually done and need to be just closed. Ash, Jens, and Kaxil had identified at least one each. 
    • There was a discussion about others which could not be done in the time left because of ripple effects and migration challenges. The team decided that we would discuss a mechanism for introducing deprecations / breaking changes without having to be as ultra-strict on semantic versioning as we are today. To avoid cognitive overload, this discussion will be brought to the dev list in May. Jarek took the action item for this. 
• Development Updates and Presentations
  • Test Plan Update Rahul Vats 
    • Rahul walked through the Airflow 3 Test plan and showed how it was now updated to also reflect the status of AIPs 81 and 82, thanks for Bugra and Vincent. 
    • He also spoke about the integration testing with Providers under way. 
    • For AWS providers, Rahul mentioned that he had been working with Dennis and that the AWS providers were mostly working, with only a couple of issues to be resolved.
    • For Google Providers, he mentioned that the Google Provider dashboard was broken and that the Google team was working on fixing the dashboard. Eugene from the Google team confirmed the same. 
  • Open issues to be resolved 
    
    • With respect to the Branch Python Operator raised by Shahar last week, Pavan said that he had fixed that issue. There was some discussion if there were any other issues regarding the BranchPythonOperator, but there was nothing on the board at this time. 
    • Kaxil mentioned that there is a high priority issue with BaseSensor, which he was working on fixing right now. 
    • There is also a critical issue with Triggerer , which Pierre is fixing and close to merging at this time. Pavan to fix the Deferrable Operator issues based on this. 
    • A high priority DAG Versioning UI / Grid View issue was also discussed and Jed said that he had a fix for that on his dev branch based on work with Brent, which he would be merging shortly.
    • Rahul mentioned that he and the team were also looking at UX issues between Airflow 3 and Airflow 2.x, and would raise them as appropriate.

...

Attendees

Summary

• Follow-up on action items from last call:
  • Edge Provider naming Elad Kalif  Jarek Potiuk:
    • Elad explained that the Edge provider will be called "edge3" for many reasons.
    • It's ready to be used and tested.
    • Once the current Provider vote is closed, Elad will cut a new wave including Git and Edge3 for an extra 24 hours voting period and then these would be released. 
• Development Updates
  • Test plan and Issue statistics update Rahul Vats and vikram 
    • Rahul shared the current state of testing based on RC3 and said the regression testing on RC3 is ongoing.
    • Rahul felt that the current confidence level is high.
    • Rahul also shared the issue statistics for Airflow 3 with:
      • Alpha releases: 40 issues reported and closed.
      • Beta releases: 129 issues reported and closed.
      • Release candidates: 102 issues reported, with 64 of them already closed. 
    • Rahul also said that there are no currently open blocking issues with the release.
    • He also expressed how many issues were fixed directly in PRs without the issues even being reported.
    • Vikram added that he had a target of at least 200 issues to be reported in the combination of Alpha, Beta, and RCs.
    • He said that seeing a total of 271 issues reported was great to see and gave confidence in testing of the release. 
• Discussion Topics
  • Dev workflow for 3.0+ Kaxil Naik 
    • Kaxil explained that we now have project milestones for 3.0.1, Airflow 3.1.0, and Airflow 3.1+. 
    • He articulated how these should be used for new work going forward with issue resolutions for 3.0.1, vs. new feature work being targeted for 3.1.0 and 3.1+ 
  • Docs approach towards AF 3 users Elad Kalif 
    • Elad reminded everyone that our docs are versioned and asked everyone to write docs assuming that the users coming are using Airflow 3.0. 
    • Elad asked doc writers to avoid "if Airflow 2, do this" and "if Airflow 3, do this" within the docs, since that would be confusing to the reader. 
  • Ongoing support for Airflow 2 Kaxil Naik 
    • Kaxil brought up the topic of "maintenance support for Airflow 2", once Airflow 3 is released.
    • Kaxil showed that we had supported Core Airflow 1.10.x maintenance releases for 6 months after the release of Airflow 2. 
    • There was significant discussion around this including the separation of maintenance of bug fixes vs. security fixes. 
    • Elad also brought up the separation of maintenance updates for Core Airflow OSS vs. Providers
    • The decision from the team was to take this discussion to the devlist. Kaxil to start a mailing list thread.

...

Attendees

Summary

• Development Updates:
  • Update on the 3.0.1 release Kaxil Naik 
    • Kaxil mentioned that over 200 commits are part of 3.0.1 RC1 including many usability improvements.
    • He said that the voting for RC1 ends on Monday, the 12th of May. 
    • Jarek added that we would probably need an RC2 because of the inclusion of an older version of an Amazon provider. 
  • 2.11 release scope / timing Kaxil Naik 
    • Kaxil mentioned that the decision between 2.11 vs. 2.10.x was made because of several features and changes in the 2.x branch, which meant beyond a patch. 
    • Specifically, he mentioned the new Timetable addition as well as the PR for making sure that the metrics are aligned for consistent reporting. 
    • Another reason for the feature release was also to add support for "airflow config lint" which enables validation of the airflow configuration before upgrading to Airflow 3. 
• Discussion Topics:
  • 3.0 release feedback / discussion topics (All)
    • Vikram shared a quick update on the Airflow downloads by version over the last two weeks since Airflow 3 was released. 
    • Vikram highlighted that Airflow 3 has seen significant adoption with about 7.5% of the total Airflow downloads. The highest downloaded version continues to be 2.10, which is the last release on the 2.x release stream, with about a third of all downloads. Of course, these metrics should be taken with the usual grain of salt, because they are susceptible to image caching and CI processes.
    • Elad asked where features "missing" in Airflow 3 were documented. Vikram said that these were documented as breaking changes within the Upgrading to Airflow 3 doc. The team identified a couple of missing elements, which Elad volunteered to add. 
    • Jarek said that there was significant user confusion around upgrading to 3.0 with respect to DAG writing practices, specifically around imports. After much discussion, the conclusion was that one of the core problems here was the UX around doc versions on the website, being too subtle and not obvious to users about the differences between 2.x and 3.0 syntax. The action item for Jarek and Elad was to look at the docs UX to see if this could be made more obvious.
  • 3.1 release scope / timing discussion Vikram Koka  
    • In the follow-up discussion about the scope of 3.1, Vikram explained that the Airflow 3.1 scope was largely based on user feedback of 3.0 including the prior discussion about missing features.
    • The early draft of Airflow 3.1 scope is documented on the wiki for discussion. 
    • Vikram mentioned that Dennis couldn't make the call today, but asked for reviews on his PR which was early work towards Deadline Alerts, which replaces SLA alerts in Airflow 2.x and is currently scoped for Airflow 3.1. Rajesh added that he was confident with the timing, since Dennis and others were actively working on this and would be ready to update the team in two weeks' time.  
    • Jarek raised the question a
    • Other language support
    • Ash shared an update on the Go SDK. He shared a link to the branch with the draft incarnation and said that a PR would be coming soon. 
    • Vikram shared an update with the early Java SDK branch. Jens chimed in referencing the offline discussion with Vikram around leveraging the Edge API for these other languages.
    • The action item was for Jens, Ash, Kaxil, and Vikram to discuss async on any changes needed to the Edge provider. 
    • Ash raised a concern that we may not see significant adoption until the DAGs were in other languages as well. Elad and Jarek argued against the viewpoint feeling that a single task in a different language would be very powerful for application integration. Elad said that "The notion of executing a task directly without a wrapper DAG is a request I hear from time to time (already)". 
  • FAB upgrades (4.6.2 first, 5.0.0 later) Jarek Potiuk 
    • Jarek raised the FAB upgrade issue which we had deferred a couple of months ago to post 3.0. He said that we are in a much better situation now, but that he needed help with the SqlAlchemy migrations and would follow up on the dev list. 
    • As part of this conversation, there was a discussion about dropping support for Python 3.9 early. However, there was no appetite for that from the attendees because of all the complexity around doing this selectively for Core Airflow vs. Providers. This was going to be conveyed back to the dev list discussion by Jens.
  • Example DAGS Jens Scheffler  Jarek Potiuk 
    • Jens and Jarek brought up the issue with the Example DAGs which have grown to the point of having become too confusing, because they are used for two different purposes: One for developers testing Airflow before release, and second for users to learn how to use Airflow. 
    • There was clear agreement on separating the examples for those two use cases and Jarek was going to take the lead on that. 
    • Jens was going to take the lead on making the true end user example DAGs more useful with a story telling concept for a broad use case.  
      • UPDATE:
        • Planning Page: Examples Refurbish
        • Discussion in Slack: #sig-examples 
  • Continuing dev calls
    • Vikram suggested that we make these dev calls every two weeks going forward, rather than weekly.
    • There was immediate agreement on this. 
• Action items:
  • Airflow docs UX update to make versions more obvious (Elad, Jarek)
  • Update on Deadline Alerts AIP progress and Github board (Dennis, Rajesh)
  • Edge executor for other languages (Jens, Ash, Kaxil, Vikram)
  • Update devlist about no-go on Python 3.9 early drop support (Jens)
  • Get help on SqlAlchemy upgrade changes (Jarek)
  • Example Dags update (Jens, Jarek)
  • Make dev calls to be every two weeks (Vikram)

...

Attendees

Summary

• Catch-up on action items from last call:
  • Docs site update (Elad, Jarek)
    • Elad said that they could not make much progress on the Docs site update over the last couple of weeks, but would have an update by the next dev call. 
    • Jarek added that there has been significant improvements to the Doc publishing process, led by Pavan.
    • The team agreed with that and gave a shoutout to Pavan, Jarek, and everyone who had contributed to these improvements. 
  • Example Dags update (Jens, Jarek)
    • Jens talked about the small team who have gathered together to start working on improvements to the Example DAGs.
    • Jens shared the wiki page which was being used to track these ideas. 
    • There was a discussion around whether "testing DAGs" are useful to end users and if they should be tagged as "dual use" to reflect that they can be used as an example.
• Development Updates:
  • Deadline alerts AIP-85 (Dennis Ferruzzi, Rajesh B)
    • Dennis and Ramit gave an update on Deadline alerts which included a long, in-depth technical discussion around the implementation options. 
    • There was significant discussion around where the loop to check for alerts should be run as well as where the dag-level callbacks should be run. 
    • The view from the team was that system monitoring i.e. if the scheduler is down is out of scope for this effort and should be done outside of Airflow, i.e. if Schedulers (or Workers) are down, no SLAs would be fired. 
    • After the extended discussion, the summary was that we aligned on a variation of option 1 from the email, with the following elements:
      • Scheduler loop would be used for checking SLA misses
      • If an async notifier is specified, the callback would be run in the triggerer
      • If a sync notifier is specified, the callback would be run on a worker
      • We recommend async callbacks using the builtin Airflow async notifiers
  • Update on the 3.0.x and 2.11 releases Kaxil Naik 
    • Kaxil mentioned that we released 3.0.1 and 2.11 already and that we would be cutting 3.0.2 in about two weeks.
• Discussion Topics:
  • UI Internationalization
    • Brent shared that we had just merged a PR to support internationalization in the UI.
    • There was a lot of excitement around this. 
    • There was an early discussion on what terms should NOT be translated such as "DAG" or "Xcom".
    • This was to be followed up offline.
  • Golang SDK
    • Ash shared that he had raised a PR for an early incarnation of the Golang Task SDK and requested feedback. 
  • Airflow 3 feedback:
    • Pavan shared that he was at the Astronomer roadshow in London the day before and the feedback from people was very positive. 
    • Quote: "People are amazed with Airflow 3 features, dag versioning, Event driven scheduling. everyone looking to upgrade AF3, that's what i hear at least from the people i met :)"

...

Attendees

Summary

• Catch-up on action items from last call:
  • Docs site update (Jarek)
    • Jarek shared a quick, AI generated view of what an updated version selection UI could look like on the docs site. 
    • Jarek also articulated an idea of showing a banner (either left or top) to highlight which Airflow version was being viewed at the moment. 
    • Vikram brought up a point of possibly only showing the major.minor versions of docs, not showing the full list of patch versions as well, to avoid an overwhelming list of choices.
    • Vikram also brought up the issue of the Airflow Core version vs. Task SDK Version. Ash said that most of the docs would probably end up associated with the TaskSDK version, but we weren't there yet. 
    • Jens highlighted that we need to be careful to not overwhelm an initial user with the internal Airflow complexity. 
    • Ended the discussion with a desire to get feedback from UX perspectives, before proceeding forward.
• Development Updates and Presentations
  • UI plugins - Pierre
    • Pierre gave a quick update regarding the status of the UI plugins AIP. 
    • The first kind of plugins being worked on are the iframe plugins and the front-end will take care of rendering them. Templates can be used here with attributes including elements as TaskID and DagID, so that the display can show the context. 
    • The second kind of plugins will be to support React plugins. Additionally, Brent is thinking about dev tools to facilitate the creation of these plugins. 
    • All of this is aimed for 3.1 and quite confident on timing. 
    • Jens raised the question of whether these should initially be marked as "experimental", so that we have the flexibility to change the interface based on initial feedback. 
  • Deadline Alerts - Dennis
    • Dennis raised a database design question for the team encountered during the development of the AIP.
    • Specifically whether to only keep the list of deadline alerts to be checked in the database table, or to keep historical records (such as deadlines met or not met) as well. The point was regarding the performance impact of keeping more data vs. the benefits of keeping this information. 
    • Pierre raised the point that keeping more than "currently in flight" deadlines would be useful from a UI/UX perspective.
    • Dennis was wondering if an index would be sufficient to mitigate performance. Ramit suggested using a view and Vikram suggested that in the worst case, you could have a separate table.  
    • After some discussion, the action item was for Dennis to make the decision using tests / benchmarks based on expected sizes. 
  • Internationalization - Shahar
    • Shahar presented the proposed internationalization policy to address challenges with maintaining translations across 300 languages with only 70 committers. 
    • Shahar mentioned that German, Polish, Dutch, and Mandarin translations have already been merged.
    • And that, Hebrew and Arabic translations are pending review.
    • The presentation covered details such as assigning two roles for each locale: a translation owner responsible for language aspects and a code owner handling technical aspects, with translation sponsors to potentially be non-committers. 
    • It also referenced a process for release managers to notify code owners about missing translation items as part of cutting the release. 
    • The release process impact and timing prompted significant discussion with the team expressing concerns about increasing the burden on release managers. 
    • Jarek suggested starting the release process a week earlier to help accomplish this.
    • Shahar explained that the fallback policy was to use English when terms are not translated, but this raised UX concerns. 
    • The team agreed to limit the number of languages to start with to reduce overhead and implement a system for periodic reminders about missing translations. 
    • Shahar took the action item to follow up with potential refinements to reduce release friction.

...

Attendees

Summary

• Catch-up on action items from last call:
  • Docs site update (Jarek)
    • Jarek mentioned that there is an issue created for the work here.
    • The docs build process is a lot better now because of the work led by Pavan, but the actual result still needs work.
    • Jarek also spoke with Kamil about the original work done on the docs site and his availability to help. 
  • Update regarding internationalization impact on release process (Shahar)
    • Shahar said there isn't any change to the process from prior discussions. 
    • The expectation is for the Release Manager to announce that we will cut a new release and that serves as a notification for the code owner to finish translation by a particular date. 
    • The medium term intent is to publish translations outside of "Core Airflow". The initial concept here is to have translations released similar to provider releases and not to be tied to the core Airflow software release. This could be just a file release or even a data update to the database, to be figured out. 
    • There was a lot of enthusiasm from the team about how this project has happened so quickly and all the community involvement here. 
• Development Updates and Presentations (Please keep the updates to less than 10 minutes):
  • UI Plugins Update (Brent, Pierre)
    • Brent and Pierre did a quick demo of the UI plugins work and shared the progress to date. 
    • This included the common abstraction for the UI plugin regardless if the UI should be in a new tab or an iFrame rendering
    • The team was enthusiastic about the great progress here.
  • Deadline Alerts: Update on conclusion for database tradeoff (Dennis)
    • Ramit shared that the development completion target for Deadline alerts is the end of July
    • Ramit also shared that right now when a DAG is created with a deadline alert, the deadline would be created and stored in the database
    • The scheduler loop change work is upcoming and the database change approach will be figured out as part of that. 
    • Brent asked about needing to sync on the API endpoints for the same. Dennis to sync with Brent over the course of the next week.
• Discussion topics:
  • Multi-team discussion from dev list - (Jarek)
    • There was long spirited discussion around the multi-team effort. Jarek opened by asking for comments. 
    • Vikram went first, saying that he wanted to make sure that we were not trading off the simplicity of Airflow starter experience for individual data engineers with the complexity needed for large organizations wanting granular operational procedures. Vikram asserted that as an open source project, viral adoption has been the key to Airflow's success and that simplicity was important to maintain and in fact make better. 
    • Jarek immediately agreed with that assertion and said that was a key point in his desire to also simplify this AIP to multi-team, from the original, far more complex, multi-tenancy proposal. 
    • Vikram also said that he was surprised by the lack of reference to AIP-82 Event driven scheduling, since this was such a commonly used mechanism for segmenting Airflow deployments across teams from a ownership perspective, following classic distributed system principles. 
    • Jens also chimed in saying that we should perhaps articulate this better in the context of how to set up Airflow for large enterprises. 
    • There was a significant level of debate around the complexity vs. simplicity tradeoff, with Rajesh stating this is probably the most common requirement that MWAA hears from their customers. Niko also chimed in saying that we should be willing to accept some tradeoffs to simplicity. 
    • There was a further discussion on what needs in priority order needed to addressed here, with Kaxil raising the point that this raised significant operational considerations and he wanted to make sure that we were not getting in over our skis in complexity with quota management, etc. 
    • This ran out of time without an explicit decision in the meeting itself.
    • Jarek concluded by asking people to comment back in the AIP soon, so that feedback could be addressed in writing. 
  • Call for actions ...ah.. review! Example Dags! (Jens)
    • Jens raised a "call for action", asking people to help review the work on the Example DAGs and to contribute

...

Attendees

Summary

• Catch-up on action items from last call:
  • None
• Development updates and presentations: 
  • Airflow 3 adoption (Vikram)
    • Vikram shared that the download numbers show encouraging signs of 3.0 adoption, with about 12% of all downloads for the last month being for 3.0 placing it in third place, with 2.10 in the lead with 29% and 2.6 (still puzzling) in second place with 16%.  
    • Vikram also noted that 2.11 which was released after 3.0 is at 5% of downloads, which showed that more people were willing to go to 3.0 directly from earlier 2.x versions. 
    • Jens noted that it was still a very good thing that we had released 2.11, since 5% of people had upgraded to it from earlier 2.x versions, presumably because they were not yet ready to upgrade to 3.0. 
    • The team noted that for several organizations there were a couple of functional gaps, notably UI Plugins and "run as user" which blocked the upgrade to 3.0. Amogh responded that "run as user" is available as part of Airflow 3.0.3.
    • Vikram noted that the two big known gaps i.e. UI plugins and Deadline Alerts are targeted for the 3.1 release.
    • The team noted that we should make an effort to clean up any such small blockers in the 3.1 release. 
    • The ask for the community is to please highlight such gaps encountered in the dev list or the Airflow 3 dev slack channel. 
  • CLI update (Bugra)
    • Bugra walked through the Github project for this AIP as well as the current status of the work. 
    • Bugra stated that the project would be ready to be integrated into the release process in a couple of weeks.
    • Bugra also volunteered to help in the release process as needed. Jarek pointed out that some additional work may be needed around tests as part of getting ready for release. 
  • Airflow 3.1 update (Vikram)
    • Vikram shared the overall scope and timing of the Airflow 3.1 release as documented on the wiki page. 
    • The one area of significant discussion was the Human in the loop as detailed below.
• Discussion topics:
  • Human in the loop
    • Jens started the discussion here by pushing for the Human in the loop feature to be built and released as a separate provider, not within the standard provider package.
    • Kaxil vocally advocated for this to be in Core Airflow for simplicity and walked through the initial PR created by Wei to illustrate his rationale. 
    • Jarek vehemently agreed with Kaxil. 
    • Pierre also agreed with Kaxil, saying that he would prefer UI plugins to be only used by users, not by us as Airflow contributors. 
    • This then led to the discussion that we have gone too far with the Provider separation from Core Airflow. Jed brought up the point that he would like to get the Executors out of the Providers into a different package. 
    • The discussion ended with the decision to make the "core" Human in the loop functionality as part of Core Airflow. 
  • Code sharing discussion (Ash)
    • The prior discussion also led to the code sharing discussion between Task SDK and Core Airflow, following the thread brought up by Ash in the dev list. 
    • This was a great discussion with a lot of participation from everyone on the call. 
    • Ash to summarize this.

...

Attendees

Summary

• Catch-up on action items from last call:
  • None
• Development Updates
  • UI Plugins Update (Pierre and Brent)
    • Pierre shared that the React plugins implementation is nearly complete, including embedded web applications now working inside the Airflow UI. 
    • Pierre also said that key technical challenges have been resolved including react library sharing between plugins and Airflow Core.
    • Pierre raised one issue which had been open about where to keep the CLI/Python scripts to bootstrap React project. Based on team feedback, it was resolved to have those live within dev-tools. 
    • Brent gave a quick demo of the React plugins working in the Airflow UI
    • Outstanding work on this stream before the 3.1 release includes documentation and examples to demonstrate both iFrame and React app integration, as well as CSS fine-tuning for wider screen layouts. 
  • Task SDK Update (Kaxil, Ash, and Amogh)
    • Kaxil shared that multiple parallel work streams are active right now for this AIP
    • Ash is prototyping the common object sharing approaches (symlink vs. rendering). 
    • Utility code migration is happening based on that to appropriate locations (task SDK vs. shared). Serialization structure changes are also being done to support this. 
    • The Core Operator cleanup is happening, largely based on TP's work. Base Operator is now removed from Airflow models. Similar removal planned for Map operator and other core Operators. 
    • Branch Operator bug identified and is being addressed. 
  • Python 3.13 (Jarek)
    • Jarek shared that Python 3.13 is now available for all users and gave a quick demo of the same. 
    • Jarek also said that the Flask Application Builder (FAB) dependency has now been moved and a truly FAB-less Python 3.13 deployment is possible. 
    • Jarek also shared that Python 3.13 fixed a long-standing file description duplication bug. This bug could explain the recent "too many files open" issues in supervisor tasks. 
  • Deadline Alerts Update (Ramit and Rajesh)
    • Ramit shared that he had done performance testing on different approaches using a standalone Python script on Postgres, with 10 million rows. 
    • He tested 3 approaches: Two separate tables, view-based single table, and indexed single table. 
    • The index approach (state + deadline time) had the best performance and therefore led to the implementation approach of adding a state column to the deadlines tables. 
    • Ramit also shared that async callbacks would be run on the triggerer and synchronous callbacks on the workers. 
    • Rajesh shared that the deadline detection code is nearly complete, with the Slack and SMTP notifiers being implemented as async functions. Documentation is also in progress, with an estimated dev completion in about 2 weeks. 
    • There were questions around performance considerations and running under scale. 
• Discussion Topics
  • Airflow 3 adoption blockers
    • Vikram started the discussion with the commonly raised blockers that he had seen or been asked about:
    • Changes in imports: Vikram said that there seemed to be a significant lack of awareness of the upgrade utilities and the upgrade guide. Pavan chimed in saying that he had seen this too. There was some discussion about how to highlight this better on the docs site or in the UI, without a clear resolution. 
    • Missing UI elements such as UI filters: Brent chimed in saying that these have been identified as Github issues and are in the process of getting resolved. Brent also asked for community help here. 
    • Database size causing long migration times, due to long database upgrade times: Again, something covered in the upgrade guide, with the recommendation to clean up the database before upgrade. 
    • Vikram also said that he has been using the "affected_version:3.0" issue tag to track adoption issues and then with priority. 
    • The ask for the community is to "self-assign" adoption issues before the next meeting.
• Action Items
  • Pierre: Continue React plugin dev tools development
  • Brent: React package requirements optimization after vacation
  • Jarek: Investigate file description issue workarounds for pre-3.13
  • Dennis and Ramit: Finalize deadline alerts implementation
  • Kaxil, et al: Continue TaskSDK / shared code changes
  • Vikram: Clean up the Airflow 3.1 wiki structure and add Augusto Hidalgo to discuss Benchmark updates to next dev call. 
  • Community: Create, tag, and self-assign (if possible) Airflow 3 adoption issues before next dev call

...