Proposed Agenda

  • Catch-up on action items from last call
    • TBD
  • Development Updates and Presentations (Please keep the updates to less than 10 minutes):
    • CLI Update (Bugra)
    • Quick update on 3.0 adoption (Vikram)
    • Airflow 3.1 overall update (Vikram)
  • Discussion topics:
    • TBD

Attendees

  • Vikram Koka

  • Jarek Potiuk

  • Tamara Fingerlin

  • Vincent Beck

  • Sebastian Ortega

  • Bugra Ozturk

  • Jens Scheffler

  • Jed Cunningham

  • Amogh Rajesh Desai

  • Ash Berlin-Taylor

  • Iliya Romm

  • Phani Kumar

  • Niko

  • Pierre Jeambrun

  • Rahul Vats

  • Zach Gottesman

  • Daniel Standish

  • Kaxil Naik

  • Sean Ghaeli

  • Pavan Kumar

  • Rajesh Bishundeo

Summary

  • Catch-up on action items from last call:
    • None
  • Development updates and presentations: 
    • Airflow 3 adoption (Vikram)
      • Vikram shared that the download numbers show encouraging signs of 3.0 adoption, with about 12% of all downloads for the last month being for 3.0 placing it in third place, with 2.10 in the lead with 29% and 2.6 (still puzzling) in second place with 16%.  
      • Vikram also noted that 2.11 which was released after 3.0 is at 5% of downloads, which showed that more people were willing to go to 3.0 directly from earlier 2.x versions. 
      • Jens noted that it was still a very good thing that we had released 2.11, since 5% of people had upgraded to it from earlier 2.x versions, presumably because they were not yet ready to upgrade to 3.0. 
      • The team noted that for several organizations there were a couple of functional gaps, notably UI Plugins and "run as user" which blocked the upgrade to 3.0. Amogh responded that "run as user" is available as part of Airflow 3.0.3.
      • Vikram noted that the two big known gaps i.e. UI plugins and Deadline Alerts are targeted for the 3.1 release.
      • The team noted that we should make an effort to clean up any such small blockers in the 3.1 release. 
      • The ask for the community is to please highlight such gaps encountered in the dev list or the Airflow 3 dev slack channel. 
    • CLI update (Bugra)
      • Bugra walked through the Github project for this AIP as well as the current status of the work. 
      • Bugra stated that the project would be ready to be integrated into the release process in a couple of weeks.
      • Bugra also volunteered to help in the release process as needed. Jarek pointed out that some additional work may be needed around tests as part of getting ready for release. 
    • Airflow 3.1 update (Vikram)
      • Vikram shared the overall scope and timing of the Airflow 3.1 release as documented on the wiki page. 
      • The one area of significant discussion was the Human in the loop as detailed below.
  • Discussion topics:
    • Human in the loop
      • Jens started the discussion here by pushing for the Human in the loop feature to be built and released as a separate provider, not within the standard provider package.
      • Kaxil vocally advocated for this to be in Core Airflow for simplicity and walked through the initial PR created by Wei to illustrate his rationale. 
      • Jarek vehemently agreed with Kaxil. 
      • Pierre also agreed with Kaxil, saying that he would prefer UI plugins to be only used by users, not by us as Airflow contributors. 
      • This then led to the discussion that we have gone too far with the Provider separation from Core Airflow. Jed brought up the point that he would like to get the Executors out of the Providers into a different package. 
      • The discussion ended with the decision to make the "core" Human in the loop functionality as part of Core Airflow. 
    • Code sharing discussion (Ash)
      • The prior discussion also led to the code sharing discussion between Task SDK and Core Airflow, following the thread brought up by Ash in the dev list. 
      • This was a great discussion with a lot of participation from everyone on the call. 
      • Ash to summarize this.