Proposed Agenda

  • Catch-up on action items from last call
    • TBD
  • Development Updates and Presentations:
    • Swim lanes updates
      • UI / API update - Brent / Pierre
      • UI test framework update - Rahul Vats
    • Airflow 3.2 Update
      • Multi-team update (Niko)
      • Deadline Alerts update? (Dennis)
      • Asset Partitions update? (Daniel Standish / Wei Lee)
  • Discussion topics:
    • AIP-95 Provider lifecycle update - Vikram
    • AIP-94 Decouple remote commands from Airflow CLI - Bugra
    • AIP-100 Eliminate Scheduler starvation on concurrency limits - Natanel
    • TBD

Attendees

  • Shahar Epstein

  • Niko

  • Christos Bisias

  • Bugra Ozturk

  • Cedrik Neumann

  • Vincent Beck

  • Ash Berlin-Taylor

  • Rahul Vats

  • Tamara Fingerlin

  • Volker Janz

  • Jarek Potiuk

  • Natanel

  • Pratiksha Badheka

  • Jed Cunningham

  • Dennis Ferruzzi

  • Dheeraj Turaga

  • Pierre Jeambrun

  • Amogh Rajesh Desai

  • Karthikeyan Singaravelan

  • Shubham Raj

  • David Blain

  • Rajeshwar Bishundeo

  • Phani Kumar

  • Sebastian Ortega

  • Kaxil Naik

  • Brent Bovenzi

  • Jens Scheffler

  • Daniel Standish

  • Ephraim Anierobi

Summary

  • Swim lane updates:
    • UI Test framework (Rahul Vats):
      • Rahul shared that we now have 65 test cases with good input from the community. 
      • He also shared that as a result CI now takes 15 minutes for these tests and optimization is needed, with one approach to move some of these into unit tests. Rahul said he is working with Brent on this. 
      • Rahul also mentioned that this has already been valuable in catching a bug in main before the 3.1.7 release. 
      • Still working on an important issue with Grid performance with large DAGs. Kaxil highlighted a great PR from Christos which fixed a performance issue i.e. #54103
    • UI / API  (Pierre Jeambrun): 
      • Pierre shared an overall update including several feature changes that will show up in Airflow 3.2 such as permissions changes for API-level access controls, UI customization changes, and React compiler optimizations. 
      • Kartik had added a significant change in PR #60274 which resulted in noticeable improvements in API server performance, by eliminating duplicate fetching.
  • Airflow 3.2 development updates:
    • Task SDK Update (Amogh)
      • Amogh said that the Task SDK work is nearing completion excluding small core imports and that he had to make changes to the Deadline Alerts for SDK separation compliance which needs review from the AWS team. 
      • He also added that he was working on a declarative connections loading scheme via YAML spec to decouple providers.
    • Deadline Alerts (Dennis Ferruzzi):
      • Dennis shared that sync callback support PR is ready for base / local executors. Niko has reviewed this and will have changes. 
      • Dennis said that the behavior of the sync callbacks will be different from the old SLA callbacks because these will be run by the workers rather than by the DAG file processor. 
      • Kaxil said this could have concurrency implications and asked for the documentation changes detailing this behavior.
    • Multi-team (Niko / Vincent):
      • Niko said the executor updates were in flight with the AWS Batch executor work completed and others assigned to community members. Niko said that the celery executor work is complex and still in progress. 
      • The second stream of work was on the UI, with the pool selector UI updates by Vincent nearing completion. 
      • Niko said that the dag_id uniqueness work was difficult and probably would not be in the 3.2 release. 
    • Asset Partitions (Daniel Standish):
      • Daniel shared that the scheduler changes for DAG scheduling based on partitions was in flight, with several PRs having been added by Wei. 
      • The next big item was backfill support for asset partitions. 
      • Daniel also shared that the UI work was being planned with Brent.
    • AIP-98 Async support for Python Operator (David Blain):
      • David shared that this had already been merged and would be part of Airflow 3.2. 
      • There was an open item for documentation around this, specifically regarding the use cases supported. 
      • David shared that this would eventually lead to the Iterable Operator work in AIP-98.
  • Discussion topics:
    • AIP-95 Providers lifecycle governance (Vikram)
      • Vikram shared that the first PR for the updated governance process was merged and that the new process enables the sponsoring of additional providers. 
      • Initial candidates identified include Vespa and Informatica, which are both in flight.
    • AIP-94 CLI Decoupling (Bugra)
      • Bugra noted that the voting period for this AIP was currently active through the week and invited feedback. 
      • Bugra elaborated that the current plan was to have Airflow CLI commands proxy to Airflow CTL for remote operations, without any user-facing changes. 
      • Local admin commands such as DB clean would remain unchanged. 
      • Kaxil suggested an alternative i.e. single Airflow CTL set of commands for all operations.