Proposed Agenda

  • Catch-up on action items from last call
    • TBD
  • Development Updates and Presentations (Please keep the updates to less than 10 minutes):
    • UI plugins update (Brent, Pierre)
    • Deadline Alerts update: Database approach (Dennis)
    • Task SDK update (Ash, Kaxil)
    • Python 3.13 (Jarek)
    • TBD
  • Discussion topics:
    • Airflow 3 adoption blockers
    • TBD

Attendees

  • Vikram Koka

  • Ash Berlin-Taylor

  • Shahar Epstein

  • Jarek Potiuk

  • Rajesh Bishundeo

  • Shubham Raj

  • Kaxil Naik

  • Bugra Ozturk

  • Vincent Beck

  • Pierre Jeambrun

  • Ramit Kataria

  • Jens Scheffler

  • Karthikeyan Singaravelan

  • Jed Cunningham

  • Phani Kumar

  • Ephraim Anierobi

  • Isaiah Iruoha

  • Amogh Rajesh Desai

  • Pavan Kumar

  • Eugene Kostieiev

  • Niko

  • Cedrik Neumann

  • Brent Bovenzi

  • Amogh Rajesh Desai

Summary

  • Catch-up on action items from last call:
    • None
  • Development Updates
    • UI Plugins Update (Pierre and Brent)
      • Pierre shared that the React plugins implementation is nearly complete, including embedded web applications now working inside the Airflow UI. 
      • Pierre also said that key technical challenges have been resolved including react library sharing between plugins and Airflow Core.
      • Pierre raised one issue which had been open about where to keep the CLI/Python scripts to bootstrap React project. Based on team feedback, it was resolved to have those live within dev-tools. 
      • Brent gave a quick demo of the React plugins working in the Airflow UI
      • Outstanding work on this stream before the 3.1 release includes documentation and examples to demonstrate both iFrame and React app integration, as well as CSS fine-tuning for wider screen layouts. 
    • Task SDK Update (Kaxil, Ash, and Amogh)
      • Kaxil shared that multiple parallel work streams are active right now for this AIP
      • Ash is prototyping the common object sharing approaches (symlink vs. rendering). 
      • Utility code migration is happening based on that to appropriate locations (task SDK vs. shared). Serialization structure changes are also being done to support this. 
      • The Core Operator cleanup is happening, largely based on TP's work. Base Operator is now removed from Airflow models. Similar removal planned for Map operator and other core Operators. 
      • Branch Operator bug identified and is being addressed. 
    • Python 3.13 (Jarek)
      • Jarek shared that Python 3.13 is now available for all users and gave a quick demo of the same. 
      • Jarek also said that the Flask Application Builder (FAB) dependency has now been moved and a truly FAB-less Python 3.13 deployment is possible. 
      • Jarek also shared that Python 3.13 fixed a long-standing file description duplication bug. This bug could explain the recent "too many files open" issues in supervisor tasks. 
    • Deadline Alerts Update (Ramit and Rajesh)
      • Ramit shared that he had done performance testing on different approaches using a standalone Python script on Postgres, with 10 million rows. 
      • He tested 3 approaches: Two separate tables, view-based single table, and indexed single table. 
      • The index approach (state + deadline time) had the best performance and therefore led to the implementation approach of adding a state column to the deadlines tables. 
      • Ramit also shared that async callbacks would be run on the triggerer and synchronous callbacks on the workers. 
      • Rajesh shared that the deadline detection code is nearly complete, with the Slack and SMTP notifiers being implemented as async functions. Documentation is also in progress, with an estimated dev completion in about 2 weeks. 
      • There were questions around performance considerations and running under scale. 
  • Discussion Topics
    • Airflow 3 adoption blockers
      • Vikram started the discussion with the commonly raised blockers that he had seen or been asked about:
      • Changes in imports: Vikram said that there seemed to be a significant lack of awareness of the upgrade utilities and the upgrade guide. Pavan chimed in saying that he had seen this too. There was some discussion about how to highlight this better on the docs site or in the UI, without a clear resolution. 
      • Missing UI elements such as UI filters: Brent chimed in saying that these have been identified as Github issues and are in the process of getting resolved. Brent also asked for community help here. 
      • Database size causing long migration times, due to long database upgrade times: Again, something covered in the upgrade guide, with the recommendation to clean up the database before upgrade. 
      • Vikram also said that he has been using the "affected_version:3.0" issue tag to track adoption issues and then with priority. 
      • The ask for the community is to "self-assign" adoption issues before the next meeting.
  • Action Items
    • Pierre: Continue React plugin dev tools development
    • Brent: React package requirements optimization after vacation
    • Jarek: Investigate file description issue workarounds for pre-3.13
    • Dennis and Ramit: Finalize deadline alerts implementation
    • Kaxil, et al: Continue TaskSDK / shared code changes
    • Vikram: Clean up the Airflow 3.1 wiki structure and add Augusto Hidalgo to discuss Benchmark updates to next dev call. 
    • Community: Create, tag, and self-assign (if possible) Airflow 3 adoption issues before next dev call