Proposed Agenda

  • Catch-up on action items from last call
    • Docs site update (Elad, Jarek)
    • Example Dags update (Jens, Jarek)
  • Development Updates and Presentations (Please keep the updates to less than 10 minutes):
    • Deadline alerts AIP-85 (Dennis Ferruzzi, Rajesh B)
    • UI plugins AIP-68 (Brent, Pierre)
    • Releases (3.0.x, 2.11) (Kaxil)
    • TBD
  • Discussion topics:
    • TBD

Attendees

  • Vikram Koka

  • Elad Kalif

  • Dennis Ferruzzi

  • Kaxil Naik

  • Ash Berlin-Taylor

  • Jarek Potiuk

  • Tamara Fingerlin

  • Amogh Rajesh Desai

  • Pavan Kumar

  • Karthikeyan Singaravelan

  • Jed Cunningham

  • Cedrik Neumann

  • Vishnu Chilukoori

  • Aritra Basu

  • Kenten Danas

  • Mathieu Montgomery

  • Pierre Jeambrun

  • Jens Scheffler

  • Ramit Kataria

  • Sean Ghaeli

  • Shahar Epstein

  • Brent Bovenzi

  • Rajesh Bishundeo

  • Michal Modras

  • Vincent Beck

  • Sebastian Ortega

  • Bugra Ozturk

  • Ephraim Anierobi

Summary

  • Catch-up on action items from last call:
    • Docs site update (Elad, Jarek)
      • Elad said that they could not make much progress on the Docs site update over the last couple of weeks, but would have an update by the next dev call. 
      • Jarek added that there has been significant improvements to the Doc publishing process, led by Pavan.
      • The team agreed with that and gave a shoutout to Pavan, Jarek, and everyone who had contributed to these improvements. 
    • Example Dags update (Jens, Jarek)
      • Jens talked about the small team who have gathered together to start working on improvements to the Example DAGs.
      • Jens shared the wiki page which was being used to track these ideas. 
      • There was a discussion around whether "testing DAGs" are useful to end users and if they should be tagged as "dual use" to reflect that they can be used as an example.
  • Development Updates:
    • Deadline alerts AIP-85 (Dennis Ferruzzi, Rajesh B)
      • Dennis and Ramit gave an update on Deadline alerts which included a long, in-depth technical discussion around the implementation options. 
      • There was significant discussion around where the loop to check for alerts should be run as well as where the dag-level callbacks should be run. 
      • The view from the team was that system monitoring i.e. if the scheduler is down is out of scope for this effort and should be done outside of Airflow, i.e. if Schedulers (or Workers) are down, no SLAs would be fired. 
      • After the extended discussion, the summary was that we aligned on a variation of option 1 from the email, with the following elements:
        • Scheduler loop would be used for checking SLA misses
        • If an async notifier is specified, the callback would be run in the triggerer
        • If a sync notifier is specified, the callback would be run on a worker
        • We recommend async callbacks using the builtin Airflow async notifiers
    • Update on the 3.0.x and 2.11 releases Kaxil Naik 
      • Kaxil mentioned that we released 3.0.1 and 2.11 already and that we would be cutting 3.0.2 in about two weeks.
  • Discussion Topics:
    • UI Internationalization
      • Brent shared that we had just merged a PR to support internationalization in the UI.
      • There was a lot of excitement around this. 
      • There was an early discussion on what terms should NOT be translated such as "DAG" or "Xcom".
      • This was to be followed up offline.
    • Golang SDK
      • Ash shared that he had raised a PR for an early incarnation of the Golang Task SDK and requested feedback. 
    • Airflow 3 feedback:
      • Pavan shared that he was at the Astronomer roadshow in London the day before and the feedback from people was very positive. 
      • Quote: "People are amazed with Airflow 3 features, dag versioning, Event driven scheduling. everyone looking to upgrade AF3, that's what i hear at least from the people i met :)"