Agenda

Attendees

  • Vikram Koka

  • Ash Berlin-Taylor

  • Jed Cunningham

  • Shahar Epstein

  • Daniel Standish

  • Jarek Potiuk

  • Ephraim Anierobi

  • Kaxil Naik

  • Jens Scheffler

  • Elad Kalif

  • TP Chung

  • Vincent Beck

  • Vishnu Chilukoori

  • Bugra Ozturk

  • Indraneel

  • Karthikeyan Singaravelan

  • Constance Martineau

  • Rahul Vats

  • Wilson Wang

  • Phani Kumar

  • Albert Okiri

  • Tamara Fingerlin

  • Pavan Kumar

  • Michal Modras

  • Ryan Hatter

  • Arunav Gupta

  • Kenten Danas

  • Shubham Mehta

  • Regan

Summary

  • Follow-up on action items from the last call:
    • AIP-83 amendment aka Data completeness discussion follow-up Daniel Standish Tzu-ping Chung
      • The docs shared earlier were:  Amendment to AIP-83 and Option 2 clarification 
      • Daniel and TP walked through the detailed "option 2 clarification" document above to address the questions under discussion
      • At a high level, there was agreement about the overall approach. There were some details to be worked through.
      • Elad suggested a deadline to close the discussion, so that this could be done in time for release. Vikram agreed and recommended that Daniel / TP start an email voting thread, so this could be closed by Thursday of next week. 
      • The team also expressed appreciation to Daniel and TP for their perseverance in working through this change, since this had generated huge debate.
  • Development Updates:
    • Update on AIP-66 DAG Bundles & Parsing Jedidiah Cunningham 
      • Jed demonstrated DAG bundles based on a Github repo, then DAG versioning and DAG fetching. 
      • He started a DAG run based on DAG version 1, updated the DAG, and then showed how the currently running DAG continued to execute subsequent tasks based on version 1 of the DAG, rather than the latest version.
      • Jed highlighted that this work had been a team effort with Ephraim and Daniel Standish.
      • DAG versionsing is a long requested feature within Airflow and it was great to see the working demo.
    • Update on AIP-81 Enhanced security of CLI by integration with API Buğra Öztürk
      • Bugra presented an update on the progress to date on the CLI
      • Bugra demonstrated the progress on making the CLI use the API for enhanced security, along with the command for connection configuration, as well as the authentication integration in flight. 
      • This AIP has been flying under the radar for a bit, so it was a treat to see the wonderful progress here.
  • Discussion topics:
    • Doc build process Jarek Potiuk 
      • Jarek brought up the concern that the Airflow doc builds were taking 30 minutes now, from around 5 minutes a couple of years ago. He, Ash, Elad, Jed, and Kaxil all highlighted that this was a ticking time bomb. 
      • As a quick analysis, there seemed to be three different areas which needed to be addressed here:
        • The doc build technology which has not been updated in years
        • Process changes around Airflow user documentation including retention, versioning, and search engine optimization
        • Doc structure changes as needed for the upcoming Airflow 3 release.
      • Jarek also brought up the point that the typical folks who have been patching this are all very busy with Airflow 3 development and will be till release. Constance volunteered to take on the third item identified above. 
      • We could not finish the discussion because of time constraints, but the takeaway was to start an async discussion thread around this topic. 
    • When do we plan to release 2.11? And are all needed features available by then? Jens Scheffler 
      • Jens brought up the topic of Airflow 2.11, saying that we had discussed cutting the Airflow 2.11 release when we started working on Airflow 3 and if that was still in the plan.
      • The Airflow 2.11 discussion was initially triggered by the forward compatibility proposal with AIP-80. That approach had been changed before AIP approval and subsequently AIP-80 itself had been deferred to post-3.0. Since then, the Airflow 2.11 release faded away from the discussion.
      • Kaxil and Jens took the action item to find the issues tagged for Airflow 2.11 and to start an async discussion around the pros/cons of that release. 
  • Action items on/before next dev call:
  • No labels