Proposed Agenda

  • Release updates - 3.3.1 - Rahul Vats 
    • Timelines
  • Catch-up on action items from last call
    • Update on discussion regarding Event state listener (Christos Bisias)
    • Update on Data Quality provider discussion (Pavan, Vikram)
    • TBD
  • Other Development Updates
    • AIP-85: DAG importer - feedback & discussion Dilnaz Amanzholova (carried forward from last seesion)
    • AIP-115 On-Demand Task Sections - Demo & Feedback
    • Road to Common AI 1.0
    • TBD
  • Discussion topics:
    • Airflow Summit
    • Pull Request handling - all options are on the table (Jarek)

Attendees

  • Elad Kalif, Christos Bisias, Kaxil Naik, Vikram Koka, Shahar Epstein, Dilnaz Amanzholova, Vincent Beck, Tamara Fingerlin, Ian Buss, Amogh Rajesh Desai, Rahul Vats, Sean Ghaeli, Bugra Ozturk, Eugene Kostieiev, Jarek Potiuk, Ephraim Anierobi, Phani Kumar, Shubham Raj, Pierre Jeambrun, Michal Modras, Ramit Kataria, Jed Cunningham, Karthikeyan Singaravelan, Kenten Danas, Jake McGrath, Dheeraj Turaga, Rafal Biegacz, Natanel, Tamir

Summary

  • Catch-up on Action Items
    • Release Updates: 3.3.1 (Rahul Vats)
      • Rahul noted RC1 is targeted for 3rd or 4th August, with release around 10th August. This leaves a 5-6 day testing window that accommodates an RC2 if one is needed.
      • Rahul noted this gives roughly 15-20 days of buffer before Airflow Summit.
      • Rahul reported the current milestone shows 48 open items, but clarified this is inflated: 40-50% of the count is backport PRs, some already merged into main.
      • Rahul is reviewing PRs he's comfortable with directly and requesting code owner reviews for the rest, and to stay cautious per the existing release process.
      • Jarek noted he will kick off the provider release vote on Friday 1st August, expected to close Monday 4th August.
        • Jarek is traveling to Beijing for Community Over Code Asia around the 4th August release date and flagged connectivity concerns, so he plans to move the provider release ahead of that trip rather than leave it pending.
        • Jarek noted Shahar's pending ad hoc hook provider release will be folded into this same cycle instead of shipped separately.
      • Jarek asked that provider PRs be reviewed ahead of 3.3.1 PRs, especially any with dependencies on 3.3.1 fixes.
    • Pandas 3 Compatibility (Rahul Vats)
      • Rahul raised this during his 3.3.1 update, flagging an open question on one of Jarek's PRs and asking whether it might affect client-side deserialization.
      • Elad asked for context for the group, and Jarek explained the background: Pandas 3 has been installable alongside Airflow since January 2026, and XCom data serialized under Pandas 2 fails to deserialize under Pandas 3 because Pandas 3 removed an import Airflow's built-in Pandas XCom serializer depends on. Jarek's PR would auto-convert Pandas 2 serialized DataFrames to Pandas 3 on deserialization. Jarek flagged this is a one-way migration once XComs are converted.
      • Elad said this sounds risky, and Vikram agreed, calling it a very significant limitation.
      • Jarek argued the risk cuts both ways: if the team does nothing, users will eventually upgrade to Pandas 3 on their own and hit the same failures without warning, so shipping the conversion may be the less risky option. Amogh clarified the deserialization Jarek was concerned about on the server side is likely just stringification for UI display purposes, not true serialization, though Jarek noted this needs to be checked closely since Pandas handling is deeply embedded in Airflow Core.
      • Elad floated dedicating a full feature release, ignoring the usual numbering, solely to this change. Vikram was hesitant, not opposed, given how few users upgrade version by version, and wanted to see other options first.
      • Elad asked whether users should be warned regardless of the final decision. Vikram agreed this was reasonable.
      • Decision for 3.3.1: defer the auto-conversion fix, and instead add a pandas < 3 constraint plus a migration note warning users not to upgrade to Pandas 3 on unconstrained installs.
      • Full Pandas 3 compatibility with auto-conversion is deferred to 3.4, which Jarek noted will likely set pandas >= 3 given it lands roughly 9 months after the Pandas 3 release.
      • Elad asked that a safeguard be added to prevent constraints from ever rolling back from Pandas 3 to Pandas 2.
      • Amogh asked whether there's enough incentive for users to actually upgrade to Pandas 3, so the minor release isn't wasted effort.
      • Elad asked whether the change has implications for custom XCom backends. Jarek confirmed it's specific to Pandas and doesn't affect custom backends.
    • Road to Common AI 1.0 (Vikram Koka, Kaxil Naik)
      • This came up as Rahul and Jarek wrapped their discussion of the provider release cycle, when Vikram mentioned he had held up the Common AI provider 0.7 release pending a PR that has since been reverted, and the provider is now ready to release.
      • Kaxil explained the reverted PR had proposed exposing all Google and AWS APIs to AI agents via hard-coded API lists. This was rejected because agents work better against CLIs or MCPs than raw APIs.
      • Both AWS and Google now ship native MCP servers, and Kaxil noted Airflow's Common AI provider already supports both MCP toolsets and agent skill toolsets.
      • Agent skill toolsets support local files as well as remote, GitHub-hosted skills via the Git provider.
      • Kaxil noted the AWS MCP server allows scoped access, for example restricting deletion operations, which provides a governance layer.
      • Jarek asked whether marketplace installs are supported. Kaxil clarified there's no native connection to an HTTP marketplace endpoint, but installing a skill from a GitHub URL sourced from a marketplace works today.
      • Full discussion of the Common AI 1.0 roadmap was deferred to the next call.
    • Kafka Event Producer, formerly Event State Listener (Christos Bisias)
      • Vikram introduced this as a follow-up from last call's catch-up on the event state listener, noting it took a while to get Christos's feedback on the original PR.
      • Christos reported on the two to-dos from last call: renaming the plugin and updating the documentation with clear use cases.
      • Christos renamed the feature from "Kafka listener" to "Kafka event producer" for clearer framing, since it's a listener to Airflow but a producer to Kafka, and the old name was causing confusion. Shahar and Vincent reviewed quickly, and the PR merged a couple of days ago, in time for the provider release.
      • Christos updated the documentation with common use cases: monitoring the state of multiple DAGs and task instances externally without querying the database, coordinating multiple DAGs together, and having an external pipeline listen for a DAG to finish, or trigger one, in either direction.
      • Elad asked how this compares to the asset watcher and whether a user considering both features would know which to choose, and whether there's a single place in the docs that explains both together. Christos confirmed no unified doc exists yet. Elad said this is something the team should build, since DAG authors look for one unified solution to a problem even if implementation details live in separate places, and offered to work with Christos on it offline.
      • Jarek clarified the core distinction: an asset watcher subscribes at the DAG level, while the Kafka event producer can operate at the level of an individual task within a DAG. Elad suggested this distinction, in a couple of sentences, is enough to add to the core docs.
      • Vikram raised a persona gap: DAG authors typically don't have access to Airflow plugin configuration, so he asked what coordination is needed between personas in that situation.
      • Christos asked whether the explanation belongs in a section under the Kafka event producer docs or somewhere more central alongside other related features, such as the consumer hook, asset watcher, or a sensor. Elad reframed this as two separate levels: documentation clarity, and a deeper concern that requiring a DAG author to touch the plugin mechanism weakens the feature itself, since many DAG authors don't have plugin access and have to coordinate with a devops team. Elad's position is that DAG-author-facing features should avoid the plugin mechanism where possible, citing timetables as a cautionary example of a plugin-gated feature with low adoption.
      • Christos explained the feature is enabled via configuration and applies filters so DAG authors aren't overwhelmed with messages, and that this uses Airflow's listener interface, which is integrated into the scheduler and workers and adds overhead, so deliberate opt-in is intentional rather than an oversight.
      • Vikram said Elad's feedback matches what he sees across most organizations, including likely MWAA and GCC, and that articulating the persona difference matters since adoption is difficult if a DAG author needs access to Airflow configuration. Elad reframed the question again: given the feature's current requirement, is there a way to achieve the same functionality without using the plugin mechanism.
      • Vikram asked Elad and Christos to take the remaining discussion offline, noting he wanted to preserve time on the agenda for Dilnaz's AIP-85 presentation. Christos agreed to follow up with Elad separately.
    • Data Quality Provider (Vikram Koka)
      • Vikram quickly noted he and Pavan have met twice since the last call and are still working through the design. The core open question mirrors the Kafka persona discussion: what data quality needs differ between the DAG author, i.e. the pipeline owner, and the business user who owns the underlying data. Vikram noted this includes deciding which data quality failures should block a pipeline entirely versus flag specific content as bad without failing the run.
      • Still open: whether this warrants a new, separate provider or should be incorporated into existing providers. Vikram and Pavan will return with more detail on a future call.
  • Other Development Updates
    • AIP-85: DAG Importer (Dilnaz Amanzholova)
      • Dilnaz presented an abstract, pluggable interface for supporting non-Python DAG formats such as YAML and Java, built around four key methods: import_file, can_handle, list_dag_files, and get_source_code.
      • This AIP was split off from a larger original proposal because of scope, and AIP-92 is a prerequisite. Tatiana's earlier review comments have been addressed and the AIP is considered in good shape.
      • Pierre raised a question on the get_source_code method, noting TP's discussion thread on AIP-108 argued against displaying source for non-Python task SDK languages, citing complications with external libraries and compiled bytecode with no accessible source file.
      • Dilnaz clarified AIP-85 is an internal developer interface only, with no end-user-visible change until a concrete importer, such as YAML or Java, is implemented on top of it. AIP-108 has been narrowed to tasks only and doesn't cover Java DAG source display, so a further AIP is likely needed for Java DAGs to work end-to-end.
      • Rafal, Kaxil, and Dilnaz aligned on YAML-based DAGs as the first concrete end-user deliverable. Kaxil and Pavan offered to help on the YAML implementation.
      • Vikram asked that the team define the full scope of work needed for YAML DAGs to function end-to-end, including any known unknowns, so the effort clearly lands on end-user benefit rather than staying purely an internal interface change.
        • Dilnaz is reading AIP-108 and syncing with TP as a follow-up.
      • Additional points raised:
        • Import errors should surface from the YAML file itself, not as Python tracebacks.
        • The DAG bundle-to-importer mapping is one-to-many, with the default bundle using the default importer.
        • Kaxil noted static YAML DAGs shouldn't need reparsing unless the underlying file changes, and linked this to AIP-95, the async DAG processor, as a likely prerequisite. This can be a follow-up rather than part of AIP-85 itself.
    • AIP-115: On-Demand Task Sections was not reached due to time constraints.
  • Discussion Topics
    • Airflow Summit was not reached due to time constraints.
    • Pull Request handling (Jarek) was not reached due to time constraints.