Proposed Agenda

  • Release updates - 3.3.1 - Rahul Vats 
    • Timelines
  • Common AI provider (Kaxil / Vikram) 
  • Data Quality provider and its demo @pavan
  • AIP-85: DAG importer - feedback & discussion Dilnaz Amanzholova (carried forward from last seesion)
    • (won't be able to attend the call) I would appreciate it if you could take a look at the updated version of AIP-85. We have improved a lot of the components based on the comments and discussions.
  • PR Demo - Event state listener under the Kafka provider - https://github.com/apache/airflow/pull/68082 - Christos Bisias (carried forward from last seesion)

Attendees

  • Iliya Romm, Christos Bisias, Ash Berlin-Taylor, Eugene Kostieiev, Jarek Potiuk, Rahul Vats, Sebastián Ortega, Vikram Koka, Karthikeyan Singaravelan, Bugra Ozturk, Pierre Jeambrun, Jake McGrath, Pavan Kumar, Vincent Beck, Elad Kalif, Brent Bovenzi, Mustafa Avcu, Amogh Rajesh Desai, Shahar Epstein, Ephraim Anierobi, Phani Kumar, Shubham Raj, Ramit Kataria, Antonio Bergonzi, Zach Gottesman, Sean Ghaeli

Summary

  • Note: Dilnaz Amanzholova was unable to attend. The AIP-85 agenda item was carried forward again; Dilnaz noted that the AIP has been significantly updated based on prior feedback and welcomes review and comments.


  • Catch-up on action items from last call:
    • None noted.


  • Airflow 3.3 Development Updates:
    • 3.3.1 Release update (Rahul Vats)
      • Rahul said that RC1 is targeted for August 3rd, with GA wrapping up around August 10th. This gives about 10-15 days for the release before the Airflow Summit.
      • Currently 61 open items tagged for this milestone, which need to be worked on through before the release.
      • Elad noted that August will also be slow for mailing list discussions, code review, and everything else given Summit prep, and asked contributors to be less demanding during that period.
      • The same applies to providers: critical provider changes should be submitted before August.
      • Elad flagged that he is not currently on the release manager roster but is available for ad hoc releases or to step in if needed.
      • Rahul noted it would be good to align a provider release just before 3.3.1 so that all constraints are up to date. Elad confirmed he can do an ad hoc provider release if needed.
    • Common AI Provider (Vikram Koka / Pavan Kumar)
      • Vikram shared that Kaxil has been driving most of the work on this and that Pavan is also heavily involved.
      • Vikram said that the provider is currently at v0.6.0, with a target of reaching 1.0 before the Summit.
      • The delay to 1.0 is intentional: the team is focused on stabilizing interfaces across OpenAI, Anthropic, and the common provider before locking the API.
        • There is significant overlap in users combining the OpenAI SDK, Anthropic agent SDK, and the common provider, making interface consistency critical.
        • AIP-105 resumable operators are also consuming this work, with a PR from Amogh recently merged using the provider.
      • Vikram also confirmed that this provider was being used in production already with no major complaints so far, but additional interaction patterns are still coming in and being incorporated.
      • Pavan added that he was looking at adding Strands SDK support as well.
      • Elad shared in chat that the community is already contributing additional capabilities to the provider, referencing PR #68847.
      • Vikram requested the community for feedback on usage patterns and anything they feel should be addressed before 1.0.
    • Data Quality Provider demo (Pavan Kumar)
      • Pavan demonstrated a new data quality provider built around a unified rule interface.
      • Core capabilities demonstrated:
        • Define rule sets as objects and attach them to asset definitions via a helper (asset_quality).
        • Execute quality checks at the task level when the asset is triggered; results show pass/fail per rule.
        • Quality scores are carried forward so that consumer DAGs can gate on a minimum score before proceeding, using a require_quality helper.
        • A plugin showed the full execution history per task. The backend used was object store with no DB dependency.
        • An LLM skill was included in the examples. The rule schema was generated by an LLM based on data structure.
      • Brent confirmed in chat that asset-level plugins will be added soon, which would allow quality scores to be surfaced in the asset view as well.
      • Discussion on scope:
        • Vikram raised a distinction between two categories of quality checks that have different personas and different recommended responses:
          • Schema and structural validation: the entire data asset is invalid (e.g. bad vendor data), the pipeline should fail immediately, and the data engineer is the relevant persona. This is similar to the existing LLM schema check operator in the Common AI provider.
          • Business rule validation: individual items fail specific rules (e.g. a contract value is too high, a date is in the past), results should be stored to object store for business owner review, and this is genuinely new capability not covered by existing tooling.
        • Vikram noted the current provider appeared to cover both and suggested keeping schema checks in the existing operators and scoping this provider to business and content validation.
        • Pavan noted the current implementation is SQL-based and that schema comparison is also possible within the same framework, but agreed the distinction is worth discussing further.
        • Action item: Vikram and Pavan to discuss scope offline, specifically whether schema/structural checks belong here or in the existing LLM schema check operator.


  • Discussion Topics:
    • Kafka provider: Event state listener demo (Christos Bisias)
      • Christos noted that the PR (#68082) had already been merged before this call, but offered to demo it given it had been deferred twice.
      • What the PR adds:
        • A listener under the Kafka provider that publishes a Kafka message on every DAG and task instance state change.
        • Configurable: can be scoped to specific DAGs or tasks via an allow list; off by default.
        • Extension to the Kafka connection config to accept Python package callbacks resolved by string reference.
      • Demo: Two DAGs with cross-DAG dependencies implemented via Kafka messages, using the existing WaitMessageSensor. Messages showed fields including task instance state, DAG ID, task ID, and run ID.
      • Discussion: Vikram asked how does this differed from asset watchers and if Vincent had reviewed this PR
        • Christos clarified that asset watchers sit on the consumer side; this feature sits on the producer side, emitting messages on every state change event within the Airflow cluster.
        • Vincent confirmed he had not followed the demo fully and would review the PR to better understand.
        • Ash clarified the key distinction from his understanding of the demo:
          • For DAG authors, asset watchers would continue to be the preferred mechanism. 
          • This feature is more suited to system monitoring, for example, an external observability tool already listening on Kafka that wants to react to Airflow state changes, or monitoring multiple Airflow instances across deployments.
        • Vikram agreed, framing it as analogous to system monitoring via OTEL: targeted at deployment admins and external systems rather than DAG authors.
        • Elad raised the core concern: if the 20 people on this call are confused about when to use this versus asset watchers, users will be far more confused. The documentation must make it immediately clear, and clarified that it should be in two sentences: what problem this solves and how it differs from asset watchers.
      • Terminology concerns:
        • The name "Kafka listener" is confusing because it implies the feature listens to Kafka, when it actually produces to Kafka.
        • "Listener" is an internal Airflow term (same pattern as OpenLineage) and is not meaningful to most users.
        • The config section is also named "Kafka listener," which compounds the confusion.
        • Jake agreed in chat that the "listener" term should not appear in user-facing docs.
        • Ash recommended avoiding the word "listener" entirely in provider-facing documentation.
      • Action item: Christos to open an issue or PR to fix the terminology and documentation before the next provider release, to avoid introducing backward compatibility issues.
        • Ash suggested creating a PR or issue and flagging it to the dev list rather than starting a full mailing list thread.
      • Backward compatibility and 2.11:
        • Shahar raised a broader question about the current stance on Airflow 2.11 backward compatibility for providers. Elad clarified that 2.11 support has already exceeded the originally promised deadline and is now best-effort.
        • Shahar noted that Claude had flagged a potential 2.11 issue in the Kafka provider. A hook that produces the skipped event does not exist in 2.11. Elad confirmed that if this causes a problem, bumping the Kafka provider minimum version is the right approach.
      • Vikram closed by thanking Christos for the demo and apologizing that the feedback came after the PR was already merged.
      • Vincent committed in chat to reviewing the PR the same day or next.




  • No labels