AIP-102 Improving Business User Interaction with Apache Airflow

Status

Motivation

Airflow is driven by data engineers and technology, thus tasks and results are viewed on a technical level. If something goes wrong users need to check logs in the UI. Since Airflow 3.1.0 added Human-in-the-Loop... but there is no “business user friendly” view existing. We propose improvements to add this!

The proposal is mainly about UX for users which are non-technical experts or no data engineers.

  In Airflow 2.x we compensated the usability gaps via Task Instance Notes and Dag Run Notes which were displayed on the details panel. The details panel was always displayed first when clicking on a task or Dag run. We auto-generated these notes and populated them such that if a user clicked on a Task Instance a high level summary or error summary was always presented. In Airflow 3 this is not not directly possible because (1) the adding/machine generation of notes requires DB access which is undesired in the new architecture as well as (2) the notes are not displayed because per default the logs panel is displayed and the notes are hidden in a dialog where the user needs to click on a button to see them. Alongside with this (3) also logs are not scrolling to the end per default but show the start of the text per default such that the user needs to scroll down to the bottom (with incremental loading might need some clicks) in order to see details or the error. So compared to our user friendly solution (limited to 1000 chars though...) the upgrade to Airflow 3 added a usability gap.

Example of usage of the notes in Airflow 2.x

(1) Dag Summary

(2) Failure report on Task Instance

(3) KPI Summary of an execution as summary

  Note that initial idea was that such capability could be provided with an extended plugin interface as initially proposed (still see mock-ups there) via AIP-68 Extended Plugin Interface for React Views. This was super-seeded with the re-implementation of the Airflow UI in 3.0 in React, plugins are possible but no generic similar function.

Considerations

We propose to add an optional feature allowing to expose a good human readable Dag Run and Task Instance Summary via Markdown Templates. We envision this is a Operator or Task specific extension as opt-in, can be used and added but does not need to.

What change do you propose to make?

Adding capabilities to create Dag Run and Task Instance Summaries as Markdown. Similar like Dag Documentation and Task Documentation can be (statically) generated and displayed as Markdown, provide options to generate Markdown in Task Instances which are stored in the DB in order to be displayed on UI.

Compared to Notes today in Airflow 3 we propose to:

• Add Dag Run / Task Instance "Summaries" as an additional concept to "Notes"
  • "Notes" are for user interaction, users can CRUD notes on UI and this is rather r/w for collaboration. Notes are stored in the meta DB and are limited to 1000 chars and are displayed in a model/pop-up. "Summary" feature does not replace but extends "Notes" as concept.
  • "Summaries" are machine generated texts to produce human readable summaries. They are r/o for users. They should not be limited to 1000 chars (but are intended to be a reasonable size, e.g. <1MB)
  • Compared to Dag or Task Documentation the "Summary" is per instance and not the same for the instance (Documentation is static but the "Summary" is per execution).
• If a Task Instance Summary is present (was generated) for a Task Instance then this panel is displayed first for the user (before the Logs view). If it is not existing then like today Logs are displayed first
• The "Summary" is generated with Markdown formatting and is presented to user in a formatted view
• The "Summary" can be produced while the task is running and while the task is in execution can report a progress or intermediate result. It is refreshed similar like logs and details are reloaded while task is running
• There is always only one or none "Summary", it it is generated it is overwriting the old one. There is no history stored for the "Summary"

Example Mock-up

Technical Interface / Specification Proposal

(1) Utility Methods to publish a new Summary

in class airflow.sdk.execution_time.task_runner.RuntimeTaskInstance (alongside with RuntimeTaskInstanceProtocol) the following utility methods are added:

• def push_task_instance_summary(self, content: str | None) -> None: ...
"""Sets and updates the current task instance summary."""
• def push_dag_run_summary(self, content: str | None) -> None: ...
"""Sets and updates the current Dag run summary.""

Once invoked in the running task execution the content is transferred via supervisor to execution API and persisted in the meta database. If None is passed any existing previous Summary is cleared.

The methods can be called during execution of a task on the worker. Similar entry to be added in async cases such that the same can be used on Triggerer as well.

(2) Additional properties to define a Markdown template file to be used

The following properties are added to the model:

• On airflow.sdk.definitions.dag.DAG
  • dag_run_summary_template: str | None = None
"""Optional path to a Markdown template which is processed via Jinja to be published as Dag run summary after Dag completion."""
• On airflow.sdk.bases..operator.BaseOperator
  • task_instance_summary_template: str | None = None
"""Optional path to a Markdown template which is processed via Jinja to be published as Task Instance summary after Dag completion."""

The template files are automatically loaded on the task runner upon task completion (if provided) and using task context and XCom values passed through a Jinja templating and published. For any Task Instance on the task runner after XCom return is processed and before the runner is completed. On the completion of a Dag run similar to a on_success/on_failure hook.

With this Operators can ship and provide a default Summary template, e.g. KubernetesPodOperator can render Pod summary per default.

(3) Public API to retrieve Notes - the public API is also used from React UI to query and render the markdown.

What problem does it solve?

We have many technical as well as non-technical users. Today it is hard for users to see and understand errors. We attempted to add an auto-triage to processes and tools but still users might be confused by long technical logs and it is hard to find the root cause of a problem in a long log. Some people seek for the first error message, some for the last summary. But nothing that a user can be really happy about.

With the optional Dag Run and Task Instance Summary we allow optionally per task to create a status progress as well as a human readable summary, as markdown also with links to other systems with more details and a structured, readable layout. Using Markdown we are also safe that no scripts can be injected. As an extension would also allow embedding images and folded "Details" areas to drill-into.

Why is it needed?

We want to open Airflow to more business users and make it easier to interpret technical results. So far as a summary the Notes we implemented in Airflow 2.x were a compromise to at least a short <1000 char summary with important external links. With the proposed Dag Run / Task Instance Summaries we can extend and allow customizations

Are there any downsides to this change?

Additional complexity and mental load is added to Airflow. But as it is an opt-in feature, nobody is required to use it.

Additional tables and (small) increase of meta database volume based on the summary content stored in the database.

Which users are affected by the change?

Positive impact on non technical users of Airflow.

Additional complexity for Dag authors if the feature is implemented. But as Markdown is a standard, it is a reasonable complexity.

How are users affected by the change? (e.g. DB upgrade required?)

Database extensions would be covered by standard Alembic migration tooling. Similar like todays Notes the Summaries would be stored in an additional relation, so no additional fields needed in `dag_run` and `task_instance` tables. Selection and query only as needed via API, minimizing side effects in performance for other components.

Proposed ORM Model:

Other considerations?

Alternative options we considered before writing this AIP:

• Further using "Notes" like we did in Airflow 2
  • Benefits: Can be used already today
  • Trade-offs (and reasons why we did not propose this further):
    • Limited to 1000 chars (this was already a challenge in the past as embedded links can easily get long)
    • Users need to know that the "Notes" button need to be clicked to see the content as well as when flipping through tasks another click is needed
    • Links in Airflow 3 are displayed in Markdown as link but are not "clickable" because modal goes to edit mode when clicking
    • Earlier proposal to always display notes was reverted post-merge (PR https://github.com/apache/airflow/pull/51764)
      
      
    • DB-free API is missing in order to generate Notes from task ínstances via supervisor, PR was discussed in https://github.com/apache/airflow/pull/61066
• Adding similar Feature as a Plugin
  • Benefits: Could be made today with React Plugin
  • Trade-offs (and reasons why we did not propose this further):
    • If a panel as plugin is added the display would not be conditional. It would always be displayed
    • No API support in supervisor existing, routes to populate content would need to be added with a Task SDK extension as custom deployment and API endpoints are separately needed to publish. Also DB storage would be needed as a custom integration
    • The solution would be a proprietary solution in Bosch but we assume that more people would benefit from it and existing Operators could provide a nice view per default
    • It is not possible (today) to influence that the plugin would be loaded before Logs, so another click for the user is needed to see Summary panel.
• Adding Markdown support for log viewer: Make logs better readable by allowing to embed Markdown into the text stream as alternate to just JSON structured log text - Similar like Markdown in a Python Notebook
  • Benefits: Would allow within logs to have better formatted results to display, e.g. tables
  • Trade-offs (and reasons why we did not propose this further):
    • Still users would need to search in the logs for a summary at the very bottom which takes time to scroll to
    • Still not comfortable for business users to find relevant information/summary in log stream

What defines this AIP as "done"?

• Task SDK Extensions are available
• UI is extended
• DB extensions and DB migrations are merged
• Documentation is extended
• At least one example is added to the example dags
• Stretch: At least one of the standard operators in any Provider implements a Summary out-of-the-box