AIP-90 Human in the loop

Status

Motivation

As we have been having conversations with Airflow users, who are using Airflow for Gen AI applications, there are a couple of features that have been brought up as being desirable. As context, based on the most recent Airflow survey, over 8% of Airflow users are now using Airflow for Gen AI use cases. 

One of the desired features is to have "human interaction" within the context of the overall orchestration flow. Here are the most common flows:

1. Choose a branch in the workflow: This is like a "human branch operator", where the human gets to pick one of the paths to move forward. This is a common pattern for business use cases such as content moderation, when the automated sentiment analysis is unsure if the content is "appropriate" and requires human judgement.  
2. Approve / Reject: This is a specialized form of the above, where the human approves or rejects the output from the prior tasks, thereby triggering a different branch in the flow. 
3. Provide input: The human is expected to provide textual input, which can then be used for subsequent steps. This is intended to be used within LLM workflows, where a human supplied context / guidance may be critical. 

For a lot of Airflow users, these operations above are expected to be performed through the Airflow UI. 

However, Airflow users especially those who use Airflow within the context of other applications may choose to build their own UX mechanisms for these human interactions. For those reasons, an API-driven interaction for the above workflows is also important. 

Though the urgency of the requirement for “human in the loop” interactions is driven by Gen AI applications, this need has existed in data pipelines earlier. A common use case within enterprise data pipelines has been for “human validations” to supplement “automated system validations”. 

Considerations

What change do you propose to make?

At a high level, we propose to make the following changes: 

• Introduce a set of “human operators” corresponding to the flows described above. These "human operators" will all be optional in a separated provider package w/o need to change core.
• When tasks are created within a DAG using these operators, they are required to specify a role, which would be associated with one or more actual “humans” within the organization. The mapping of the role to the human is expected to be defined outside of the DAG and changeable based on the evolving needs of the organization. 
  Note: If Airflow in future supports a mechanism of groups in the user management, roles can be replaces by user groups.
• The state of the individual task while waiting for human input is proposed to be “deferred”. However, this is TBD as we get closer to implementation, since there are potential tradeoffs. It is assumed that all human Operators implement async per default.
• UI support:
  • A user (with the appropriate role) can see the tasks waiting for their input. 
  • The user can complete the task as appropriate by choosing one of the available options for that task, including providing input as needed. This is conceptually similar to the Trigger Dag form, but is tailored based on the operator. 
  • In the case of a “branch” or “approve / reject” (i.e. flows 1 and 2 above), the Airflow UI will show the available options to the user. 
  • In the case of the “provide input” (i.e. flow 3 above), the Airflow UI would display a form similar to that for DAG params today. 
• When multiple people have the same role, the first person to take action on a task “completes the task”. After that completion, other individuals cannot take action on that task and it is removed from their list of available actions. 
• API support: 
  • While this task is waiting for this input, it can be queried using the API for details such as incoming Xcom data and available options. 
  • This task can be marked as completed using the API, but requiring the key result as needed including the data needed for following tasks.
  • An URL endpoint is provided that can be used to provide a prepared link that can be shared (utility to generate included in provider to automate creation of notifications)
    • Directly opens and re-directs to the dialog with details
    • Directly trigger one action as a single-click (e.g. single click on a link in an email to approve) - assuming user is logged-in before
    • Security first! This would need to be explicitly enabled in configuration and a token would be added when generating to ensure CSRF is not made by artificial URLs. URLs would need to be generated by the utility before distribution.
• Note: As the human tasks are behaving like other tasks in a Dag, a business approval might be skipped if an operator or administrator with elevated permissions marks a human task as "success" or "failed". We do not assume a specific/extra protection above other tasks.

Existing unchanged DAG / task behavior of relevance:

• While waiting for human input, the status of the containing DagRun will be a regular valid DagRun as being a "running" task. It is modeled and behaving like any other deferred (async) task that polls/waits for a file.
• The tasks can use incoming XCom data for the human to evaluate the operation and also supply outgoing XCom data as the result of human intervention. The semantics here are unchanged from regular task handling. 
  incoming / upstream XCom and DAG params can be used to render a markdown text with a template to make a overview for the w/o technical expertise user (business user).
• Results of the human interaction are posed via XCom to be used downstream in the DAG logic.

To be figured out:

• Notification options and settings for human oriented workflow systems tend to be customizable at an organizational level and at an individual level, rather than at a DAG / pipeline level. This is to be figured out for the right user experience and configurability / customizability.

Technical Details

Provider name

• Human Interaction
• apache-airflow-providers-human

Proposed Operators being Available in the provider package

• HumanOperator(BaseOperator)
  Base class for all following HumanOperators which the dedicated operators inherit from. Also custom operators could derive from there.
  Parameters
  • subject: str - Headline/subject presented to the user for the interaction task - templated field
  • body: str | None = None (longer text possible, CLOB - Markdown) - descriptive text that might give background, hints or can provide background or summary of details that are needed to decide - templated field
  • params: ParamDict | None = None - dictionary of parameter definitions that are in the format of Dag params such that a Form Field can be rendered. Entered data is validated (schema, required fields) like for a Dag run and added to XCom of the task result
  • options: list[str] - List of options that the human can select from and click to complete the task. Buttons on the UI will be presented in the order of the list
  • default: str | None = None - The default result (highlighted button) and result that is taken if timout is passed (using existing timeout from BaseOperator)
• ApprovalOperator(HumanOperator)
  Convenience operator for approval tasks. Options are pre-defined with ["Approve", "Reject"].
• HumanTerminationOperator(HumanOperator)
  Convenience operator that inherits the function from ShortCirquitOperator to terminate the Dag run by human decision. Options are pre-defined with ["Stop", "Proceed"].
• HumanBranchOperator(HumanOperator)
  Convenience operator that inherits the function from SkipMixIn to implement a branching functionality based on human selection
  Parameters:
  • multiple: bool = False - If the human can select one or multiple options
• HumanEntryOperator(HumanOperator)
Allows UI forms to be presented so that user can add further information with all options that a TriggerForm allows (same like Dag params). Options and default default to ["OK"] but can be over-ridden.

Screen Mockups

Overview about human tasks as plugin page

If one element is clicked the user is routed to the task where in an additional panel the human interaction is displayed. Here with some example markdown/template filled, a form with two fields to pass numbers and 3 buttons as options

Which users are affected by the change?

Current users of Airflow, using Airflow for existing use cases, will be unaffected by this change. Users wanting this new feature would need to install the "human" provider package.

How are users affected by the change? (e.g. DB upgrade required?)

Current users who don't use this new feature are unaffected by this change. 

As of storing the state and details efficiently one additional DB table is needed to store human task status. This DB table would need to be created by the provider. It assumed to be low volume and impact to the complexity of the database.

Other considerations?

The only other AIP which is currently in progress which could be leveraged / impacted (though not necessarily) is AIP-68 Extended Plugin Interface. The plugin interface is needed to add the new panels and screens seamlessly into the UI.

Human failure to respond:

• Clearly, humans are fallible and may not always be responsive. So a timeout might help to automatically passively approve.
• Repeat notifications, escalations, timeouts for tasks waiting for human input is an important topic to consider. But this does not need specific providers but can also be modeled with existing DAG logic, e.g. existing branch operators. This is also true for cases where there is a demand for parallel or serial approvals of multiple stakeholders. No extra complexity needed. We assume each human interaction is one screen activity for one task.
• These processes exist today within organizations for such situations, but similarly to notifications, there is a high level of configuration and customization here. Note that complexity in notifications can also be modelled in the Dag itself and does not need extensive support in the initial versions. Might be re-considered in the future.

What defines this AIP as "done"?

• A new provider package is released that can be installed in Airflow 3.1++
• Being able to create Dags with the three flows described in the beginning of the document and run them through the Airflow UI and using the API (i.e. without the UI).
• Examples are included as well as the normal level of quality for documentation