AIP-50 Trigger DAG UI Extension with Flexible User Form Concept

Status

Motivation

As user of Airflow for our custom workflows we often use DagRun.conf attributes to control content and flow. Current UI allows (only) to launch via REST API with given parameters or using a JSON structure in the UI to trigger with parameters. This is technically feasible but not user friendly. A user needs to model, check and understand the JSON and enter parameters manually without the option to validate before trigger.

Similar like Jenkins or Github/Azure pipelines we desire an UI option to trigger with a UI and specifying parameters. We'd like to have a similar capability in Airflow.

Considerations

This AIP was created after working a longer time with an UI plugin which we implemented customized allowing users to "easily" trigger DAGs with parameters. More and more we realize that every use case and demo we make requires an simple entry form to show "how easy" it is (not to scare users and have them running away). We propose to "productize" the feature from a plugin and merge it into the Airflow Core UI - hoping that more people can use this feature, having a lower entry barrier for "new" users and also to close gaps from standard UI path to trigger a DAG today compared to how the UI plugin is integrated.

We also had (in the past) a separate UI (outside of Airflow) just to collect parameters and trigger the Airflow DAG in the backend via REST API - but this was a full separation of the user from Airflow and at the end requires a very redundant UI being developed and maintained.

What change do you propose to make?

Implementation Part 1) Specifying Trigger UI on DAG Level - DONE

We propose to leverage the params attribute of the DAG class with some additional optional attributes so that an UI (one form per DAG) can be specified in the DAG itself (optional). As the params attribute is built upon the Params class to define meta-data to serve the purpose of JSON schema validations, this definition already serves 80% of details needed to render a form. See also https://airflow.apache.org/docs/apache-airflow/stable/concepts/params.html.

The following details to model a form can directly be leveraged from existing Params class definition:

• The order of fields of the form
• Name of the field, using the dict key. As this might not be very descriptive the existing JSON schema field title can be used if available for a better "Human readability".
• Type of the field derived from the JSON schema type and formatting clause
  • Type boolean can render a checkbox or on/off control
  • Elements of JSON format date and date-time can render a date picker
  • Type of arrays can be made as lists of text string and object types as a JSON sub-element entry
• Elements defined with an enum can be used to define drop-down selects for a pre-defined list op options
• If JSON schema allows to have types of null can control if a field is optional or required.
• The JSON schema attribute description can be used to render field help text
• Some JSON schema attributes for text like maxLength can be used to restrict user input on the form immediately
• Further attributes can be flexibly defined on Params class if we want to add form features in the future. Nevertheless available options and features can not be validated inside your editor as it is just extending kwargs. Therefore a later switch from FAB to a modern UI can be done w/o changing the structure f Form definition, just add a different UI logic.
• If just an arbitrary dict is used as default params then the existing dict value data types can be used to "guess" a suitable form even if the Params class for JSON schema is not used.
• We can use more attributes on the Params class also to allow rendering of custom HTML field

Some additional (optional) attributes are proposed to be added to control a form layout:

• An additional attribute hidden can be used to pass static values without populating to forms. Maybe not all values shall be presented to user. ( Update Note: Was implemented finally via the const attribute)
• An additional attribute advanced can be used to have a user focusing on core elements and additional fields which are "expert options". ( Update Note: Was implemented finally via the section attribute)

Nevertheless re-using the existing Params class comes with a set of limitations compared to a separate definition of a user entry form:

• Initial idea to allow definition of multiple forms per DAG is complex, re-using Params restricts to a single entry form per DAG (which might be acceptable to ease definition)
• Deep nested complex JSON schema (other than simple key/value) will be very complex to handle in a generic form so the proposed implementation would be limited to this. Otherwise any complex and optional deep nested JSON structure is killing any generic
• Usage of enum to define pick lists unfortunately restricts the user to add optional additional entries because JSON schema validation will kick-in.

With this extension the current behavior is continued and users can specify if a specific UI form is offered for the Trigger DAG option.

Implementation Part 2) UI Changes for Trigger Button - OPEN

Note: During the implementation PR there was a short discussion with Ash Berlin-Taylor and not to block the merge, the implementation was reverted. As there might be multiple opinions on the "Trigger Button" implementation a vote is being called again in the devlist (Link https://lists.apache.org/thread/4dkbwob1wyl3xjbqdsmbd1mvgzflzp1f)

The function of the trigger DAG button in DAG overview landing ("Home" / templates/airflow/dags.html) as well as DAG detail pages (grid, graph, ... view / templates/airflow/dag.html) is adjusted so that per DAG the available form(s) can be controlled for better user guidance. 

Option B) SELECTED Trigger Button does not show the selection list anymore. Page handling is based on the DAG definition. If the DAG has params defined (which are not const only), the current view "Trigger DAG w/ config" is shown. If the DAG has none (or only const) params defined, the UI is skipped and the trigger is executed directly, returning to the origin page. Flowchart like this:

Implementation Part 3) Standard Implementation for Forms (Actually core new feature) - DONE

Implement/Contribute a user-definable key/value entry form (general layout following todays JSON entry) which allows the user to easily enter parameters for triggering a DAG.

An example parameter set could look like

with DAG(
    dag_id="example_form_demo",
    (...)
    params={
        "param1": Param(
            "default value",
            type="string",
            title="Parameter1",
            description="(Optional Description and hints)",
        ),
        "parameter_2": Param(
            "user selection",
            enum=["foo", "bar", "user selection"],
        ),
        "flag3": Param(
            False,
            type="boolean",
            title="Parameter3"),
            description="(Optional Description and hints)",
        ),
    },
) as dag:
    run_this_last = EmptyOperator(
        task_id="run_this_last",
    )
    (...)

A form for this example could look like:

   Parameter1: <HTML input box for entering a value>
               (Optional Description and hints)
   
   parameter_2: <HTML Select box of options>
   
   Parameter3: <HTML Checkbox on/off>
               (Optional Description and hints)
   
   <Trigger DAG Button>

The resulting JSON would use the parameter keys and values and render the following DagRun.conf and trigger the DAG:

{
  "param1": "user input",
  "parameter_2": "user selection",
  "flag3": true
}

The number of form values, parameter names, parameter types, options, order and descriptions should be configurable in the DAG definition.

Example view how today's solution is made via a UI plugin to integrate into Airflow UI (but is not linked into the standard "Trigger DAG" button):

Note that the advanced parameters and generated Job Config are hidden per default and need to be un-folded

Implementation Part 4) Examples - DONE

Provide 1-2 example DAGs which show how the trigger forms can be used. Adjust existing examples as needed.

Implementation Part 5) Documentation - DONE

Provide needed documentation to describe the feature and options. This would include an description how to add custom forms above the standards via Airflow Plugins and custom Python code. Proposal is to extend the docs section of "Params" (https://airflow.apache.org/docs/apache-airflow/stable/concepts/params.html) for the purpose of describing how to generate a form from the params definitions.

What problem does it solve?

With this proposal the functional gap is closed allowing to add user friendly trigger UI forms for staring DAGs with parameters (w/o the need that every user must understand the specific required JSON structure to trigger a DAG). Also it adds missing functionality which users know and expect to have from e.g. CI/CD pipelines of Github/Azure DevOps, Jenkins (and many more) that simple forms can be added to trigger with parameters.

Why is it needed?

• Adding missing functionality allowing user friendly trigger of workflows/DAGs w/o specific expertise of DAG conf needed (learning curve)
• Also such entry form could provide an easy path to validate parameters before triggering (comparing to try to trigger with any JSON and then the DAG hits the wall because of bad parameters passed if user is not expert)
• Do not scare users when showing how to (self-service) workflows
• Prevent the need to add custom UIs in front of Airflow to "hide" Airflow internal complexity

Are there any downsides to this change?

• Airflow Codebase will be extended by ~1000 LoC (additional potential maintenance)
• Extension of UI must be discussed in conjunction with ongoing discussion and implementation of AIP-38 Modern Web Application

Which users are affected by the change?

Standard and power users of Airflow are not affected, there is mainly a very positive option possible for rare or entry users allowing to trigger DAG w/o mus experience, lowering the entry complexity.

How are users affected by the change? (e.g. DB upgrade required?)

With the proposed change no user or integrator is affected, the proposed change is just an extension w/o breaking change and no need to change DB layout

Other considerations?

The initial idea of implementing forms as structured Python classes has been dropped by the feedback received to leverage the existing Params structure. Otherwise of course specific dedicated user entry form modules still can be added later. The proposed approach is a 80/20 proposal (gaining 80% of use cases direct benefit with 20% complexity).

Current workarounds known and used in multiple places are:

1. Implementing a custom (additional) Web UI which implements the required forms outside/on top of Airflow. This UI accepts user input and in the back-end triggers Airflow via REST API. This is flexible but replicates the efforts for operation, deployment, release as well and redundantly need to implement access control, logging etc.

2. Implementing an custom Airflow Plugin which hosts additional launch/trigger UIs inside Airflow. We are using this but it is actually a bit redundant to other trigger options and is only 50% user friendly as the standard "Run" buttons point to the JSON trigger dialog but not to the plugin and this is mis-leading from user perspective (not well integrate-able)

Open Questions needed to be discussed and decided before moving ahead?

• How does this extension relate and runs into a "potential dead end" of FAB if AIP-38 Modern Web Application is implemented in parallel? From editors perspective the proposed structure is not bound to FAB and param structure can also be transferred to a new modern UI.
  
  • And as a direct follow-up: Will AIP-38 need to be completed BEFORE making this? Will this be made in parallel and then needs to be migrated over? Can the described concept be included in AIP-38 from the beginning? (First hand w/o customizations would be 90% already)
• Shall we make a PoC implementation in form of a PR for the discussion?
• Might be out-dated with Draft v2? → Deployment: "security is super-important" - we need to know what will be the deployment model of those "customisations" - "who will be able to add them, how, what are the risks involved, whether ther will be any sandbox the custom code will be executed? Will it be server-side as well as client-only, or maybe it should only execute in the browser's javascript). I think there are a lot of "infrastructural" qustions to answer"
  Note/Partial Answer: Currently I would assume "standard" forms are with DAG development following any deployment/securty considerations as there is already potential to run/inject any arbitrary python code. Following same problems. For "custom" UI components it should follow the same approach like current Airflow Plugins. I'd propose to use FAB endpoints in a plugin and so this follows the same "problems" and challenges there are today. I do not want to add further attack points with this AIP. But might be related to question #1, if React will/shall be used.
• General question: "how UI customization should look like in a modern web app environment?" (Note: "modern" meaning "React")
• "ease of use for the end user is also a critical point (it should remain quite easy to extend the UI)" - How will this be achieved with this implementation?

What defines this AIP as "done"?

• Airflow delivers the option "out of the box" to define at least one form (per DAG) with simple input fields to trigger a DAG via UI and passing standard scalar parameters, e.g. string values
  
• Such forms can also be extended via UI plugins so that custom functionality can be directly linked into the standard UI "trigger DAG" buttons and allow replacing the standard dialogs w/o parameters or JSON dict entry