As user of Airflow for our custom workflows we often use
DagRun.conf attributes to control content and flow. Current UI allows (only) to launch via REST API with given parameters or using a JSON structure in the UI to trigger with parameters. This is technically feasible but not user friendly. A user needs to model, check and understand the JSON and enter parameters manually without the option to validate before trigger.
Similar like Jenkins or Github/Azure pipelines we desire an UI option to trigger with a UI and specifying parameters. We'd like to have a similar capability in Airflow.
This AIP was created after working a longer time with an UI plugin which we implemented customized allowing users to "easily" trigger DAGs with parameters. More and more we realize that every use case and demo we make requires an simple entry form to show "how easy" it is (not to scare users and have them running away). We propose to "productize" the feature from a plugin and merge it into the Airflow Core UI - hoping that more people can use this feature, having a lower entry barrier for "new" users and also to close gaps from standard UI path to trigger a DAG today compared to how the UI plugin is integrated.
We also had (in the past) a separate UI (outside of Airflow) just to collect parameters and trigger the Airflow DAG in the backend via REST API - but this was a full separation of the user from Airflow and at the end requires a very redundant UI being developed and maintained.
What change do you propose to make?
This (technical) proposal is made based on current Flask AppBuilder (FAB) and existing structures, Changes made for AIP-38 Modern Web Application might influence the final implementation. Feedback welcome.
Implementation Part 1) Specifying Trigger UI on DAG Level
We propose to leverage the
params attribute of the
DAG class with some additional optional attributes so that an UI (one form per DAG) can be specified in the DAG itself (optional). As the
params attribute is built upon the
Params class to define meta-data to serve the purpose of JSON schema validations, this definition already serves 80% of details needed to render a form. See also https://airflow.apache.org/docs/apache-airflow/stable/concepts/params.html.
The following details to model a form can directly be leveraged from existing
Params class definition:
- The order of fields of the form
- Name of the field, using the dict key. As this might not be very descriptive the existing JSON schema field
titlecan be used if available for a better "Human readability.
- Type of the field derived from the JSON schema type and formatting clause
booleancan render a checkbox or on/off control
- Elements of JSON format
date-timecan render a date picker
- Type of
arrays can be made as lists of text string and
objecttypes as a JSON sub-element entry
- Elements defined with an
enumcan be used to define drop-down selects for a pre-defined list op options
- If JSON schema allows to have types of
nullcan control if a field is optional or required.
- The JSON schema attribute
descriptioncan be used to render field help text
- Some JSON schema attributes for text like
maxLengthcan be used to restrict user input on the form immediately
- Further attributes can be flexibly defined on
Paramsclass if we want to add form features in the future. Nevertheless available options and features can not be validated inside your editor as it is just extending
kwargs. Therefore a later switch from FAB to a modern UI can be done w/o changing the structure f Form definition, just add a different UI logic.
- If just an arbitrary dict is used as default
paramsthen the existing dict value data types can be used to "guess" a suitable form even if the
Paramsclass for JSON schema is not used.
- We can use more attributes on the
Paramsclass also to allow rendering of custom HTML field
Some additional (optional) attributes are proposed to be added to control a form layout:
- An additional attribute
hiddencan be used to pass static values without populating to forms. Maybe not all values shall be presented to user.
- An additional attribute
advancedcan be used to have a user focusing on core elements and additional fields which are "expert options".
Nevertheless re-using the existing Params class comes with a set of limitations compared to a separate definition of a user entry form:
- Initial idea to allow definition of multiple forms per DAG is complex, re-using
Paramsrestricts to a single entry form per DAG (which might be acceptable to ease definition)
- Deep nested complex JSON schema (other than simple key/value) will be very complex to handle in a generic form so the proposed implementation would be limited to this. Otherwise any complex and optional deep nested JSON structure is killing any generic
- Usage of
enumto define pick lists unfortunately restricts the user to add optional additional entries because JSON schema validation will kick-in.
With this extension the current behavior is continued and users can specify if a specific UI form is offered for the Trigger DAG option.
Implementation Part 2) UI Changes for Trigger Button
The function of the trigger DAG button in DAG overview landing ("Home" /
templates/airflow/dags.html) as well as DAG detail pages (grid, graph, ... view /
templates/airflow/dag.html) is adjusted so that per DAG the available form(s) can be controlled for better user guidance. Propsal is to add a DAG attribute
If there is a single Trigger UI specified for the DAG, the button directly opens the form on click
If a list of Trigger UIs is defined for the DAG, then a list of UI's is presented, similar like today's drop-down with the today's two options (with and without parameters).
Menu names for (2) and URLs are determined by the UI class members linked to the DAG. The following options are proposed:
none": Skips any user entry form and triggers the DAG w/o further details using the default parameter values (like today's option "Trigger DAG")
json": Brings the user to the form where he can enter the JSON as dict like today with "Trigger DAG w/ config"
form": Shows the new form as described above as new feature in AIP-50
- a tuple with
("display": "url")can be used to provide a generic text label and a URL to forward the user to a custom plugin or UI as seamless integration if more complex UI is needed
DAGs not defining the attribute
trigger_ui will have the options "none" and "json" like today plus if the DAG has
params defined also a "form" ("form does only make a benefit if
params are defined, else there is no base to render an empty (use-less) form)
Implementation Part 3) Standard Implementation for Forms (Actually core new feature)
Implement/Contribute a user-definable key/value entry form (general layout following todays JSON entry) which allows the user to easily enter parameters for triggering a DAG.
An example parameter set could look like
A form for this example could look like:
Parameter1: <HTML input box for entering a value> (Optional Description and hints) Parameter2: <HTML Select box of options> Parameter3: <HTML Checkbox on/off> (Optional Description and hints) <Trigger DAG Button>
The resulting JSON would use the parameter keys and values and render the following
DagRun.conf and trigger the DAG:
The number of form values, parameter names, parameter types, options, order and descriptions should be configurable in the DAG definition.
Example view how today's solution is made via a UI plugin to integrate into Airflow UI (but is not linked into the standard "Trigger DAG" button):
Note that the advanced parameters and generated Job Config are hidden per default and need to be un-folded
Implementation Part 4) Examples
Provide 1-2 example DAGs which show how the trigger forms can be used. Adjust existing examples as needed.
Implementation Part 5) Documentation
Provide needed documentation to describe the feature and options. This would include an description how to add custom forms above the standards via Airflow Plugins and custom Python code. Proposal is to extend the docs section of "Params" (https://airflow.apache.org/docs/apache-airflow/stable/concepts/params.html) for the purpose of describing how to generate a form from the
What problem does it solve?
With this proposal the functional gap is closed allowing to add user friendly trigger UI forms for staring DAGs with parameters (w/o the need that every user must understand the specific required JSON structure to trigger a DAG). Also it adds missing functionality which users know and expect to have from e.g. CI/CD pipelines of Github/Azure DevOps, Jenkins (and many more) that simple forms can be added to trigger with parameters.
Why is it needed?
- Adding missing functionality allowing user friendly trigger of workflows/DAGs w/o specific expertise of DAG conf needed (learning curve)
- Also such entry form could provide an easy path to validate parameters before triggering (comparing to try to trigger with any JSON and then the DAG hits the wall because of bad parameters passed if user is not expert)
- Do not scare users when showing how to (self-service) workflows
- Prevent the need to add custom UIs in front of Airflow to "hide" Airflow internal complexity
Are there any downsides to this change?
- Airflow Codebase will be extended by ~1000 LoC (additional potential maintenance)
- Extension of UI must be discussed in conjunction with ongoing discussion and implementation of AIP-38 Modern Web Application
Which users are affected by the change?
Standard and power users of Airflow are not affected, there is mainly a very positive option possible for rare or entry users allowing to trigger DAG w/o mus experience, lowering the entry complexity.
How are users affected by the change? (e.g. DB upgrade required?)
With the proposed change no user or integrator is affected, the proposed change is just an extension w/o breaking change and no need to change DB layout
The initial idea of implementing forms as structured Python classes has been dropped by the feedback received to leverage the existing
Params structure. Otherwise of course specific dedicated user entry form modules still can be added later. The proposed approach is a 80/20 proposal (gaining 80% of use cases direct benefit with 20% complexity).
Current workarounds known and used in multiple places are:
Implementing a custom (additional) Web UI which implements the required forms outside/on top of Airflow. This UI accepts user input and in the back-end triggers Airflow via REST API. This is flexible but replicates the efforts for operation, deployment, release as well and redundantly need to implement access control, logging etc.
Implementing an custom Airflow Plugin which hosts additional launch/trigger UIs inside Airflow. We are using this but it is actually a bit redundant to other trigger options and is only 50% user friendly as the standard "Run" buttons point to the JSON trigger dialog but not to the plugin and this is mis-leading from user perspective (not well integrate-able)
Open Questions needed to be discussed and decided before moving ahead?
- How does this extension relate and runs into a "potential dead end" of FAB if AIP-38 Modern Web Application is implemented in parallel? From editors perspective the proposed structure is not bound to FAB and param structure can also be transferred to a new modern UI.
- And as a direct follow-up: Will AIP-38 need to be completed BEFORE making this? Will this be made in parallel and then needs to be migrated over? Can the described concept be included in AIP-38 from the beginning? (First hand w/o customizations would be 90% already)
- Shall we make a PoC implementation in form of a PR for the discussion?
Note/Partial Answer: Currently I would assume "standard" forms are with DAG development following any deployment/securty considerations as there is already potential to run/inject any arbitrary python code. Following same problems. For "custom" UI components it should follow the same approach like current Airflow Plugins. I'd propose to use FAB endpoints in a plugin and so this follows the same "problems" and challenges there are today. I do not want to add further attack points with this AIP. But might be related to question #1, if React will/shall be used.
- General question: "how UI customization should look like in a modern web app environment?" (Note: "modern" meaning "React")
- "ease of use for the end user is also a critical point (it should remain quite easy to extend the UI)" - How will this be achieved with this implementation?
What defines this AIP as "done"?
- Airflow delivers the option "out of the box" to define at least one form (per DAG) with simple input fields to trigger a DAG via UI and passing standard scalar parameters, e.g. string values
- Such forms can also be extended via UI plugins so that custom functionality can be directly linked into the standard UI "trigger DAG" buttons and allow replacing the standard dialogs w/o parameters or JSON dict entry