AIP-68 Extended Plugin Interface for React Views

Status

Motivation

After the completion of AIP-79 Remove Flask App Builder we need to explore new ways to build UI plugins. Out-of-the-box, custom plugin views and menu items will no longer be supported. 

Additionally, after the completion of AIP-38 Modern Web Application we are looking forward to close gaps in UI for a better business integration. From a perspective of business users the UI might be very technical. You can see task details, logs and run-times. This is very good for a pipeline developer and operations teams, business users would also like to see custom details of the workflow that is represented behind the scene.

Today this is mostly compensated by embedding Airflow behind other UI's or business applications and just drive or schedule a pipeline, hiding the UI to business users. But Airflow UI - especially after AIP-38 - does not need to be shy and could be opened up also for (technical oriented) business users.

As Airflow UI is generic and allows deep customization's of Operators, Provider Packages, Workflows, the UI is quite generic. It would be great to be able to extend the Grid view to populate custom information via adding widgets or details besides adding full web pages or the need to populate another UI in front of the workflow.

Considerations

General knowledge of the UI and state of development is known by the AIP authors. From this perspective the following options have been considered as options to make a business user UI:

1. Implement a "Business Targeted" UI in front of Airflow - this might be exactly tailored to the UX and needs a user has to see what is going on with his business processes and details. Such a UI might offer the possibility to trigger workflows manually ( Here the trigger UI would need to be re-implemented mostly, triggering via API) and also as kind of reporting a business user could see his status of success ( For details a couple of state information points must be sourced from Airflow via API. Parts of the workflow must be re-modeled in a custom UI and for details probably a deep link to Airflow UI is needed. Up to 50-80% of complexity might be re-implement UI features). Sometimes Reporting tools like Grafana, Power BI etc. might serve the need to aggregate the results from resulting systems or also provide dashboards of workflows from Airflow database. Nevertheless a lot of UI, authentication might need to be implemented for the customer use case with overlap to Airflow - which means a big investment and maintenance. Also not forgetting the efforts to harden a separate UI for security and implementing Authentication, authorization, access control and redirecting such permissions to back-end API.
2. If Airflow delivers most business workflow information and just an overview is missing, you could assume that 80% of UI is reasonable but 20% is missing. In such case some Flask UI Plugins could be added to render custom web pages which allow an overview for the business user, maybe alongside a Grafana, Power BI etc. reporting for statistics. But most detailed information behind an overview might be visible in Grid view. But the Grid view provides mostly textual data on the technical level based on logs, properties and statistics. Sometimes you could use Operator Extra Links to connect the basic technical information with an external system showing more details. Still the UI would have some "custom parts" and some "Airflow standards" which are served from the same place but which are not integrated well.
3. Extensions as proposed in the AIP below, allowing the option to extend the Grid view for business information based on use case needs via an extended Plugin interface
4. Cutting down modules of current UI, allowing Airflow UI components to be embedded into other UX / business views. Whereas this is technically possible it would be a lot of re-work on the UI and would also limit end-2-end navigation which would need to be replicated by a container. It would be a lot of re-work of efforts from AIP-38.

What change do you propose to make?

We propose within the scope of this AIP that the existing Airflow Plugin Interface (after the cut-down of removing FAB and previous extension points for UI) is extended for the purpose of adding custom panels based on business needs into Grid views. Compared to separate Flask pages, business specific information could be added at the right place into panels where needed.

In detail the following proposals for plugin extensions are made possible:

• First Version / Increment
  • Grid View → DAG Overview
    • Custom Panel(s) on DAG Details
  • Grid View → DAG Run
    • Custom Panel(s) on DAG Run Details
    • Custom Tab (allowing to fill the whole content panel with a heading and optional icon) on DAG Run
  • Grid View → Task Instance
    • Custom Panel(s) on Task Instance Details
    • Custom Tab (allowing to fill the whole content panel with a heading and optional icon) on Task Instance
• Next increments
  • Custom Cluster Activity dashboard components
  • Dataset-specific plugins 
  • Trigger DAG Run, custom HTML rendered field entry (which was removed from initial AIP-50 implementation because of security)

Examples - Extended Grid views with a "Donkey example":

DAG Run View	Task Instance View
Custom Panel Area populated for DAG Run	Custom Panel and Tab for Donkey Statistics and History in Task Instance View
Custom Tab Added to DAG Run

Technical approach assumed to provide the panels:

• Panels are created in JavaScript, and registered as a plugin with a new entry_file field on a Plugin. A user can also specify a tab name and even icon. This all can take two approaches:
  • Simple Path
    • Write any JavaScript, and it will be loaded in isolation inside of the Airflow UI. The JavaScript can render any content on the UI
      ( Note that it is assumed that the plugin owner injects only trustworthy code as the script has full access to the browser context. So it can not be made by a DAG author or user)
      
  • Advanced Path:
    • Build a React application using a provided template that has the same dependencies as the Airflow UI
    • We can pass specific props to the plugin such as Task Instance details
    • Experience can look identical to the rest of the Airflow UI
    • It would be nice to include a sort of airflow init plugin cli command to generate the necessary templates
• Airflow Plugin provides
  • Flask endpoints to load JS code fragments of the plugin
  • Via the endpoint static resources like images, CSS and JavaScript must be able to be added besides the entrypoint.
  • Registers the plugin such that the React UI loads the fragments where needed
  • Basic JavaScript APIs/functions to integrate with the page context as well as functions to call further details via REST API from the backend
• Security and deployment is in the responsibility and ownership of Deployment Manager
• Panel details and plugin selection is made on DAG name/tags and Operator type. (Similar like the mapping to Extra Operator Links today)
  

It is assumed that multiple plugins can be loaded and extend the UI.

What problem does it solve?

Once FAB is removed we need a new way to do plugins. Those plugins can be better integrated into the UI, showing up next to where the user needs them vs. tacken on as a new page. The new/current Grid view is generic and shows all technical information of an Operator - with the option to extend via Plugins the UI can be enriched with business relevant information above the current Operator Extra Links

Why is it needed?

Save users to develop another UI on top if Airflow serves for 60% of needs and UX is required to be added to close the gap.

It also gives the opportunity besides OperatorExtraLinks to deeply integrate further functions from other backend systems (like the current provider of Databricks provides a plugin) into the UI so that additional standard operations can be controlled w/o pointing to external UIs.

Are there any downsides to this change?

• (Small) Added complexity
• Web page structures and functions (e.g. to access REST backend) must be a stable API allowing the plugin to integrate and be usable across multiple versions of Airflow
• Risks if bad plugins are added by developers
  • Risk of performance problems (e.g. UI is badly implemented and creates a lot of API calls to backend)
  • Security problems generated by bad implemented plugins
  • XSS problems by bad HTML code generated by Plugins

Which users are affected by the change?

Plugin would be optional to be extended

How are users affected by the change? (e.g. DB upgrade required?)

Assumed no DB changes. All by Python code, JS, templating and API extensions

Other considerations?

• Airflow 2.x and discussions about upcoming Airflow 3.0 development: AIP authors initially targeted to have this AIP implemented in Airflow 2.10 In light of efforts in UI integration and planned rework (especially to remove FAB) the plugin concept of Airflow 2.x must be reworked in general.
  
• Therefore the scope of this AIP has been shifted to be integrated in Airflow 3.x only.
  
• This AIP is specific to a new style of Javascript plugins  The work of removing FAB plugins is part of AIP-79.

What defines this AIP as "done"?

• The extended Plugin interface is with the described features available in a first increment
  
• Further intended features are implemented listed in the scope
• Airflow users/developers can implement a plugin based on a provided template.
• Documentation is complete
• Example is provided