AIP-84 UI REST API

Status

Motivation

Today, the interaction between the frontend and backend is quite messy and inconsistent. In AIP-79 Replace Flask AppBuilder, we plan to remove much of FAB. We will need to rewrite a lot of endpoints, so we should take the opportunity to standardize on a new UI API.

Problems with our current approach

Right now, the React application communicates with the webserver in three different ways: using the Public REST API, FAB jinja variables, and then custom endpoints.

Public REST API

A major consideration of AIP-38 Modern Web Application was for us to use our own public REST API in order to keep it well maintained. We should do even more to add to our public API, but it is not appropriate to add everything to it. Once it's in the public, we have to guarantee backwards compatibility. This is very burdensome for endpoints that power specific UI views. For example, the Grid and Graph view. We should keep the flexibility of improving UI designs from version to version and ensuring backward compatibility would inhibit our ability to iterate.

FAB Jinja Templates

The React parts of the UI are loaded as javascript files inside of a FAB html template. Therefore, sometimes we pass static information to the React component via jinja templates. This is either embedding them inside of html meta tags in the head of the document (mainly API urls) or by specifying global variables in a separate script element. 

It’s a lot of manual boilerplate code and also makes both testing and linting harder because we need to add exceptions for both to be aware of. All of this should be replaced.

Custom Webserver Endpoints

This is how we power a lot of the React UI when the public REST API doesn’t quite make sense. Unfortunately, we have not done a good job of enforcing any standards on these endpoints. This has resulted in an inconsistent mess of testing, error handling, parameter and responses formats. The UI can’t infer anything and using any endpoint requires the developer to know exactly how the endpoint works without the help of types. This has also resulted in giving us an escape hatch to adding to the public REST API since it required less effort to write a bad custom endpoint. All of this exists in `views.py` which is cumbersome to understand with nearly 6000 lines of code. Much of this is also very tightly coupled with FAB. Therefore, most of, if not all of, this file should be split apart and rewritten. 

Proposal

We should try to add as much as possible to the public REST API but for what doesn’t make sense for that should exist in its own UI REST API. The React UI will consume both, and only communicate to the backend through these two APIs.

The UI REST API should be structured just like the public REST API. It should have an OpenAPI spec from which the UI can auto-generate types and even the query functions.

We looked into having this exist inside of the existing REST API and adding a flag to UI-only endpoints to exclude from documentation. If that proves to be too fragile then we can have a fully separate UI. While the public REST API maintains backward compatibility, the UI REST API does not. So it would be best not to mix them up. Since both APIs will be so similar, it makes it isn't as big of a step to contribute to the public REST API. Committers should use their judgement and keep the UI-only REST API minimal.

What defines this AIP as done?

• The React UI only communicates to the backend through with the public and UI-only REST APIs
• All queries and types can be auto generated in the UI from the APIs spec