Status

Current state:  PoC Available

Discussion thread: https://lists.apache.org/thread/knd5xgkswp83x92c0sclh7ghcob7bpfx

JIRA: SOLR-14414 - Getting issue details... STATUS

Released: -

(Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast). Confluence supports inline comments that can also be used.)

Motivation

  1. Improve overall security by removing EOL code from the code base

  2. Preserve functionality by providing an updated Admin UI

  3. Make Solr more approachable by both users and devs

  4. Improve maintainability of UI code base

  5. Improve UI/UX of the UI (in favor to SIP-10)

  6. Plan and implement new Admin UI features

  7. Modernize the UI look and feel

  8. Migrate to v2 API

  9. Improve error handling

  10. Improve limited user's experience

  11. Introduce a desktop client

Public Interfaces

No interface will be changed, existing API is consumed for the new implementation.  Existing Admin UI will coexist with the new UI until the new UI is production ready and can replace current UI.

The option to provide new interfaces for extending the UI may be considered in the future.

UI-specific interfaces that only the current UI is consuming will be deprecated and removed together with the current UI.

Proposed Changes

The proposed changes include a full rewrite of the Admin UI by implementing step-by-step a new UI and co-hosting it with the current UI until it can be fully replaced. Various links in the current UI should allow users to test the new implementation early on and provide feedback.  (See for that the current POC PR#2605.)

The current POC uses Compose Multiplatform, a Kotlin-based UI framework that is very popular in Mobile app development and comes with desktop and web support. See the detailed writeup here for more details on why ComposeMP was chosen as a candidate.  Other UI frameworks, mainly web-based frameworks, are still considered if Compose MP is proven unsuitable in the POC.

The new UI should also aim to address as many existing issues as possible, starting with the issues already associated with our JIRA project's "Admin UI" component. Still, focus should remain on the most "important" features that are necessary for replacing the current UI. New features may be introduced in a later stage.

The proposed design (i.e. look-and-feel) and page layout can reviewed using the provided Figma mockups. 

**TODO**: A new discussion thread for the new designs and accessible by non-devs should be created and linked here soon.

Compatibility, Deprecation, and Migration Plan

In order to realize the proposed changes and proceed with the migration plan (see below), the new UI implementation has to be accessible via web. The new UI should ideally be linkable in the old UI so that users can start using the new UI during the experimental / development phase and give feedback.

The work should be split into two phases:

Preparation Phase (steps not necessarily in chronological order):

  1. Create new designs based on current state (ongoing)
  2. Create epic and prepare Jira issues (planned)
  3. Choose a framework/language (ongoing)

Development Phrase:

  1. Implement and merge base for new UI (planned)
  2. Implement new UI (planned)
  3. Deprecate old UI (planned)
  4. Remove old UI (planned)

Preparation Phase

Create new Designs based on current state (ongoing)

In order to address many UI / UX flaws of the current UI, we should plan ahead and create new designs, discuss them and in an iterative way improve and extend them if necessary. This process should involve the Solr users too.

There are already a couple designs created for the new UI that can be found here. Use the current discussion thread for feedback, ideas and any other kind of input related to the new designs.

Create epic and prepare Jira issues (planned)

To address the large scope of the new UI implementation, we should make use of an epic and splits the UI into smaller chunks the community can easily be work with. This way we will avoid a single, huge PR that is hard to review.

Choose framework/language (ongoing)

In order to select a suitable UI framework, we should create a proof-of-concept (POC) for the most interesting candidate and validate its integration, code complexity and learnability.

  1. Create a POC (completed) - the current POC uses Compose Multiplatform for a better integration experience and dependency management that fits well into the current Solr project (Gradle/JVM)
  2. Review POC (ongoing) - the POC implemented with Compose Multiplatform requires the code contributors to review the changes and judge based on their experience the UI framework and integration.  Use the Github PR for asking questions related to the POC code.

Development Phase

Implement and merge new UI's base (planned)

In order to work with the new UI and merge continuously and over time new changes, a proper base needs to be available. This could be covered partly or completely by a POC.

The new UI can be distributed with any upcoming release and can simply be marked experimental for the time being.

Implement new UI

Once the base is available and the new UI coexists with the current UI, developers can start working on the Jira issues and implement new UI components.

The epic should include all features necessary in order to complete this step and proceed to the deprecation and removal of the old UI.

Deprecate old UI

Once the new UI is usable and could potentially replace the old UI, we may not have received enough user feedback and tested it in production. For that reason, we should mark and disable the old UI as deprecated for a period of time until we can safely remove the old UI and fully replace it with the new UI.

In this phase new plans and features can be planned and developed for the new UI.

Remove Old UI

Once we ensured that the new implementation is tested and used enough by users, we can proceed and remove the source code of the old UI.

Security Considerations

The current UI contains EOL JS code and libraries. They pose a variety of risks and leave Solr more vulnerable at that vector. New UI frameworks address many of the web application vulnerabilities listed by OWASP and therefore are considered an improvement in security.

Even though new UI frameworks are considered safer now, they become more and more complex and have many dependencies that could also pose new risks. Therefore, maintenance and tool support will also play a significant role.

Test Plan

The typical test plan includes unit tests and integration tests that are executed with our automation scripts.

Since UI testing is not always straight-forward, an additional test plan is added with the proposed migration plan. "Early adopters" / Users will be able to use the new UI during its development. In this phase, feedback can be provided early on, allowing us to address any matter early on. This includes feedback, feature requests, UI/UX issues and bugs in general.

Note about Compose MP (POC)

In the POC code changes, a WebAssembly app is launched aside the current webapp, which requires the `script-src` flags `unsafe-eval` (in development) and/or the `unsafe-wasm-eval` (in production) to be allowed (see unsafe WebAssembly execution for more details).

Alternatives

An alternative to the proposed UI framework is a solution based on web-frameworks like Angular, Vue and React. Choosing one of these frameworks would have the following benefits:

  • Large community base (varies between framework)
  • Industry-standard

The drawbacks that could be expected are:

  •  Hard time to find web developers join a Java-focused project if the code base lives inside

These alternative frameworks have not been rejected yet and could be considered in case Compose Multiplatform is proven unsuitable as a framework.

Rejected Alternatives

Moving UI code out of Solr project

Although the motivations behind this approach are reasonable, it would have a larger impact on Solr, as it would turn into a headless system.

This is considered a huge drawback for beginners, as the setup steps are increased and third-party tools have to be trusted to get a UI dashboard. Additionally, new UI developers may find their way to the UI project, but not to Solr, resulting to potentially fewer contributions in both UI and the Solr project.

Continuing with EOL AngularJS

See motivation of this SIP-7.

Additional References

Old Discussion Thread - https://mail-archives.apache.org/mod_mbox/lucene-dev/202004.mbox/browser

Glossary

  • Admin UI, UI, webapp - Any official frontend of Solr (regardless of the implementation) that is used to control and interact with Solr
  • Current UI, current webapp, old UI, old webapp - The Admin UI implementation that is written in AngularJS and is subject to be replaced.
  • New UI - The Admin UI implementation that is planned to be introduced and that is supposed to replace the current implementation
  • No labels

6 Comments

  1. Marcus Alexander Eagan

    Here are a few pros and cons of each framework, as I have read on the web and influenced by my limited experience:

    FrameworkProsCons
    Angular
    1. Same framework, though considerably different from previous UI
    2. TypeScript by default with awesome support for it
    3. Opinionated, restricting developers to do things a few ways
    4. Backed by Google, Licensing and its history safe
    5. Great integration and support with tools
    1. Slow performance
    2. Decreasing in popularity and community, particularly on newer versions
    3. Far fewer up to date libraries and community contributed docs
    4. I have found some docs that are official to be incomplete or
    5. Big files can be tricky to follow.

    React
    1. Most popular in terms of adoption in the market
    2. Largest Dev community, so easy to find contributors
    3. Provides interesting options for doing things like porting to mobile app
    4. Fast 
    5. Keeping updated is very easy

    Heavy, so it would add some weight to installations that enable UI 

    1. History of licensing concerns but they walked those back quickly
    2. Templating a logic can live in the same file and that can be confusing
    3. Pro for me but functional programming paradigm can be difficult for some
    4. Redux or Relay dependency for state management
    5. Un-opinionated which can grow unwieldy for open source
    Vue
    1. Fastest of the three major frameworks
    2. Lightweight (20KB)
    3. Slowly growing community
    4. Open source friendly
    5. Interesting portability to desktop and mobile apps with Electron
    1. Few resources
    2. By far smallest community by many multiples
    3. Not a con for everyone in this group, many discussions in Chinese
    4. Risks with Vue for large projects because it was designed for prototypes
    5. Few developers are experienced in managing Vue at scale

    https://medium.com/@TechMagic/reactjs-vs-angular5-vs-vue-js-what-to-choose-in-2018-b91e028fa91d

  2. Marcus Alexander Eagan

    I believe that from my perspective, the most important thing is to quickly provide a modern UI that is safe and helps users meet their needs. All options provide this option.


    Secondarily, I think we need to attract a couple really dedicated contributors.


    Thirdly, it needs a sub-repo to become more manageable.

  3. Marcus Alexander Eagan

    Jan Høydahl and others, I went through the Vue project after I finished with life stuff and work stuff and I like it a lot. I also reached out to the author. There are two things I think might be helpful to get to a place where I (personally) would better understand the YASA project better as an option:

    1. We need to work through all the dependencies because I have seen some that are not necessarily the standard or maybe seem out of date. If I run the npm tools I get warnings and things and one of them is for ace-code-editor. Using this library as an example to nit over, we probably want to use monaco, the Microsoft package. It's kept up to date and powers many important IDEs/text editors. It will be some work to go through each dependency in the package.json  and remove them and refactor the components. It's templating system is more similar to  the existing Angular one, and simpler than the JSX JavaScript-XML portmanteau. On the bright side, this dependency clean up work probably would have a longer shelf life than with the other frameworks because Vue doesn't change that fast. There is a risk that some packages will need to be fixed upstream if we find something really hairy one day. I suppose that's always a risk in open source.

    2. The Vue project, while the furthest along, does not have any tests afaik. Even the scaffolding tests in the other projects are good and a big jump forward because they verify that a component compiles after changes. We can probably get a lot of the equivalent with YASA quickly , but I would recommend someone or I write a couple to understand what the complexity is there. It doesn't seem to be too high. Here is a great starter resource.



    From my list of pros and cons above I want to call out two cons I listed.

    1st, a React con:

    Un-opinionated which can grow unwieldy for open source

    In my limited experience with Solr, adding an un-opinionated way of doing things to the open source project could be a risk. If there were more leadership and focused stewardship on this element of the project, I would feel comfortable with React. Without it, I think it's going to be more trouble than it's worth and actually lead to problems eventually. I have worked on and/or led React projects at different points in my life, and sadly they benefited from autocrats.


    2nd, a Vue con:


    Not a con for everyone in this group, many discussions in Chinese

    While I first listed as a con, this could almost be considered a pro for the overall Solr project. In China, and other places in Asia where Chinese is widely spoken, Solr has a robust user community. I don't know if the big tech companies using Sorl in the past still are, but I know there are a number of companies in the Chinese-speaking region using Solr. I have interacted with some of them. Invigorating a pool of any # of developers > 0 in the region seems like a great opportunity to me. There are other areas in which Solr could improve for its users in the Chinese-speaking region and I like any additional entry points. Most importantly, the charge that a lot of the documentation out there is in Chinese mostly speaks to early on, Google Translate has gotten a lot better if you really need to know what's going on, and I need to improve my Chinese quite a bit. 


    Tomorrow I will try to get the application running. Ran into a few issues tonight on the Solr side. Will need to dig in with more clarity/time.



  4. Marcus Alexander Eagan

    The front end engineers I know consistently say that React has a big lead and that most front end engineers know and prefer React. Without governance, it cannot be an option. 

  5. Jan Høydahl

    I saw the YASA developer just reached out on the dev list, willing to help. That is another PLUS for YASA/Vue (smile) 

    I think the next steps on this SIP would be to rewrite most of the text, especially the "Proposed Changes" section. Instead of discussing details, test strategies and exactly how the development environment will be, focus on short sentences of what we propose. Phrase it so that people easily can understand and vote over it. Under "Rejected Alternatives", we can perhaps add sentence for each rejected alternative.

    Here are replacement texts for the SIP, reflecting my current personal opinion. Feel free to use (smile):

    Proposed Changes

    After fully implementing this SIP, Solr will ship with a completely rewritten Admin UI, based on modern frontend development in the Vue.js framework.

    These are the major parts of the proposal

    1. Solr's binary distribution will include the Admin UI out of the box without additional downloads, as today. The UI can be disbled with a System property.
    2. The new Admin UI source code and deveopment tools will not live in solr git repo, but in its separate git repo. This makes a cleaner experience for both Java development (main repo) and UI development (ui repo)
    3. The new UI will be a build-time dependency of Solr, where gradle downloads and unpacks the precompiled UI distribution for inclusion in the Solr binary distro. Solr can ship with the same version of UI for several releases if needed.
    4. The Admin UI source code will live in a separate git repo, generating its own precompiled releases. A benefit is that UI can easily be upgraded or patched without a new Solr release.
    5. The UI will be based on YASA (https://github.com/yasa-org/yasa), an existing 3rd party drop-in replacement Admin UI. YASA gives a cleaner layout, new features, and a good platform to continue development
    6. YASA is developed in Vue.js and for that reason Vue.js is chosen as framework. Vue is a modern, popular frontend framework

    The YASA author is an Apache committer and will continue to contribute to the UI and is willing to donate all code to ASF. We propose to do the first couple releases from the yasa-org GitHub repository. Once the Solr TLP is established we can explore the option of establishing Admin-UI as a sub project in a 'solr-admin' git repository. But until that happens those who wish can join the yasa-org github organization to become comitters.

    See migration plan for the steps planned during the migration.

    Security Considerations

    Security is a main motivation for this SIP. Removing the EOL AngularJS UI is a win for security. The new UI is not expected to add any threat vectors to Solr, and with the ability to disable the UI with a switch, those who don't need a UI will never run the UI. The new UI has the potential to open security threats in Solr if it is not kept up to date. It is far easier to stay on top of JS library upgrades with the npm tooling than the manual procedures in current UI.

    Compatibility, Deprecation, and Migration Plan

    Since the new UI is a complete rewrite, it is not "compatible" with the old UI. So all screenshots in guides, blogs etc will be obsolete, and all old Admin UI URLs will no longer work.

    We plan to deprecate the old Admin UI once the new UI is releases, but keep shipping both with a toggle button until the new UI achieves feature parity. Then the old Admin UI will be deleted.

    In first phase we'll develop the new yasa UI under its current GitHub organization and perform releases as a 3rd party app. At a later stage (not part of this SIP) we may establish a Solr sub project for the Admin UI and move the git repo under ASF control. That means we will also do separate UI releases from the Solr project, downloadable from the mirrors.

    Test Plan

    The new UI will have built-in unit tests, unlike the old UI. The unit tests will assert basic UI functionality.

    There is also an interest to establish end-to-end tests with Solr+UI, run by a separate Jenkins job. We propose setting up a couple of example tests as part of this SIP, so there is something to build upon.

    Rejected Alternatives

    Continuing with current EOL UI was not an option.

    Shipping Solr without any UI was not an option, since we believe Admin UI is such an important part of Solr for our users that it needs to be released with Solr.

    We considered keeping the UI source code in main solr repo, but chose a separate repo because it is cleaner, and developer feedback indicates that both Java and Frontend developers will be happier with dedicated playgrounds. It should thus be easier to attract pure UI contributors when they are not intimidated by all the Java and gradle stuff.

    We considered the proposal of shipping the UI as a Solr package. Decoupling the UI as a build dependency as we propose here is a step in that direction. However, to keep the scope and risk of this SIP down, we propose not to change the distribution method now, but keep the UI as part of the release. Perhaps this can be an option in the future, if/when the package framework gets more such hooks, and a repo for native packages released by the project.

  6. Marcus Alexander Eagan

    This is a fine proposal thanks Jan Høydahl