Status

Current state: Under Discussion

Discussion thread: https://mail-archives.apache.org/mod_mbox/lucene-dev/202004.mbox/browser

JIRA: SOLR-14414 - Getting issue details... STATUS

Released: 9.0.0

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast). Confluence supports inline comments that can also be used.

Motivation

  1. Improve overall security by removing EOL code from the code base
  2. Preserve functionality by providing an updated Admin UI
  3. Make Solr more approachable by both users and UI devs, while hopefully making it easier to get started with Solr today.

Public Interfaces

Same interface, new look. Existing Solr Admin UI would be removed by the time we are preparing to release 9.0 because we will have reached feature parity and have integrated the application into the standard pipelines for starting and deploying Solr. This could happen before we reach 9.0 as well and it may behoove the community at large to strip out the old UI as soon as 

Proposed Changes

Build an Admin UI.

The new UI will use a module loader and npm to start the app.

Many of the existing features will improve in terms of accessibility and capabilities. For example, logs will be sortable by every field, searchable, and paginated in the end state.The front page updates will make it easier to discern issues for new users or regular users of the Admin UI.

Development

The proposed run time will be be delivered over the course of 2 phases, with the option of a third phase if we can find the time to collaborate but it is by no means required because it adds lots of work. Sandbox, as a Jetty-issued command, and then possibly as a package manager if I can collaborate with the package management team (currently not an option). 

As of today, here are the proposed phases:

1) In the sandbox phase, where we are today, users can run Solr as they normally would. If they would like to try the new UI, they need to navigate to the same directory solr/ where webapp/ lives, and they will need to cd into webapp2/  dir. After that, they need to run npm start. That command will build the UI and then allow users to visit both Admin UI's simultaneously. This is a simple effort that will allow the community and the team to provide real-time feedback such that improvements can be made. The application will not be bundled at all in production environments in this phase. Phase one, though not entirely feature complete, is very important for our community to get familiar with the new application and its look and feel. If anyone would like to work through a first commit with me, I will talk with my company and ensure I have the appropriate time to do this work at a reasonable hour (I'm working on this projects 98% of the time, on nights and weekends). I am always available for critical bug fixes or security vulnerabilities because my company depends on Solr quite a bit. The new Admin UI would be available in this phase at 4200 .

2) In phase 2, the app will start with the existing Solr start command. In this case, the admin UI would have production mode enabled, and will be served up in production mode by default. This change will result in a much smaller application, minified, that is compiled AOT, and stored in a dist  folder in the UI application's root. The application will then be packaged by Gradle into a JAR file using the gradle-node-plugin, thereby eliminating the need to have node installed on the system running the application. For implementation details on how this would look, you can view an example here: https://dzone.com/articles/integrating-java-and-npm-builds-using-gradle. You can also count on the documentation that I produce in the README to be very thorough and inclusive of steps, tactics for address errors, and alternative methods of serving the application. The new Admin UI would be available in this phase at 8983  but will be configurable to start at any port that you desire.

Tests

Tests will be executed in a similar fashion and as described in the link above, in the appropriate phases. By the time we reach phase 2, I hope to have worked with committers to get all the Jenkins configurations worked out to support end-to-end testing. There are some things that I need to work out before we are there, but should be done well before we are ready for Solr 9 to be merged. In fact, we should have developed a regular engagement and collaboration for working through the integrating the front end into the global CI pipeline. The added time to the UI tests should be very minimal as the application is much smaller in comparison. I will leave it to the community as to whether we make the UI tests run on every build or only for specific target. My suggestion is that if you expect the UI to work when a backend change is made, it should be added to the standard target. Admittedly, this point warrants further discussion. 

Development

As for users seeking to work on the UI, the steps will be most similar phase 1, where you need to cd from lucene-solr/solr/  into webapp2/  and run npm start. The initial build will take some time, but all subsequent changes will be reflected in the UI immediately after saving a file. If it is not a big file, it should transpile in under one second. The files are mostly structured as TypeScript files that are converted to JS at compilation, SCSS files which compile to CSS files at compilation but support standard CSS, and HTML files. There are also two files for managing packages package.json  and package-lock.json . There are thousands of node modules that are downloaded in development mode but those comprise generated code and should not be added to the repository. I have added a .gitignore to filter those and other generated files out.

Security Considerations

The current UI contains EOL JS code and libraries. They pose a variety of risks and leave Solr more vulnerable at that vector. Last year we saw an instance where using the unsupported framework led to very dangerous code living in the Admin UI application. This new framework addresses many of the web application vulnerabilities listed by OWASP. It treats all input as untrusted by default, and it has tools built in for preventing XSRF and XSSI. There are more features as well.

Eventually, it would be great to introduce some more integrated security features for the user community of Solr. Many Solr managers around the world have other teams that would like to use the Solr Admin UI, but they cannot risk those teams having access to options like deleting a collection. With this application architecture, it is very simple to expose certain modules or components based on privilege. I also have some knowledge of the authentication frameworks because I have worked with it previously and have contributed to the basic auth plugin in Solr today (with great guidance from Jan and Shalin). 

Compatibility, Deprecation, and Migration Plan

  • Users will be happier.
  • Both UI will exist in parallel for some time until we agree it is time to deprecate and then remove old UI.
  • When we reach feature parity and the Application has been integrated into Gradle and Jenkins (Phase 2 above), it will be safe to remove the old UI and move forward.

Test Plan

The current Admin UI has no automated testing capabilities while the new Admin UI will have basic unit testing capabilities at the component level. Evnetually, in phase 2, it will support e2e integration tests.

Rejected Alternatives

Continuing with EOL AngularJS

  • No labels

6 Comments

  1. Marcus Alexander Eagan

    Here are a few pros and cons of each framework, as I have read on the web and influenced by my limited experience:

    FrameworkProsCons
    Angular
    1. Same framework, though considerably different from previous UI
    2. TypeScript by default with awesome support for it
    3. Opinionated, restricting developers to do things a few ways
    4. Backed by Google, Licensing and its history safe
    5. Great integration and support with tools
    1. Slow performance
    2. Decreasing in popularity and community, particularly on newer versions
    3. Far fewer up to date libraries and community contributed docs
    4. I have found some docs that are official to be incomplete or
    5. Big files can be tricky to follow.

    React
    1. Most popular in terms of adoption in the market
    2. Largest Dev community, so easy to find contributors
    3. Provides interesting options for doing things like porting to mobile app
    4. Fast 
    5. Keeping updated is very easy

    Heavy, so it would add some weight to installations that enable UI 

    1. History of licensing concerns but they walked those back quickly
    2. Templating a logic can live in the same file and that can be confusing
    3. Pro for me but functional programming paradigm can be difficult for some
    4. Redux or Relay dependency for state management
    5. Un-opinionated which can grow unwieldy for open source
    Vue
    1. Fastest of the three major frameworks
    2. Lightweight (20KB)
    3. Slowly growing community
    4. Open source friendly
    5. Interesting portability to desktop and mobile apps with Electron
    1. Few resources
    2. By far smallest community by many multiples
    3. Not a con for everyone in this group, many discussions in Chinese
    4. Risks with Vue for large projects because it was designed for prototypes
    5. Few developers are experienced in managing Vue at scale

    https://medium.com/@TechMagic/reactjs-vs-angular5-vs-vue-js-what-to-choose-in-2018-b91e028fa91d

  2. Marcus Alexander Eagan

    I believe that from my perspective, the most important thing is to quickly provide a modern UI that is safe and helps users meet their needs. All options provide this option.


    Secondarily, I think we need to attract a couple really dedicated contributors.


    Thirdly, it needs a sub-repo to become more manageable.

  3. Marcus Alexander Eagan

    Jan Høydahl and others, I went through the Vue project after I finished with life stuff and work stuff and I like it a lot. I also reached out to the author. There are two things I think might be helpful to get to a place where I (personally) would better understand the YASA project better as an option:

    1. We need to work through all the dependencies because I have seen some that are not necessarily the standard or maybe seem out of date. If I run the npm tools I get warnings and things and one of them is for ace-code-editor. Using this library as an example to nit over, we probably want to use monaco, the Microsoft package. It's kept up to date and powers many important IDEs/text editors. It will be some work to go through each dependency in the package.json  and remove them and refactor the components. It's templating system is more similar to  the existing Angular one, and simpler than the JSX JavaScript-XML portmanteau. On the bright side, this dependency clean up work probably would have a longer shelf life than with the other frameworks because Vue doesn't change that fast. There is a risk that some packages will need to be fixed upstream if we find something really hairy one day. I suppose that's always a risk in open source.

    2. The Vue project, while the furthest along, does not have any tests afaik. Even the scaffolding tests in the other projects are good and a big jump forward because they verify that a component compiles after changes. We can probably get a lot of the equivalent with YASA quickly , but I would recommend someone or I write a couple to understand what the complexity is there. It doesn't seem to be too high. Here is a great starter resource.



    From my list of pros and cons above I want to call out two cons I listed.

    1st, a React con:

    Un-opinionated which can grow unwieldy for open source

    In my limited experience with Solr, adding an un-opinionated way of doing things to the open source project could be a risk. If there were more leadership and focused stewardship on this element of the project, I would feel comfortable with React. Without it, I think it's going to be more trouble than it's worth and actually lead to problems eventually. I have worked on and/or led React projects at different points in my life, and sadly they benefited from autocrats.


    2nd, a Vue con:


    Not a con for everyone in this group, many discussions in Chinese

    While I first listed as a con, this could almost be considered a pro for the overall Solr project. In China, and other places in Asia where Chinese is widely spoken, Solr has a robust user community. I don't know if the big tech companies using Sorl in the past still are, but I know there are a number of companies in the Chinese-speaking region using Solr. I have interacted with some of them. Invigorating a pool of any # of developers > 0 in the region seems like a great opportunity to me. There are other areas in which Solr could improve for its users in the Chinese-speaking region and I like any additional entry points. Most importantly, the charge that a lot of the documentation out there is in Chinese mostly speaks to early on, Google Translate has gotten a lot better if you really need to know what's going on, and I need to improve my Chinese quite a bit. 


    Tomorrow I will try to get the application running. Ran into a few issues tonight on the Solr side. Will need to dig in with more clarity/time.



  4. Marcus Alexander Eagan

    The front end engineers I know consistently say that React has a big lead and that most front end engineers know and prefer React. Without governance, it cannot be an option. 

  5. Jan Høydahl

    I saw the YASA developer just reached out on the dev list, willing to help. That is another PLUS for YASA/Vue (smile) 

    I think the next steps on this SIP would be to rewrite most of the text, especially the "Proposed Changes" section. Instead of discussing details, test strategies and exactly how the development environment will be, focus on short sentences of what we propose. Phrase it so that people easily can understand and vote over it. Under "Rejected Alternatives", we can perhaps add sentence for each rejected alternative.

    Here are replacement texts for the SIP, reflecting my current personal opinion. Feel free to use (smile):

    Proposed Changes

    After fully implementing this SIP, Solr will ship with a completely rewritten Admin UI, based on modern frontend development in the Vue.js framework.

    These are the major parts of the proposal

    1. Solr's binary distribution will include the Admin UI out of the box without additional downloads, as today. The UI can be disbled with a System property.
    2. The new Admin UI source code and deveopment tools will not live in solr git repo, but in its separate git repo. This makes a cleaner experience for both Java development (main repo) and UI development (ui repo)
    3. The new UI will be a build-time dependency of Solr, where gradle downloads and unpacks the precompiled UI distribution for inclusion in the Solr binary distro. Solr can ship with the same version of UI for several releases if needed.
    4. The Admin UI source code will live in a separate git repo, generating its own precompiled releases. A benefit is that UI can easily be upgraded or patched without a new Solr release.
    5. The UI will be based on YASA (https://github.com/yasa-org/yasa), an existing 3rd party drop-in replacement Admin UI. YASA gives a cleaner layout, new features, and a good platform to continue development
    6. YASA is developed in Vue.js and for that reason Vue.js is chosen as framework. Vue is a modern, popular frontend framework

    The YASA author is an Apache committer and will continue to contribute to the UI and is willing to donate all code to ASF. We propose to do the first couple releases from the yasa-org GitHub repository. Once the Solr TLP is established we can explore the option of establishing Admin-UI as a sub project in a 'solr-admin' git repository. But until that happens those who wish can join the yasa-org github organization to become comitters.

    See migration plan for the steps planned during the migration.

    Security Considerations

    Security is a main motivation for this SIP. Removing the EOL AngularJS UI is a win for security. The new UI is not expected to add any threat vectors to Solr, and with the ability to disable the UI with a switch, those who don't need a UI will never run the UI. The new UI has the potential to open security threats in Solr if it is not kept up to date. It is far easier to stay on top of JS library upgrades with the npm tooling than the manual procedures in current UI.

    Compatibility, Deprecation, and Migration Plan

    Since the new UI is a complete rewrite, it is not "compatible" with the old UI. So all screenshots in guides, blogs etc will be obsolete, and all old Admin UI URLs will no longer work.

    We plan to deprecate the old Admin UI once the new UI is releases, but keep shipping both with a toggle button until the new UI achieves feature parity. Then the old Admin UI will be deleted.

    In first phase we'll develop the new yasa UI under its current GitHub organization and perform releases as a 3rd party app. At a later stage (not part of this SIP) we may establish a Solr sub project for the Admin UI and move the git repo under ASF control. That means we will also do separate UI releases from the Solr project, downloadable from the mirrors.

    Test Plan

    The new UI will have built-in unit tests, unlike the old UI. The unit tests will assert basic UI functionality.

    There is also an interest to establish end-to-end tests with Solr+UI, run by a separate Jenkins job. We propose setting up a couple of example tests as part of this SIP, so there is something to build upon.

    Rejected Alternatives

    Continuing with current EOL UI was not an option.

    Shipping Solr without any UI was not an option, since we believe Admin UI is such an important part of Solr for our users that it needs to be released with Solr.

    We considered keeping the UI source code in main solr repo, but chose a separate repo because it is cleaner, and developer feedback indicates that both Java and Frontend developers will be happier with dedicated playgrounds. It should thus be easier to attract pure UI contributors when they are not intimidated by all the Java and gradle stuff.

    We considered the proposal of shipping the UI as a Solr package. Decoupling the UI as a build dependency as we propose here is a step in that direction. However, to keep the scope and risk of this SIP down, we propose not to change the distribution method now, but keep the UI as part of the release. Perhaps this can be an option in the future, if/when the package framework gets more such hooks, and a repo for native packages released by the project.

  6. Marcus Alexander Eagan

    This is a fine proposal thanks Jan Høydahl