...
| Project / Repo Name | New? | Dependencies | Description |
|---|---|---|---|
nifi-api | new - extracted from nifi | Contains any Java APIs that need to be agreed upon by multiple top-level projects, such as nifi-framework and nifi-extensions | |
nifi-framework | new - extracted from nifi | nifi-api, nifi-maven, nifi-fds, nifi-registry, nifi-commonsstandard-libs | Contains the NiFi web server and front end without any unneeded flow extension bundles such as processors, controller services, and reporting tasks. |
nifi-extensions | new - extracted from nifi | nifi-api, nifi-maven, nifi-standard-commonslibs | Extension bundles for NiFi, such as processors, controller services, and reporting tasks. |
nifi-release | new - extracted from nifi | nifi-framework, nifi-extensions | Will be the new home for the nifi-assembly that produces convenience binaries such as Eventually, this repository will also take over the current role of the nifi-minifi project/repository by providing the |
nifi-toolkit | new - extracted from nifi | nifi-framework, nifi-registry, nifi-commonsstandard-libs | Contains the nifi-toolkit source code and assembly |
nifi-maven | existing | nifi-api | Contains the Maven plugin used for packaging NARs |
nifi-fds | existing | NiFi Flow Design system - reusable front end components used to create consistent front ends, such as the nifi and nifi-registry web UIs | |
nifi-registry | existing | nifi-api, nifi-fds, nifi-commonsstandard-libs | A web service for centralized storage and versioning of NiFi flows and extensions |
nifi-minifi-cpp | existing | A native implementation of libminifi and the MiNiFi agent | |
nifi-standard-commonslibs | new - introduced later | Contains shared code / implementations, such as security provider implementations for authentication and authorization. Note that introducing a nifi-standard-commons libs project/repository is a long-term goal as part of the project structure roadmap and not part of the initial restructuring. The initial decomposition of nifi into new projects will not include any code changes, and therefore extracting shared code out of nifi-framework and nifi-registry into nifi-standard-commons libraries will happen in a future phase. |
...
- Easier for Release Mangers as they are performing source releases of an entire repository.
- Quicker to build an entire repository. Easier to setup reliable, fast CI builds. In a mutlimulti-repo setup, builds will never span multiple projects as commits are, by definition, isolated to a single repo. Also avoids having to "build all project" or setting up CI tools such as Travis or Jenkins to scope builds based on contents of commits.
- Separate repos forces pull requests to stay scoped to a given component, making for overall smaller PRs. Easier to look at a repo and see what work/contributions are still open.
- Easier to understand the history for a single project or module in the git history, e.g., "show me all changes to this project since the last tag / release"
- Lower learning curve for making changes to a single project, as the overall code base is smaller.
...
- Additional documentation and guidance be prepared for developers and contributors, especially getting started guides that target new comers. For example, document the project and module structure to help newcomers navigate repositories.
- Setup a CI job somewhere that publishes a SNAPSHOT build of every project's main branch, and allow development to depend on SNAPSHOT versions of dependencies. This will enable work to continue without releasing stable versions of dependencies. The CI infrastructure and where to host SNAPSHOT artifacts has not been identified. It is possible that GitHub Actions could be setup to run the CI jobs to publish SNAPSHOT artifacts to the Apache Maven repository managed by ASF infra.
...
- One or more community volunteers will manually attempt the above steps, taking detailed notes of commands they use and changes that they need to make in order to get everything working, such as modifications to pom files.
- From those notes, a script will be made that streamlines (ideally, fully automated) the restructuring the nifi repository into new repositories. This will enable us to repeatedly perform the restructuring from any given revision of the main branch.
- The restructuring automation script(s) and instructions for use will be distributed on the Apache NiFi dev@ mailing list. The NiFi dev community will be asked to verify the procedure and resulting output in a manner similar to a RC verification vote. The final output assembly of the restructured process should match the output assembly of the current nifi-assembly module from the 1.x line on the main branch today.
- The automation process will go through as many reviews as needed until it passes a vote by the NiFi PMC.
- A specified date and time will be chosen for the restructuring. Any large branches from main should be merged before that date to avoid burdensome rebasing that will be necessary for any changes not merged before the restructuring.
- On the agreed upon date and time, with the assistance of ASF Infra, the existing nifi repository will be frozen from modification. The restructuring script will be run and the resulting repositories will be added as top level Github repositories, again, with the help of ASF Infra.
- Unmerged pull requests to the nifi repo that were under active development will need to be recreated as PRs to the new repositories. This responsibility will fall to the author of each PR. PRs will stay open to the old nifi repository for a time as a historical record, but only those that are recreated against the new repositories by their authors will be considered to be merged.
- Contributions continue as normal, but from forks of the new repositories.
- The first release after the restructuring will be a release from all repositories/projects in the order dictated by the dependency graph shown above. After that, releases will only need to be performed as needed from each repository/project.
- NOTE: There should be a community discussion on how versioning and releasing of the new, smaller projects should be handled. That is, should we always release everything so that versions stay aligned, even if there are not changes to a particular component such as nifi-api, which changes very infrequently, or should we follow semantic versioning for each new project and allow them to advance and be released independently of each other and only as needed.
Phase 2: nifi-standard-
...
libs
Introduce nifi-standard-commons libs as a home for shared, top-level, reusable libraries of components across the new collection of projects
Steps
- A new nifi-standard-commons libs project and repository is introduced.
- Similar code across nifi-framework and nifi-registry, and possibly other projects such as nifi-toolkit and nifi-extensions, is rewritten as generic, reusable libraries in sub-modules of nifi-commonsstandard-libs, e.g., nifi-standard-commonslibs-security could contain shared authentication and authorization APIs and provider implementations.
- Code from nifi-framework, nif-registry, etc. is removed and replaced with library implementations from nifi-standard-commons libs modules.
Phase 3: nifi-minifi
Migrate code modules from nifi-minifi into nifi-framework, nifi-extensions, and (possibly) nifi-commonsstandard-libs. Migrate the nifi-minifi assembly into nifi-release.
...