Goals

  • Allow users to upload NARs and immediately use components in their flow
  • Allow users to install NARs from external artifact repositories
  • Allow users to discover components that do not yet exist in the current NiFi 
  • Automatically install required NARs when importing a flow

Background and Strategic Fit

There are generally two models to consider for consuming custom NARs, push and pull.

Push Model

The push model refers to a user being able to directly add a NAR to a running NiFi instance.

The primary use case is the development of a new custom NAR which requires the user to iterate on making code changes and testing them on a development NiFi instance. The user is likely developing the NAR on their local machine prior to having added the code to an official source control repository with a release process.

The current capabilities of NiFi allow pushing a NAR to the auto-load directory of each node, but this requires low-level access to the underlying nodes to transfer files, which most users of the web UI would not have.

Pull Model

The pull model refers to pulling NARs into NiFi from external sources. 

This model supports several use cases:

  1. Sharing officially released NARs from established artifact repositories 
  2. Discovering NARs/components that a user may want to add to their NiFi
  3. Automatically resolving missing components during flow import 

The current capabilities of NiFi allow pulling NARs through the ExternalResourceProvider extension point. While this works well for the sharing use case above, the API does not work well for the other use cases. In addition, the resource providers can only be configured through nifi.properties, which requires low-level access to the underlying nodes.

Code Changes

This section describes the improvements to be implemented in order to support the use cases described above, as well as address the limitations of the current capabilities.

NAR Upload

A new REST API for managing NARs will be implemented. The intent is to only manage NARs that are also added through this same API, meaning that you can’t use this API to remove/replace NARs that were provided through one of configured NAR directories in nifi.properties. 

The actual REST APIs are subject to further refinement. A high level specification would be the following:

---------------------------------------------------------------------------------------------------------

Upload

Method: POST 

Path: /controller/nars/upload 

Content: application/octet-stream

Response: Entity containing information about the NAR and it’s contained extensions

---------------------------------------------------------------------------------------------------------

Delete

Method: DELETE

Path: /controller/nars/{group}/{artifact}/{version}?force=(true|false)

Response: Entity containing information about the deleted NAR

The force query parameter indicates if the delete should be allowed even if components are instantiated from the NAR, which if true, would ghost them.

---------------------------------------------------------------------------------------------------------

List

Method: GET

Path: /controller/nars

Response: Listing containing summary information of each NAR

---------------------------------------------------------------------------------------------------------

During upload of NAR the following rules and processes will apply:

  • Upload will replicate the NAR to each node of the cluster
  • Node will reject if missing META-INF/MANIFEST
  • Node will reject if another NAR exists with the same coordinate exists that is not part of the persistence provider (i.e. you can’t upload the same NAR already provided in lib)
  • If accepted, the NAR will be unpacked and loaded the same as a NAR that is dropped in the autol-load directory
  • If the NAR is replacing a NAR that is already uploaded to the persistence provider (i.e. upload 1.0.0-SNAPSHOT again), then the previously loaded NAR will be unloaded prior to loading the new NAR
  • After loading any NAR, the flow will be checked to see any components can be unghosted (i.e. find ghost components with a compatible bundle coordinate and reload them)

During delete of a NAR, the following rules and processes will apply:

  • If force is false, check flow for components instantiated from the NAR, reject the delete if any are found
  • If force is true, find components instantiated from the NAR, ghost the components if any are found, proceed with deletion

NAR Persistence Provider

The NARs uploaded through the above API will be persisted through a new nifi-framework-api component, NarPersistenceProvider. This would be configured through nifi.properties.

The specific methods of this API are subject to further refinement. A high level specification would be the following:

void initialize(NarPersistenceProviderInitializationContext initializationContext);

File createTempFile(InputStream inputStream) throws IOException;

File saveNar(NarPersistenceContext persistenceContext, File tempNarFile) throws IOException;

void deleteNar(BundleCoordinate narCoordinate) throws IOException;

InputStream readNar(BundleCoordinate narCoordinate) throws FileNotFoundException;

boolean exists(BundleCoordinate narCoordinate);

Map<BundleCoordinate, File> getNarFiles();

void shutdown();

Extension Registry Clients

Introduce a new extension point in nifi-api called ExtensionRegistryClient for listing and retrieving NARs from external locations.

These are created and managed through the REST API just like FlowRegistryClients.

The specific methods of this API are subject to further refinement. A high level specification would be the following:

List<Group> getGroups();

List<Artifact> getArtifacts(groupId);

List<ArtifactVersion> getVersions(groupId, artifactId);

InputStream getArtifact(groupId, artifactId, version);

InputStream getExtensionManifest(groupId, artifactId, version);


In addition to the REST API for managing the clients themselves, additional REST APIs will support utilizing a given extension registry. The specific methods of this API are subject to further refinement.

---------------------------------------------------------------------------------------------------------

View information about extensions

Method: GET

Path: /extension-registry/{id}/extensions

Response: Entity containing component manifests for all NARs in the given registry

---------------------------------------------------------------------------------------------------------

Install NAR from extension registry

Method: POST

Path: /extension-registry/{id}/install

Body: Entity specifying information about NARs to install

---------------------------------------------------------------------------------------------------------

In order to provide the information about all extensions that exist in a given registry, NiFi will periodically crawl a given registry and retrieve all extensions manifests from the registry to compute a local “index” of the extension information. This can be done without retrieving all of the actual NARs themselves.

In addition to installing NARs through the above API, the framework code for importing a flow will also leverage extension registry clients to attempt to automatically install any missing NARs that are referenced in the flow definition.

The primary implementations of ExtensionRegistryClient would be:

  1. Nexus - Apache NARs are already released to Maven central and Apache Nexus, and many organizations already publish NARs to a Nexus instance
  2. NiFi Registry - Already supports all of the APIs necessary for this
  3. Object Storage (S3) - Would require following some type of convention in how the storage location is structured in order to support the API 

Security Considerations

  • Uploading and deleting NARs will require WRITE to /controller
  • Listing previously uploaded NARs will require READ to /controller
  • CRUD of an ExtensionRegistryClient will require WRITE to /controller
  • Usage of an ExtensionRegistryClient to install a NAR from an API, or during flow import, will not require any additional permissions, as someone with permissions has already defined the registry client as a trusted source of NARs
  • No labels