This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • Extension Repositories (aka Extension Registry) for Dynamically-loaded Extensions

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Requires authentication with a GitHub account.
  • Registering a package requires pointing to a public GitHub repo owned by the given user, with a LICENSE and README. 
  • A name and description are also provided during registration.
  • There is a command line tool to help start new packages and publish them.
  • The web site lets a user search for packages and view details of each package.
  • The details of a package show how to use the given package with various tools such as the spark-shell, sbt, and/or Maven, see example
  • For Maven it shows a dependency and repository snippet that could be added to a pom, and there seems to be a repository.
  • Packages can also be voted on and tagged.

5. Use Cases

NOTE:  Hereafter, we use the term "extension" to mean all types of extension including both NAR packages and Templates, as well as future types not yet defined.  (We need some generic term, and it didn't seem necessary to limit "extension" to just NAR packages.)

We are guided by the following use cases:

...

An ExtensionRepository instance represents a repository serving extension packages, and provides APIs for NiFi to access such repositories in a standard way.  It can present is recommended that there always be at least one trusted ExtensionRepository, designated the System Repository, installed as part of the NiFi deployment, and serving the set of standard NiFi extensions published as part of NiFi releases.  ExtensionRepository implementations provide the following functionality:

  • Present metadata about some or all of the extension packages in the repo, to facilitate human-interactive discovery

...

  • .
  • Manage information about the security signature of extension packages, and only

...

  • deliver packages that are properly signed, by a signing authority recognized and approved at the system level. 
  • Manage information about the dependencies of the extension, which are declared in metadata, and only deliver packages whose dependencies are either:
    • already available in the system, or System Repository,
    • or are available from the same repository instance.
    • (For dependencies which are not available in the system nor in the same repository, delivery of the specified package will fail until the dependencies are resolved by User action. This avoids recursive calls to repository code, and issues of cross-access between repositories without human approval. Note extensions in the System Repository should not have outside dependencies; the System Repo should have closure.)
  • Deliver individual extension package files, and where possible their dependencies, on demand.

ExtensionRepositories don't know anything about the internals of the extension package, or the packaging itself. Their responsibility ends after delivering, to an agreed location in the local filesystem, a security-approved the concrete package file that can be loaded by some other mechanism appropriate to the extension type.  

Our initial implementation will support as sub-classes:

  • Filesystem directory-based repositories mounted locally on the NiFi server, and
  • Maven repositories enabled in the NiFi server's maven configuration.

...

An ExtensionSpec instance is an envelope holding the metadata needed to support identification, discovery, delivery, and loading of a single extension package and its dependencies. This includes such metadata as extension name, extension type, packaging type, version, repository containing the extension, address or locator information within that repository, and human-readable description, and dependency list.  If more than a couple lines of descriptive text are required to assist human-interactive discovery, the metadata may include a pointer to a supplementary documentation file (see sub-section 'B' below).  When an ExtensionRepository presents metadata about an extension, it constructs an ExtensionSpec instance.  When the GUI presents extension info to the user prior to loading that extension, it obtains that info from an ExtensionSpec instance.  The ExtensionSpec class is sub-classed at need, when different sets of metadata are needed for the purposes of different kinds of extension.  However, many different kinds of extension may share a common ExtensionSpec subclass, being distinguished by the extension type and packaging type member fields.  (In other words, don't expect a different ExtensionSpec subclass for every different kind of extension, as we found no need for this.)

...

In this implementation, the set of supported extension types and ExtensionSpec sub-classes can only be extended via core code changes.  Furthermore, we choose to leave the extension loader code (load and registration process) for each extension type in core code, rather than try to incorporate this highly extension-type-specific code into this project.  This is partly to reduce the complexity of the implementation by not having to refactor lots of existing loader code, and partly because we didn't see the need since adding new extension types will require core code changes anyway.

B. Metadata, Descriptive Documentation, and Security Signatures

For both the filesystem-based repository and the maven repository, we propose to require that each extension be distributed as a set of 6 files:

  • a single-file package containing the extension itself, as a NAR or other file format appropriate to the extension type
  • a maven POM file for the extension, containing certain mandatory XML elements, from which the extension's metadata is and dependencies are obtained
  • MD5 and SHA1 signature files for each of the package file and POM file.

These requirements mirror the requirements for a normal Maven repository deployment, and are not unreasonably burdensome for a filesystem-based repository deployment. Other  Other repository types may share the same means of managing the metadata and security signature requirements, or they may choose to manage this info in other ways.

Some types of extension may need more than a brief textual description; for instance, a user exploring Templates would benefit from a rendered flow diagram for each Template.  Such info is not appropriate in a POM file, so in addition to the above, we propose, optionally and only when needed:

  • a documentation file (in adoc format), including the signature of the described extension
  • and MD5 and SHA1 signature files for the adoc file

C. Note about Registration as part of Loading

...