Goals

• Introduce the concept of a Parameter Provider, which allows Parameter names and values to be fetched from sources external to NiFi
• Provide a way to specify both Sensitive and Non-Sensitive Parameter Providers in a Parameter Context

Background

Parameter Contexts give NiFi flows powerful configurability and portability.  But what if their parameters could be provided by external values or services?  Stateless NiFi already uses the concept of a Parameter Provider – this could also be applied to Parameter Contexts in NiFi.  An EnvironmentVariableParameterProvider would be a natural fit for a containerized NiFi, and a HashiCorpVaultParameterProvider can be envisioned for providing sensitive values. This feature proposal walks through some of the details required in adding support for integrating Parameter Providers into NiFi's Parameter Contexts.

Basic Concept

Parameter Providers would be introduced at the Controller Settings level as a way to fetch parameters from an external source.  These could be selectable in the Add/Update Parameter Context view under the SETTINGS tab.  Two new drop-down fields would be added on this tab:

• Sensitive Parameter Provider - If configured, allows sensitive parameters for this Parameter Context to be fetched via the Parameter Provider.  If unset, sensitive parameters could be set manually in the PARAMETERS tab as usual.  Sensitive parameters in a Parameter Context may be either user-entered or provided by a Parameter Provider, but not both.  
• Non-Sensitive Parameter Provider - Same behavior as above, but with Non-sensitive parameters.

If a Parameter Provider was previously unset and the user selects one, a warning should be displayed indicating that existing values with that sensitivity will be overwritten.

Configuration

Some Parameter Provider implementations would require configuration themselves in order to function.  Taking the HashiCorpVaultParameterProvider as an example, you can imagine the following configuration properties:

• Vault URI
• Authentication configuration
• TLS configuration, if applicable

Parameter Providers would be configured in the Controller Settings view.  Here are the tabs of this view at the time of this writing:

A fifth tab, "PARAMETER PROVIDERS", could be added, containing a list of Parameter Providers as well as a "+" button to add one.  This would bring up an Add Parameter Provider view, which would be configurable with Properties, much like Controller Services, Reporting Tasks, and other similar components.

Some requirements for Parameter Provider configuration include:

• Configuration Properties for Parameter Providers should not support Parameters themselves, as this could lead to confusing circular dependencies.
• The REPORTING TASK CONTROLLER SERVICES should be renamed to CONTROLLER LEVEL CONTROLLER SERVICES.  This would allow such Controller Services to be used by both Parameter Providers and Reporting Tasks.

External Modification

The underlying technology of certain Parameter Provider implementations may allow parameter values to be changed external to NiFi (for example, if an administrator rotates a password in a HashiCorp Vault server).  This introduces the problem of how to handle external parameter updates in order to protect the integrity of the NiFi flow configuration.  This necessitates the use of a cache for external values, as well as a mechanism for fetching values.

Fetch Parameters Button

In the PARAMETER PROVIDERS list tab, a "Fetch Parameters" button could be available on each Parameter Provider, which would cause it to pull all available parameters and values from the external source, based on the current configuration.  These would be cached in the Parameter Provider as a set of proposed changes, but not actually persisted yet.  Next, a Apply Fetched Parameters view should pop up, which initiates the application of the new values to the flow, similar to when a Parameter Context is updated today:

It would also be possible to have both an APPLY and an APPLY AND FETCH button in the Add Parameter Provider view.  The former would apply the configuration changes and close the dialog, and the latter would apply the changes, close the dialog, and then launch the fetch/apply sequence described above.

Opt-in Auto-refresh

To remove the need for manually refreshing Parameter Providers in order to apply external changes (as in the case of password rotation), an auto-refresh feature could be configured on the Add/Update Parameter Provider view.  This should be disabled by default, and a clear warning should be displayed when enabled, indicating that an updated value would automatically stop and start processors and controller services in the flow.

ClassLoader Isolation

Since different Parameter Provider implementations could have very different dependencies, the NAR pattern should be used to separate their ClassLoaders.  Therefore, you might have a nifi-hashicorp-vault-nar that contains a HashiCorpVaultParameterProvider and its associated dependencies, and a nifi-aws-nar that contains an AWSSecretsManagerParameterProvider.  With this design:

• Parameter Providers would be instantiated using the ServiceLoader pattern
• Their NARs could be discovered using an ExtensionManager
• In the UI, Type and Bundle fields should appear in the Parameter Provider list view (see below for a similar example of the existing Reporting Tasks list view)
• Type and Bundle should also appear in the "Add/Update Parameter Provider" Settings tab (see below for a similar example of the existing Configure Reporting Task Settings tab)
• Parameter Providers should be selectable via a dialog similar to the "Add Processor" view (see below)
• The Type and Bundle of a Parameter Provider should be serialized the flow.xml.gz and downloaded flow definition JSON.

Serialization

Parameter Providers and their configuration would be serialized in the flow.xml.gz, with sensitive property values encrypted.

Because Parameter Provider configuration is very likely system-dependent, they should be serialized much like external Controller Services (ones outside of the downloaded process group):

• Parameter Providers would be serialized in downloaded Flow Definition JSON format only with their identifier, name, type, and bundle, but no configuration.
• References to Parameter Providers within Parameter Contexts would be stored by identifier (see below for a similar example)
• When reconstituting a flow from Flow Definition JSON, the framework would attempt to look up a Parameter Provider by first identifier, and then by name if no matching identifier is found.  If even the name is not found, a new unconfigured Parameter Provider should be created with the serialized name, type, and bundle, and it would have to be manually or programmatically configured.

Here is an example of externalControllerServices being referenced as today:

     "properties": {
        "HTTP Context Map": "09ca4ec5-d271-35bc-9791-c2c3d4b1048b"
      }

  "externalControllerServices": {
    "09ca4ec5-d271-35bc-9791-c2c3d4b1048b": {
      "identifier": "09ca4ec5-d271-35bc-9791-c2c3d4b1048b",
      "name": "StandardHttpContextMap"
    }
  }

Displaying Provided Parameters

While it may seem natural to display parameters provided by the Parameter Providers in the Parameter Contexts, this could needlessly complicate the user's experience.  Rather, these parameters should be implicit and only discoverable via Auto-complete when referencing a parameter.  If a Parameter Context has a Sensitive Parameter Provider selected, the user would only be able to see and create non-sensitive parameters.  Likewise the inverse if only a Non-sensitive Parameter Provider is selected.  If both types of Parameter Provider are selected, the PARAMETERS tab could even be omitted.

This would be the least distracting approach to displaying provided parameters.  Consider the case of a HashiCorpVaultParameterProvider, which may contain some secrets needed by the data flow, but many others that are unrelated.  If all secrets from the Vault instance were fetched as parameters, the user would see a host of parameters in their Parameter Context, only a few of which they needed.

Parameter Name Mapping

Depending on the technology backing the Parameter Provider, parameter names in the external service may not be what is desired inside the data flow.

User-defined Properties

User-defined properties could be added to the configuration of the Parameter Provider.  A user could add a dynamic property like this:

The property name ("SFTP Password" here) would be the Parameter Name used in the data flow, and the property value ("external.password" here) would be the parameter name from the external service.  In this example, the property would be referenced via the #{SFTP Password} notation.  If no mapping property is supplied, the parameter name from the external service would be used.  Any naming conflicts between mapped property names and external service property names could be detected by the Parameter Provider framework and prevented.

Access Policies

At the time of this writing, there are two access policies related to Parameter Contexts: "access parameter contexts" of subtype "view" and "modify".  The "modify" policy would be applicable for the Fetch Parameters operation, since this would potentially update parameters.

Stateless NiFi Parameter Providers

Stateless currently implements a simpler version of a Parameter Provider, which is currently called ParameterProvider.  This implementation allows a provider to return a value, given a Parameter Context name and Parameter Name.  While it achieves a similar purpose to the feature described above, it focuses on providing only the value, and not the entire Parameter.  Therefore, the existing ParameterProvider in Stateless should be renamed to ParameterValueProvider, and the NiFi framework core API will create a new ParameterProvider interface for the feature described above.