Goals
- Provider users the ability to perform more powerful verification of a component's configuration
- Allow users the ability to more quickly iterate when creating dataflows by provider quicker feedback about a proposed configuration
Background and strategic fit
NiFI allows the user to configure any number of properties for each component. Once the user has completed all of the configuration for a component (Processor, Reporting Task, Controller Service), the user must then apply those changes.
At that point, the component may be valid or invalid, and it may take a few seconds for the validation to complete. If invalid, the user can hover over the invalid icon to understand the reasons for the validation failure, and then again configure the component and update the necessary configuration.
However, because validation is performed frequently for stopped components, it is not advisable to perform any sort of complex verification of the information given. For example, a Processor that interacts with a cloud service should not create network connections to that cloud service in order to reach out and see if the given credentials are valid. Doing so would be extremely expensive and would generate huge numbers of unnecessary network connections. As a result, the user must first start the component and see if there are any errors.
This Feature Proposal aims to significantly simplify this process.
User interaction and design
When a user opens the configuration dialog for a component, we will provide a new "Verify Config" button (perhaps named simply Verify or Check Config, etc.)
At that point, the UI will fire off a request to the server in order to create a Config Verification Request for the component. The request should provide all of the necessary configuration to verify. The UI should then poll every couple of seconds until that Verification Request is complete, and then delete the Verification Request.
When a Verification Request is submitted, the framework should perform validation of the given component based on the given configuration. I.e., based on the configuration in the user's browser, NOT the configuration that has been applied to the component. If that configuration is invalid, the Configuration Request should be updated to indicate that the config is invalid any why.
If the request is valid, the framework should check the given component to see if it implements a specific interface (i.e., VerifiableProcessor, VerifiableControllerService, VerifiableReportingTask). If so, the framework should invoke the verification method and report back the results of both the component validation and this verification method. The data model should support supplying an explanation that can be helpful about each step that was taken in order to verify the configuration.
In doing this, components are able to provide far more powerful verification of a given configuration - to include ensuring that NiFi is able to connect to the given URL, authenticate with the given config, etc. And this will be done only on demand, which means that we don't have to worry about NiFi making huge numbers of unwanted network connections or resource utilization.
By providing this "Verify Config" button, users are now able to quickly update properties, verify that the results are correct, and if not, update the configuration again. This can be done much more rapidly than the current pace.
Additionally, many times, Processors are configured to make use of FlowFile attributes in the property configuration. Typically, validators must check for this and when EL is used, assume the configuration is valid, since they do not have all of the information necessary.
To account for this, when the user clicks the Verify Config button, the user will need to be able to specify key/value pairs to use for attribute values. Clicking the button should provide the user a table similar to the Properties table for entering user-defined key/value pairs that can be treated as FlowFile attributes by the processor. Additionally, when the configuration for a Processor is retrieved, the names of any attributes explicitly referenced by properties via Expression Language should be provided back to the UI so that the UI can prompt for those values.
Ideally, once some configuration is entered for FlowFile attributes into the table, the user should be able to simply click the "Verify" button again to run the verification with the previously entered values or have another button (perhaps a dropdown on the button) to choose to enter different FlowFile attributes. However, this is not necessarily required in the initial release, just a nice feature to have.
Not Doing
- This Feature Proposal specifically is not intended to be used to provide arbitrary user-invoked functions to be run on components. That might be considered in a future Feature Proposal