This section describes the changes made to NiFi 1.0.0. Because NiFi 1.0.0 is a major release, you should review this section carefully, and ensure that you understand the impact that these changes may have on your existing dataflows.
- Minimum JRE/JDK Support
- Kerberos System Properties
- DBCPConnectionPool Service
- MonitorDiskUsage Reporting Task
- Connection/Relationship Default Back Pressure Settings
- Multi-Tenant Authorization Model
- HTTP(S) Site-to-Site
- Zero-Master Clustering
- Secured ZooKeeper
- QueryDatabaseTable Processor
- TailFile Processor
- LDAP referral strategy ‘IGNORE’ bug fix
Minimum JRE/JDK Support
Java 8 is now the minimum JRE/JDK supported.
Kerberos System Properties
SPNEGO and service principals for Kerberos are now established via separate system properties.
New SPNEGO properties:
New service properties:
The Database Driver Jar Url property has been replaced by the Database Driver Location(s) property which accepts a comma-separated list of URLs or local files and folders containing the driver JAR.
Existing processors that reference this service will be invalid until you have configured the new Database Driver Location(s) property.
MonitorDiskUsage Reporting Task
This standard reporting task has been simplified to let you specify a logical name, a directory, and a threshold to monitor. Previously it was tightly coupled to the internal flow file and content repositories in a manner that did not align to the pluggable nature of those repositories.
The new approach gives the user total control over what they want MonitorDiskUsage to monitor.
Connection/Relationship Default Back Pressure Settings
In previous versions, no backpressure settings were supplied by default. In NiFi 1.0.0, new connections made have a default value set of 10,000 flow files and 1GB worth of data size.
Authority Provider model has been replaced by a Multi-tenant Authorization model. Access privileges are now defined by policies that can be applied system-wide or to individual components. Details can be found in the Multi-tenant Authorization section of the System Administrator's Guide.
The system properties nifi.authority.provider.configuration.file and nifi.security.user.authority.provider have been replaced by nifi.authorizer.configuration.file and nifi.security.user.authorizer, respectively. Details on configuration can be found in the Authorizer Configuration section of the System Administrator's Guide.
You can convert NiFi 0.7.0 authorized users and roles to the new authorization model. An existing
authorized-users.xmlfile can be referenced in the
authorizers.xml"Legacy Authorized Users File" property to automatically generate users and authorizations. Details on configuration can be found in the Authorizers.xml Setup section of the System Administrator's Guide.
HTTP(S) protocol is now supported in Site-to-Site as an underlying transport protocol.
HTTP(S) protocol is enabled by default (nifi.remote.input.http.enabled=true). Configuration details can be found in the Site-to-Site Properties section of the System Administrator's Guide.
With both socket and HTTP protocols supported, nifi.remote.input.socket.host has been renamed to nifi.remote.input.host.
nifi.remote.input.secure is now set to false by default.
Master/slave clustering model has been replaced by a Zero-Master Clustering paradigm. Each node in a NiFi cluster performs the same tasks on the data, but each operates on a different set of data. A DataFlow manager can now interact with the NiFi cluster through the UI of any node.
ZooKeeper elects a single node as the Cluster Coordinator and also handles failover. All cluster nodes report heartbeat and status information to the Cluster Coordinator, which is responsible for disconnecting and connecting nodes. Additionally, every cluster has one Primary Node, also elected by ZooKeeper.
Configuration details can be found in the Clustering Configuration, the Cluster Common Properties, the Cluster Node Properties, and the ZooKeeper Properties sections of the System Administrator's Guide.
NiFi Cluster Manager (NCM) configuration and properties are no longer relevant and have been removed.
The following properties should be set on each node:
nifi.cluster.node.address=<fully qualified hostname of the node>
nifi.cluster.node.protocol.port=<node protocol port>
nifi.zookeeper.connect.string=<A comma-separated list of host:port pairs to connect to ZooKeeper. For example, my-zk-server1:2181,my-zk-server2:2181,my-zk-server3:2183>
Coordinated dataflow selection across cluster nodes. During startup, a cluster coordinator is selected at random, and manages the distribution of the dataflow across all nodes. You should set the following to properties, to ensure that the cluster coordinator and other nodes have time to select the correct dataflow:
nifi.cluster.flow.election.max.candidates=<number of NiFi nodes in the cluster>
Embedded ZooKeeper setup
The zookeeper.properties file needs to be populated with a list of each node's embedded ZooKeeper server. The servers are specified in the form of server.1, server.2, to server.n. Each of these servers is configured as <hostname>:<quorum port>[:<leader election port>]. For example, server.1=nifi-node1-hostname:2888:3888.
zookeeper.propertiesfile has a property named dataDir which is set to ./state/zookeeper by default. For each node, create a file named myid and place it in this directory. The contents of this file should be the index of the server as specified by the server.<number>. Configuration details can be found in the Embedded ZooKeeper Server section of the System Administrator's Guide.
State Management – In the
state-management.xmlfile, set the “Connect String” property to the same list of ZooKeeper host:port pairs used for the nifi.zookeeper.connect.string property value.
Secure Clustered Environment – The identities for each node must be specified in the
authorizers.xmlfile. The authorization policies required for the nodes to communicate will then be created during startup. Details on configuration can be found in Authorizers.xml Setup section of the System Administrator's Guide.
The username and password mechanism to provide ZooKeeper authentication is no longer supported. As a result, the “Username” and “Password” properties in the
state-management.xmlfile have been removed.
The “Access Control” property in the
state-management.xmlfile is now set to “Open” by default. It should be changed to “CreatorOnly” when ZooKeeper is secured via Kerberos.
The 'SQL Pre-processing Strategy' property has been replaced by the 'Database Type’ property. This property sets the type of database for generating database-specific code. Property values include ‘Generic' (default) and ‘Oracle’ (for custom SQL clauses).
TailFile originally stored state in a local file, then state management was added in 0.5.0 to support reading in the local state and moving it into the state manager. In 1.0.0, the auto-migration from the old state mechanism has been removed.
If upgrading from a pre-0.5.0 version of NiFi, it is suggested to upgrade to a version greater than or equal to 0.5.0 first, then go to 1.0.0 to not lose state on existing TailFile processors.
Errors occurred if the Referral Strategy was set to ‘IGNORE’ due to a typo in the code to accept ‘INGORE' instead. The
login-identity-providers.xml file should now be configured with the intended value of ‘IGNORE’ for the Referral Strategy property.