It is fairly easy to set up an unencrypted data flow from MiNiFi c++ to NiFi: https://nifi.apache.org/minifi/getting-started.html.
If you want the data flow to use HTTPS, that is a bit more complicated; below is a description of how to do it. (Tested with NiFi 12.120.1 0-M1 and MiNiFi c++ 0.715.0.)
Step-by-step guide
...
- Install NiFi. When you a access https://
...
- <hostname>:8443/nifi/ the first time in your browser, you will get a "Potential Security Risk Ahead" warning about NiFi's self-signed certificate, but you can click Advanced → Accept the Risk and Continue.
Create a NiFi flow with an Input Port. Double-click the Input Port, and note its ID (1342ea64-018d-1000-a5cf-7bda6830909b in the screenshot.)
- Create a MiNiFi flow with a Remote Processing Group and an Input Port as shown in this example: https://github.com/apache/nifi-minifi-cpp/blob/main/examples/site_to_site_config.yml. Update
Remote Processing Groups/Input Ports/id
andConnections[0]/id
to the Input Port ID from NiFi, andRemote Processing Groups/Input Ports/url
to the NiFi URL you use to access NiFi in your browser.
- Create a self-signed certificate; there are many how-tos on the internet. Below, we'll assume that your generated files are /opt/certs/agent-cert.pem and /opt/certs/agent-key.pem; the CN of the certificate is "my-agent-ID".
Add the following settings to your NiFi installation's
nifi.properties
file:Code Block # Site to Site properties nifi.remote.input.host=localhost nifi.remote.input.secure=true nifi.remote.input.socket.port=7777 nifi.remote.input.http.enabled=true
Note that the port you configure here, 7777 in this example, will be used internally by the site-to-site communication, but in the MiNiFi
config.yml
file, you should use the same NiFi address you use in your browser, NOT this site-to-site port.Export the NiFi certificate from the NiFi trust store, and import the MiNiFi agent certificate into the NiFi trust store:
Code Block keytool -exportcert -alias nifi-cert -rfc -keystore truststore.p12 > /opt/certs/nifi-cert.pem keytool -importcert -alias agent-cert -file /opt/certs/agent-cert.pem -keystore truststore.p12
keytool will prompt you for the truststore password; you can get this from the
nifi.properties
file.Add the agent ID (which is the CN of the agent certificate) as the "Initial User Identity 1" value in the "userGroupProvider"
- Download the latest version of
nifi-toolkit
from https://nifi.apache.org/download.html Unpack the contents:
Code Block tar xzvf nifi-toolkit-1.12.1-bin.tar.gz -C /opt/nifi/
Create a directory for the certificate files:
Code Block mkdir -p /opt/nifi/data/ssl
Determine the hostname that will be used to access NiFi from web browsers and from MiNiFi. Then generate the certificates using tls-toolkit:
Code Block # Set the location of Java; the executable should be ${JAVA_HOME}/bin/java export JAVA_HOME=... # Replace <hostname> with the hostname used to access NiFi /opt/nifi/nifi-toolkit-1.12.1/bin/tls-toolkit.sh standalone -n '<hostname>' -C 'CN=minifi, OU=NIFI' -o /opt/nifi/data/ssl # Unpack the certificate and key from the .p12 bundle # Enter the contents of CN=minifi_OU=NIFI.password as the password when prompted (both times) openssl pkcs12 -in /opt/nifi/data/ssl/CN=minifi_OU=NIFI.p12 -out /opt/nifi/data/ssl/nifi-rest.key -nocerts -nodes openssl pkcs12 -in /opt/nifi/data/ssl/CN=minifi_OU=NIFI.p12 -out /opt/nifi/data/ssl/nifi-rest.crt -clcerts -nokeys
Protect the files by ensuring that only the current user have access to them:
Code Block chmod 755 /opt/nifi/data/ssl chmod 600 /opt/nifi/data/ssl/CN\=minifi_OU\=NIFI.* chmod 700 /opt/nifi/data/ssl/<hostname>
Copy the value of the properties in the
nifi.remote
,nifi.web
,nifi.security
sections from/opt/nifi/data/ssl/<hostname>/nifi.properties
to your NiFi installation'snifi.properties
file, except the for the following two, which should be set like this:Code Block nifi.security.keystore=/opt/nifi/data/ssl/<hostname>/keystore.jks nifi.security.truststore=/opt/nifi/data/ssl/<hostname>/truststore.jks
and add the following settings to your NiFi installation's
nifi.properties
file:Code Block nifi.rest.host=<hostname> nifi.rest.keystorePath=/opt/nifi/data/ssl/CN=minifi_OU=NIFI.p12 nifi.rest.keystorePassword=/opt/nifi/data/ssl/CN=minifi_OU=NIFI.password nifi.rest.keystoreType=PKCS12
Uncomment the
file-provider
section ofauthorizers.xml
in your NiFi installation, and set the Initial Admin Identity:Code Block <authorizer><userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileAuthorizer<FileUserGroupProvider</class> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Users File">./conf/users.xml</property> <property name="Initial AdminUser Identity">CN=minifi, OU=NIFI</property> <property name="Legacy Authorized Users File"></property> 1">my-agent-ID</property> <!-- Provide the identity (typically a DN) of each node when clustered, see above description of Node Identity.</userGroupProvider>
and restart NiFi; it will create a new user in
users.xml
which looks like this:
and restart NiFi; it will create a new user with your Initial Admin Identity name inCode Block <tenants> <groups/> <users> <property<user nameidentifier="Node Identity 1"></property> --9a889e09-6e86-360a-a324-8f3ee341842a" identity="my-agent-ID"/> </authorizer>
users.xml
.users> </tenants>
Add authorizations for your user in
authorizations.xml
in your NiFi installation, copying the user identifier fromusers.xml
:Code Block <authorizations> <!-- generate UUIDs for the policy identifiers --> <policies> <policy identifier="1f6ae57a-08bc-11eb-9242-bf69163fde10" resource="/site-to-site" action="R"> <!-- copy the user identifier from users.xml --> <user identifier="9cd6ec429a889e09-b3da6e86-3a3a360a-8405a324-7264746e0e428f3ee341842a"/> </policy> <policy identifier="282818e0-08bc-11eb-8508-2b51c9d70d42" resource="/site-to-site" action="W"> <user identifier="9cd6ec429a889e09-b3da6e86-3a3a360a-8405a324-7264746e0e428f3ee341842a"/> </policy> <!-- copy the port identifier from Remote Processthe Groups/Input Ports/idPort in NiFi into the minifiresource config.ymlstring --> <policy identifier="f512f796-7afb-4c9f-ab68-b5eaf6d5d0cf" resource="/data-transfer/input-ports/c171f9da-689f-41e2-98c4-9d785c59c306" action="R"> <user identifier="9cd6ec429a889e09-b3da6e86-3a3a360a-8405a324-7264746e0e428f3ee341842a"/> </policy> <policy identifier="b4e836ee-d526-4e16-8bf3-ee1d8fa3d5e6" resource="/data-transfer/input-ports/c171f9da-689f-41e2-98c4-9d785c59c306" action="W"> <user identifier="9cd6ec429a889e09-b3da6e86-3a3a360a-8405a324-7264746e0e428f3ee341842a"/> </policy> <!-- you will also need a pair of policies for resource="/data-transfer/output-ports/..." if you want S2S data transfer from NiFi to MiNiFi -->
...
</policies> </authorizations>
Restart NiFi again, and test that you can connect to it using https:
Code Block # This should give a large HTML response curl -k --key /opt/
...
certs/agent-key.pem --cert /opt/
...
certs/agent-cert.pem https://<hostname>:
...
8443/nifi/ # This should give a JSON response curl -k --key /opt/
...
certs/agent-key.pem --cert /opt/
...
certs/agent-cert.pem https://<hostname>:
...
8443/nifi-api/site-to-site/
For Mac, https://kylo.readthedocs.io/en/v0.10.0/security/ConfigureNiFiWithSSL.html has instructions on how to tell your browser to trust this self-signed certificate. On Firefox/Windows, follow these steps: https://support.globalsign.com/digital-certificates/digital-certificate-installation/install-client-digital-certificate-firefox-windows. On Firefox/Linux, the steps are similar; the View Certificates dialog is at Preferences → Privacy & Security (scroll to the bottom). After you have added the CN=minifi_OU=NIFI.p12
client certificate under Your Certificates (enter the contents of CN=minifi_OU=NIFI.password
when prompted), and you try to access https://<hostname>:9443/nifi/, you will still get a "Potential Security Risk Ahead" warning about the certificate, but now you can click Advanced → Accept the Risk and Continue (you only need to do this the first time).
If everything is OK so far, then configure MiNiFi to use the certificate:
- Set up the certificate in
minifi.properties
Finally, configure MiNiFi to use secure site-to-site:
Code Block nifi.remote.input.secure=true nifi.security.need.ClientAuth=falsetrue nifi.security.client.certificate=/opt/nifi/data/ssl/nifi-rest.crtcerts/agent-cert.pem nifi.security.client.private.key=/opt/nifi/data/ssl/nifi-rest.keycerts/agent-key.pem nifi.security.client.pass.phrase=/opt/nifi/data/ssl/CN=minifi_OU=NIFI.password<key passphrase if any> nifi.security.client.ca.certificate=/opt/nificerts/data/ssl/nifi-cert.pem # These are not needed #nifi.rest.api.user.name= #nifi.rest.api.password=
Update the NiFi address in
config.yml
by changinghttp
tohttps
and changing the port from 8080 to 9443 in the Remote Process Group:Code Block Remote Process Groups: - id: 1ca9d943-0175-1000-2188-4d25f7418459 name: https://<hostname>:9443/nifi/ url: https://<hostname>:9443/nifi/ comment: '' timeout: 30 secs yield period: 10 sec transport protocol: RAW proxy host: '' proxy port: '' proxy user: '' proxy password: '' local network interface: '' Input Ports: - id: c171f9da-689f-41e2-98c4-9d785c59c306 name: c171f9da-689f-41e2-98c4-9d785c59c306 comment: '' max concurrent tasks: 1 use compression: true Output Ports: []
And restart MiNiFi.
Related articles
Content by Label | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
hidden | true |
---|
...
- Restart MiNiFi. It should be sending data to NiFi now. Congratulations!