Goals
- Exchange flow files between two NiFi environments using Site-to-Site via HTTP(S)
Background and strategic fit
Some environments only allow network communication through HTTP(S) port, typically with multi-datacenter deployments. In order to exchange data between NiFi environments using Site-to-Site in such restricted deployments, we should add HTTP(S) as a transport protocol for Site-to-Site.
Assumptions
Requirements
| # | Title | User Story | Importance | Notes |
|---|---|---|---|---|
| 1 | Minimize required network pots to go through Firewall | The target NiFi server only allows access for HTTP/HTTPS. Raw Socket Site-to-Site requires additional port (typically 9990). | Must Have | To minimize required open ports, the new HTTP endpoints are added under /nifi-api/site-to-site, using the same port with the existing NiFi API. |
| 2 | Selectable Transport protocol | A DFM can select transport protocol to use from NiFi Web UI. Available protocols are 'RAW' and 'HTTP'. | Must Have | |
| 3 | Support HTTPS and auth | The network communications can be secured by HTTPS. When to do so, use source NiFi sends its certificate and target NiFi validates if it is registered within a trust store. | Must Have | |
| 4 | Support HTTP Proxy | To reach the target NiFi all communications have to go through a HTTP Proxy server. | Must Have | |
| 5 | Same level of transaction characteristics as RAW Socket | For the flow-files transferred from NiFi-A to NiFi-B, the transaction should be committed on NiFi-A and NiFi-B, only if NiFi-A confirms that NiFi-B received the all sent data intact. Similar for flow-files retrieval operation. Details are described below. | Must Have | |
| 6 | Same level of port availability check as RAW Socket | The availability of data transport should be the same as RAW socket such as followings:
If the target port is not validated, then the peer (a host owning the port) should be penalized for a while to let other peers to be used. | Must Have | |
| 7 | Follow target NiFi environment topology change | If target NiFi cluster add/remove nodes and its topology changed, then the source NiFi environment should be able to detect the change automatically, meaning be able to use newly added nodes, or stop sending requests to removed nodes. | Must Have |
User interaction and design
Questions
Below is a list of questions to be addressed as a result of this requirements document:
| Question | Outcome |
|---|---|