Goals
- Exchange flow files between two NiFi environments using Site-to-Site via HTTP(S)
Background and strategic fit
Some environments only allow network communication through HTTP(S) port, typically with multi-datacenter deployments. In order to exchange data between NiFi environments using Site-to-Site in such restricted deployments, we should add HTTP(S) as a transport protocol for Site-to-Site.
Assumptions
Requirements
| # | Title | User Story | Importance | Notes |
|---|---|---|---|---|
| 1 | Minimize required network pots to go through Firewall | The target NiFi server only allows access for HTTP/HTTPS. Raw Socket Site-to-Site requires additional port (typically 9990). | Must Have | To minimize required open ports, the new HTTP endpoints are added under /nifi-api/site-to-site, using the same port with the existing NiFi API. |
| 2 | Selectable Transport protocol | A DFM can select transport protocol to use from NiFi Web UI. Available protocols are 'RAW' and 'HTTP'. | Must Have | |
| 3 | Support HTTPS and auth | The network communications can be secured by HTTPS. When to do so, use source NiFi sends its certificate and target NiFi validates if it is registered within a trust store. | Must Have | |
| 4 | Support HTTP Proxy | To reach the target NiFi all communications have to go through a HTTP Proxy server. | Must Have | There is an existing JIRA issue to allow enabling security per Port NIFI-304 , but this proposal doesn't address it, provide security per server basis as Raw Socket does. |
| 5 | Same level of transaction characteristics as RAW Socket | For the flow-files transferred from NiFi-A to NiFi-B, the transaction should be committed on NiFi-A and NiFi-B, only if NiFi-A confirms that NiFi-B received the all sent data intact. Similar for flow-files retrieval operation. Details are described below. | Must Have | |
| 6 | Same level of port availability check as RAW Socket | The availability of data transport should be the same as RAW socket such as followings:
If the target port is not validated, then the peer (a host owning the port) should be penalized for a while to let other peers to be used. | Must Have | |
| 7 | Load balancing | Load balancing capability same as RAW Socket should be provided. The target port which has more data in its queue will receive less than others for sending flow-files, and it will be pulled more often than others for receiving. | Must Have | |
| 8 | Follow target NiFi environment topology change | If target NiFi cluster add/remove nodes and its topology changed, then the source NiFi environment should be able to detect the change automatically, meaning be able to use newly added nodes, or stop sending requests to removed nodes. | Must Have | |
| 9 | Protocol version management | In order to provide backward compatibility in the future, the client and server component should negotiate protocol version, and downgrade its behavior when counter part only supports old version. RAW Socket implementation already has protocol versions from 1 to 5 as of this writing. In order to let HTTP transport protocol version improve independently, yet reuse the existing same logic with Socket impl, this proposal uses two protocol versions, 'transport protocol version' and 'transaction protocol version'. | Must Have | Since this is the 1st timing to introduce HTTP Site-to-Site protocol: transport protocol ver: 1 transaction protocol ver: 5 |
| 10 | Batch up file transport | Must Have | ||
| 11 | Compression | Must Have | ||
| 12 | Cluster aware endpoints | Must Have |
User interaction and design
REST endpoints
Following REST endpoints will be added by this proposal:
- /site-to-site/
- GET: Returns required information of Site-to-Site for the source NiFi environment. Representing Controller of target NiFi environment.
- /site-to-site/peers/
- GET: Returns available peers of this NiFi environment.
- /site-to-site/input-ports/{portId}/transactions/
- POST: Initiate new transaction to send data from source to target NiFi. A new transaction id is published and returned.
- /site-to-site/input-ports/{portId}/transactions/{transactionId}
- POST: Transfer data from source to target NiFi. The transaction will be held on server side instead of commit it immediately, in order to provide 2-phase style commit.
- DELETE: Commit the transaction which is held on server side.
- /site-to-site/output-ports/{portId}/transactions/
- POST:
- /site-to-site/output-ports/{portId}/transactions/{transactionId}
- GET: Initiate new transaction to receive data from target to source NiFi. A new transaction id is published and returned.
- DELETE: Commit the transaction which is held on server side.
Questions
Below is a list of questions to be addressed as a result of this requirements document:
| Question | Outcome |
|---|---|