Child pages
  • Leveraging gRPC Processors
Skip to end of metadata
Go to start of metadata

It's possible with the additions of InvokeGRPC (NIFI-4037) and ListenGRPC (NIFI-4038) to leverage gRPC to transport FlowFiles in NIFI. You may ingest, egress, or transport FlowFiles between NIFIs since the gRPC service IDL is the same in each case. It's worth mentioning here that gRPC is designed to transport fairly lightweight messages up to 4MB in size by default. They recommend that if you expect the size of your messages to exceed this, you should seek another solution as this use case was not a primary design tenet.

Step-by-step guide

Ingest

There are two pieces to ingesting FlowFiles via gRPC. An external-to-NIFI gRPC client, which can be generated in 10 different languages from the gRPC FlowFileService IDL, and a ListenGRPC processor running in NIFI. 

  1. Generate the gRPC service code following the gRPC documentation and implement the client service. See the gRPC quick start guide for information on how to do this.

    An example client implementation can be found here. This client just sends a FlowFile with a random ID and "hello world" in the FlowFile content to a remote ListenGRPC processor.

  2. Create a NIFI flow similar to this template:
    generate_and_invoke_grpc.xml 

    Configure the ListenGRPC processor to listen on the port you configured in your client. Enable all of the components on the canvas. Below is an example flow:

Run your client to send one or more FlowFileRequest messages to the processor. You should see the 'Success' relationship triggered on the ListenGRPC processor each time you send a message from your client.

Egress

Similar to ingest, there are also two pieces to egressing FlowFiles via gRPC. An external-to-NIFI gRPC service, which can be generated in 10 different languages from the  gRPC FlowFileService IDL, and an InvokeGRPC processor running in NIFI. 

  1. Generate the gRPC service code following the gRPC documentation and implement the FlowFileService. See the gRPC quick start guide for information on how to do this.

    An example service implementation can be found here. This server listens on a configured port and just logs any messages received from a remote InvokeGRPC processor.

  2. Create a NIFI flow similar to this template:
    listen_and_log_grpc.xml

    Configure the InvokeGRPC processor to send messages to the desired host and the port you configured in your service. Enable, but don't start, each of the components on the canvas. Below is an example flow:

Run your gRPC service to start listening for messages. Run all of the components on your canvas to begin generating FlowFiles and sending them to your service. You should see the 'Original' and 'Response' relationships triggered on the InvokeGRPC processor each time the gRPC service responds successfully to receiving a message.