Motivation

Data streamer (IEP-102: Data Streamer) needs more flexibility:

• Transform data on the servers (distributed processing)
• Update more than one table (e.g. normalize data)

Description

Requirements

• Stream arbitrary data into the cluster
• Data items do not need to conform to any schema of any table
• Process the data on the servers with a user-defined receiver
• Receiver can do anything. It may update zero or more tables, use transactions, invoke any other Ignite APIs.
• Return receiver invocation results back to the caller (optionally, one value per data item). Can be used to report processing errors.
• Partition awareness: items should be partitioned using the result of user-provided keyFunction and sent in batches to the corresponding nodes
• Improves table update performance
• Performs load balancing when there are no table updates
• Receiver invocation guarantees: at-least-once. In case of connection issues the receiver can be invoked more than once for the same batch.

Design

API

• Keep existing basic streamer API as is
• Add another overload with receiver

Risks and Assumptions

TBD

Discussion Links

• TBD

Reference Links

• PoC: TBD
• IEP-102: Data Streamer

Tickets