You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
« Previous
Version 3
Next »
ID | IEP-121 |
Author | |
Sponsor | |
Created | |
Status | |
Motivation
Data streamer (IEP-102: Data Streamer) needs more flexibility:
- Transform data on the servers (distributed processing)
- Update more than one table (e.g. normalize data)
Description
Requirements
- Stream arbitrary data into the cluster
- Data items do not need to conform to any schema of any table
- Process the data on the servers with a user-defined receiver
- Receiver can do anything. It may update zero or more tables, use transactions, invoke any other Ignite APIs.
- Return receiver invocation results back to the caller (optionally, one value per data item). Can be used to report processing errors.
- Partition awareness: items should be partitioned using the result of user-provided keyFunction and sent in batches to the corresponding nodes
- Improves table update performance
- Performs load balancing when there are no table updates
- Receiver invocation guarantees: at-least-once. In case of connection issues the receiver can be invoked more than once for the same batch.
Design
API
- Keep existing basic streamer API as is
- Add another overload with receiver
Risks and Assumptions
TBD
Discussion Links
Reference Links
Tickets
Key
|
Summary
|
T
|
Created
|
Updated
|
Due
|
Assignee
|
Reporter
|
Priority
|
Priority
|
Priority
|
Priority
|
P
|
Status
|
Resolution
|