Status
| dev@asterixdb.a.o thread | (permalink to discuss thread) |
|---|---|
JIRA issue(s) | |
Release target | (release number) |
Members
- Glenn Galvizo
- Mike Carey
Motivation
Currently, we have messages that can be embedded in the last tuple slot of a frame after "data" tuples. While this is sufficient for many scenarios, this requires all operators that receive such a frame to be aware of the message tuple's existence and integrate their special handling into their "nextFrame" logic.
The main point of this new feature is to not alter existing formats and existing code, but still give developers who want to communicate between operators in-band a more complete (though less space-efficient) approach.
Proposed changes
An ideal approach to getting all operators to recognize a new type of frame is to a) extend FrameTupleAccessor and b) do some dependency injection, but this still requires a substantial amount of change to the existing operators.
The approach proposed with this issue involves decorating existing operators and wrapping their "nextFrame" method with our own. To determine if we have a "message frame" or a "data frame", ideally we would to reserve a bit in the frame itself – instead the proposed approach is to "hijack" the last four bytes reserved for the tuple count (i.e. if we have 0xFFFFFFFF, the maximum integer, we'll say that this is a message frame and not a data frame).
Right now, message frames are used in Graphix to get operators to 1) flush their buffers using IFrameWriter#flush() and 2) communicate to the message generator whether or not they wrote any tuples to their output on message receipt. Users that want to leverage this specific functionality for message frames (eagerly flushing their buffer upon receiving a message frame) can use the StateReleaseConnectorDescriptor, StateReleaseOperatorDescriptor, and StateReleaseRuntimeFactory classes.
Compatibility/Migration
No changes are required for existing Hyracks users, and the code path for existing Hyracks / AsterixDB users remains unchanged — except for profiling (where a check for message frames is required before doing operations like counting tuples — see ProfiledFrameWriter).
Testing
- Unit test in PushRuntimeTest to verify the recognition of message frames (and actions upon the receipt of message frames).
- Unit test to verify the on-marker-receipt action for persistent build JOIN.
- E2E tests in Graphix for using message frames to detect a fixed-point.
- Perf tests specifically for message frames are not planned (though, perf tests for the fixed-point operator can be used as a proxy for perf testing the message frame machinery).