...
Modified segment completion protocol:
There are 2 variations in the LLC segment completion protocol prescribed in this design:
1) Choose a commit leader
...
- This will only work when we want to flush based on # rows within our input streams. This strategy will not work with time based flush approach. This might be OK since even if the current segment isn’t flushed, the in-memory state can still be queried by the broker.
- One edge case here is if the segment is not flushed for a long time (low QPS input stream). In such a situation, if we restart the servers then we can lose the latest data (since the last committed segment). One way to get around this is to do periodic checkpointing of server’s in-memory state onto local disk (even if the segment is not committed yet).
Note: This modification is optional. We can keep the current approach of choosing a commit leader and persist the corresponding segment as described below.
2) Persist a segment
Currently the ‘commit-leader’ makes a synchronous ‘commit’ call to the Pinot Controller to upload the segment being committed to a deep storage system. If this upload succeeds then the segment is deemed as committed and can be brought online.
...