...
- The commit server sends the segment metadata to the controller with segment uri set to be the configured (if any) deep storage location.
- Note: When the segment uri is set, the segment upload in step (2) has not occurred. Here we use a technique of Presumed Success: it is based on our experiences that the deep storage is up most of the time and thus the following upload step (2) will very likely to succeed. Even the upload is not successful, there are backup download mechanisms (such as download from peer server). To make metadata commit before segment upload has the advantage that if metadata commit failed, there will be no segment data in deep storage.
- If step (1) commit to controller is successful, the server performs best-effort upload of the segment to the deep storage. i.e., the server does not need to wait for the segment load to succeed before proceeding.
- Otherwise, the present commit failure handling kicks in.
The following diagrams show how the new protocol behaves under various settings and failure scenarios.
RealtimeValidationManager changes
...
- What happens when the segment upload fails but the preceding metadata commit succeeds ?
- In this case, if a server needs to download the segment, it needs to download from the commit server which has a copy of the data.
- We in fact want to minimize downloading from peer servers: so the segment completion mode should be set as DEFAULT instead of DOWNLOAD.
- In the background, RealtimeValidationManager will fix the upload failure by periodically asks the server to upload missing segments.
- What happens if another server gets a "download" but the committer has not gotten to ONLINE state yet?
- Server no longer upload segment data. Instead during the commit step, only segment metadata is upload to the controller (to be written into zk).
- Server can submit an asynchronous task to upload the segment build to a configured deep storage.
- When a slow server is asked to download a segment, it will go through the segmentLocation list for the segment. It will first try to download from the deep storage. If not available, it will then download from the servers
- To account for the fact that the metadata commit happens before the segment upload, another server should do retries (with exponential backoff) when downloading.
- The retries with wait can greatly reduce the issues caused by the above race condition.
New interaction diagrams on segment completion protocol
Compared with the existing protocol, the main differences are
- .
- To facilitate download from servers, Pinot servers will have a new REST api for segment download.
Pros:
- Simple to implement
- Minimal changes to the current controller/server FSMs used in the LLC segment protocols: some noticeable changes would be in the controller FSM – the Committer Uploading state now becomes obsolete for obvious reason and should be generalized to a state which reflects the segment is being persisted.
...
Dependency changes
RealtimeSegmentRelocator
The class moves completed segments to other servers. After the segment move, we should also update the segmentLocation of the LLC segments.
...