...
- What happens when the segment upload fails but the preceding metadata commit succeeds ?
- In this case, if a server needs to download the segment, it needs to download from the commit server which has a copy of the data.
- We in fact want to minimize downloading from peer servers: so the segment completion mode should be set as DEFAULT instead of DOWNLOAD.
- In the background, RealtimeValidationManager will fix the upload failure by periodically asks the server to upload missing segments.
- What happens if another server gets a "download" but the committer has not gotten to ONLINE state yet?
- To account for the fact that the metadata commit happens before the segment upload, another server should do retries (with exponential backoff) when downloading.
- The retries with wait can greatly reduce the issues caused by the above race condition.
Dependency changes
RealtimeSegmentRelocator
The class moves completed segments to other servers. After the segment move, we should also update the segmentLocation of the LLC segments.
Implementation Tasks (listed in rough phase order)
...