...
- SplitSegmentCommitter will check this value. If it exists, the segment committer will be able to finish segment commit successfully even if the upload to the segment store fails. The committer will report to the controller that the segment is available in peer servers.
- When Pinot servers in LLRealtimeSegmentDataManager fail to download segments from the segment store during goOnlineFromConsuming() transition, they also check this field's value. If it exists, it can can init a PeerServerSegmentFetcher to
- First discover the segment location server URI u.
- Construct the complete uri using the configured scheme (http or https) and use the appropriate segment fetcher to download it.
...
- Both of the options above open the door for further development allowing server to download offline segments from peers. But one can not simply use peer download of offline segment without check like realtime segments because an offline segment can be refreshed or changed. A race condition can happen for offline segments which can be dangerous. If a segment has been updated with a newer version, and server A and B have old versions. Both of them get notified of the newer version. They may try to fetch the segment and fail, and eventually fetch from each other, and end up thinking that they have the newest version of the segment. (Example given by Subbu Subramaniam)
While there are some ways to fix this (e.g. in the segment update message, send the crc of the new version), we need to vet these well before adopting these.
Failure cases and handling
...