...
- Add a new optional boolean field enablePeerSegmentDownload to the SegmentsValidationAndRetentionConfig in the TableConfig.
- Add a new file scheme called SERVERPEER, when SegmentFetcherFactory sees this scheme in the server instance file system config, it will initialize a new kind of segment fetcher PeerServerSegmentFetcher which givens a segment name, handles both server discovery and segment download. The PeerServerSegmentFetcher can be further configured using http or https fetcher for fetching segment from peers.
...
- SplitSegmentCommitter will check this value and if true, behaves exactly like Option 1.
- When Pinot servers in LLRealtimeSegmentDataManager fails to download segments from the segment store during goOnlineFromConsuming() transition, it can use the PeerServerSegmentFetcher (if SERVER scheme PEER scheme is configured) to discover and download the segment from peers.
Notes and Caveats:
- Both of the options above open the door for further development allowing server to download offline segments from peers. But one can not simply use peer download of offline segment without check like realtime segments because an offline segment can be refreshed or changed. A race condition can happen for offline segments which can be dangerous. If a segment has been updated with a newer version, and server A and B have old versions. Both of them get notified of the newer version. They may try to fetch the segment and fail, and eventually fetch from each other, and end up thinking that they have the newest version of the segment. (Example given by Subbu Subramaniam)
While there are some ways to fix this (e.g. in the segment update message, send the crc of the new version), we need to vet these well before adopting these.
Failure cases and handling
...