...
Enable best effort segment upload in SplitSegmentCommiter and download segment from peer servers.
Option 1New table config: Add a new optional string field peerSegmentDownloadScheme to the SegmentsValidationAndRetentionConfig in the TableConfig. The value can be http or https.
...
Note this is a table level config. We will test the new download behavior in realtime tables in incremental fashion. Once fully proven, this config can be upgraded to server level config.
Option 2:
...
.
...
During segment completion phase,
...
...
...
Notes and Caveats:
- Both of the options above open the door for further development allowing The new table config also allows server to download offline segments from peers. But one can not simply use peer download of offline segment without check like realtime segments because an offline segment can be refreshed or changed. A race condition can happen for offline segments which can be dangerous. If a segment has been updated with a newer version, and server A and B have old versions. Both of them get notified of the newer version. They may try to fetch the segment and fail, and eventually fetch from each other, and end up thinking that they have the newest version of the segment. (Example given by Subbu Subramaniam)
While there are some ways to fix this (e.g. in the segment update message, send the crc of the new version), we need to vet these well before adopting these.
...