...
- Interface and API change
- Add to Pinot Server a new API for segment download (need performance test here on impact server query performance).
- Server changes
- Segment completion protocol: during segment commit, the commit server asynchronously and optionally uploads built segments to a configured external store. It will wait a short amount of time for the upload to finish. Regardless of the upload result, it will move on to send the segment metadata (with the segment uri location if available) to the controller.
- Consuming to online Helix state transition: refactor the server download procedure so that it (1) first tries to download segments from the configured segment store if available; otherwise (2) discovers the servers hosting the segment via external view of the table and then download from the hosting server.
- Controller changes
- Segment completion protocol: controller skips uploading segments – this can be done without controller code change initially because Pinot server can just stop calling LLCSegmentCompletionHandlers.segmentUpload().More details next section
- The changes in (a) implies that now the controller accepts segment commit metadata with empty deep storage uri.
- RealtimeValidationManager
- During segment validation, for any segment without external storage uri, ask one server to upload it to the store and update the segment store uri in Helix.
...