...
- The commit leader becomes the owner of the segment being committed. It will make this segment downloadable using a new HTTP end point. After the leader commits the segment metadata, it
- During segment completion protocol, the committer server uploads the segment to the configured deep storage as a backup. The . It will wait a short period of time (say 5 seconds) and then send the segment metadata to the controller. The wait maximizes the chances that other server can download the segment from deep storage instead of the commit server. The deep storage uri will be added to the segment location list in Zookeeper.
- Similarly other server replicas will keep updating this 'segment location' via the Controller.
- When the controller asks a server to download the segment for whatever reason, the server will check the segment location list. It will first try to download it from a deep storage uri to minimize the impact on servers. If there is no such uri, it will go down the list of servers to download the segment.
- All existing error handling routines remain unchanged. The next section lists diagrams showing the critical scenarios of (1) happy path, (2) controller failure and (3) slow server downloading from deep storage or Peer server.
b) Use the RealtimeValidationManager to fix LLC segments only has a single copy. Currently the validation manager only fix the most recent 2 segments of each partition. With this change, it will scans all segments's Zk data, if it finds a segment does not have a deep storage copy, it will instruct a server to upload the segment to the deepstorage and then updates it segment location url with the deep storage uri.
c) New interaction diagrams on segment completion protocol:
...
- Server no longer upload segment data. Instead during the commit step, only segment metadata is upload to the controller (to be written into zk).
- Server can submit a an asynchronous task to upload the segment build to a configured deep storage.
- When a slow server is asked to download a segment, it will go through the segmentLocation list for the segment. It will first try to download from the deep storage. If not available, it will then download from the servers.
- To facilitate download from servers, Pinot servers will have a new REST api for segment download.
...