...
- The commit leader becomes the owner of the segment being committed. It will make this segment downloadable using a new HTTP end point. After the leader commits the segment metadata, it uploads the segment to the configured deep storage as a backup. The deep storage uri will be added to the segment location list in controller.
- Similarly other replicas will keep updating this 'segment location' via the Controller.
- When the controller asks a server to download the segment for whatever reason, the server will check the segment location list. It will first try to download it from a deep storage uri to minimize the impact on servers. If there is no such uri, it will go down the list of servers to download the segment.
- All existing error handling routines remain unchanged. The following next section lists diagrams show showing the critical scenarios of (1) happy path, (2) controller failure and (3) slow server downloading from deep storage or Peer server.
b) New interaction diagrams on segment completion protocol:
Compared with the existing protocol, the main differences are
- Server no longer upload segment data. Instead during the commit step, only segment metadata is upload to the controller (to be written into zk).
- Server can submit a asynchronous task to upload the segment build to a configured deep storage.
- When a slow server is asked to download a segment, it will go through the segmentLocation list for the segment. It will first try to download from the deep storage. If not available, it will then download from the servers.
- To facilitate download from servers, Pinot servers will have a new REST api for segment download.
...