...
1) Choose a commit leader: one of the replicas is "designated" as a segment commit leader. This replica/server is then responsible for "persisting" this segment..
2) Persist the Segment: In the current design, the commit leader sends the newly created segment to the Pinot Controller which in turn uploads it to deep storage using PinotFS API.
...
- Add new fields to the Segment ideal state:
- 'endOffset’: specifies at what Kafka offset the segment is deemed as complete and is ready to become ONLINE.
- Modify "segmentLocation" to be a list of String. This enables multiple owners for the corresponding segment.
- When the Pinot controller creates a new segment, calculate end offset based on the # rows we want to contain within this segment. Note that #rows is simply used as a guidance. It is not necessary for the offsets to be continuous. End offset is a way for all the replicas to agree in a deterministic fashion.
- Modify the server so that it will stop ingesting when it reaches this 'endOffset'.
- The Pinot Server should still call the ‘consumed’ REST API. Each such Pinot server becomes a "commit leader". In other words, in this new model there will be multiple "commit leaders" for each segment.
...