...
- No synchronous call to an external system
- Failure isolation: Reduce correlation failures between independent systems (eg: Pinot and HDFS)
- Cost efficiency: reduce amount of data duplication
- Operational ease: resulting system should have less moving parts. We could also simplify the handshake between controller and the servers.
The proposed solution is in fact simple - leverage Pinot’s replication mechanism to alleviate the strong dependency on a deep storage system. By making a change to the segment commit protocol, we can remove this requirement. However, please note - the aim is not to completely get rid of the deep storage system but not use the deep storage differently.
...