...
- Refactor existing controller code to run Helix only controller (https://github.com/apache/incubator-pinot/pull/3864).
- Refactor the existing code so that Pinot controller can have a unique interface for all the periodic workloads (https://github.com/apache/incubator-pinot/pull/3264).
- Add logic to create the new resource but disable it in HelixSetupUtils class. The rebalance mode can be set as FULL-AUTO (https://github.com/apache/incubator-pinot/pull/4047).
- Add controller config in Pinot controller side to choose whether to use (i.e. Pinot only mode, Helix only mode or dual mode(default mode)) (https://github.com/apache/incubator-pinot/pull/4323).
- Add logic in controller side to start checking whether new resource is enabled or not. Pinot controller will cache the partition number once it becomes master of the partition. If lead controller resource is yet disabled, controller won’t get any state transition messages (https://github.com/apache/incubator-pinot/pull/4323).
i. When there’s a state transition from Slave to Master for Partition_X: Cache Partition number X in Pinot controller.
ii. When there’s a state transition from Master to Slave for Partition_X: Remove Partition number X from cache in Pinot controller.
iii. When a periodic task is run, or real-time segment completion request is received:
- Add logic in server side to look at new resource if it’s disconnected from Helix controller & new resource is enabled or not. Currently server side logic caches the previous lead controller. With this new feature, the caching logic will still be on, and new checks will happen only when disconnected or we get not_leader message back. Since Pinot server only fetches external view once and will cache the new leader information, it doesn't increase ZK reads by too much.
...