...
- Will it be necessary to support multiple keys at once, in the event that a follower worker makes a request to the internal endpoint during a rebalance (in which case the follower and worker would be using different keys)? Is this event even possible?
- The
DistributedHerderclass appears to retry infinitely when failures are encountered in task reconfiguration. If this happens on a separate thread from (or just doesn't block) the rebalance logic (which would be responsible for updating the key used by the herder) then it's possible this is fine. However, if this happens on the same thread as (and effectively blocks) the rebalance logic, then there will be deadlock as the worker will have to successfully complete the request for task reconfiguration before receiving its new key, and it will have to receive its new key before it can successfully complete the request for task reconfiguration.
- The