...
The leader will only accept requests signed with the most current key. This should not cause any major problems; workers already engage in an infinite retry loop when requests to forward tasks to the leader fail with a short backoff period in between each retry. If a follower attempts to make a request with an expired key (which should be quite rare and only occur if the request is made by a follower that is not fully caught up to the end of the config topic), this retry behavior can be leveraged almost entirely as-is to keep attempting the request until the updated session key is read by the follower. The only change to this retry behavior will be that a grace period of one minute will be added before emitting error-level log messages on request failures. During this one-minute grace period, if the request is rejected with a 403 response, only a debug-level log message will be emitted. This grace period should leave sufficient room for the follower to read the new session key from the config topic. If longer than that is required, the usual error-level log messages will begin to be generated by the worker.
...
This scenario is unlikely to occur with any normal usage of Connect, but the circumstances leading to it will need to be called out very carefully (in the upgrade notes for any release that begins to support this new behavior) in order to avoid putting the cluster into a bad state and flooding logs with scary-looking error messages. Additionally, it will be vital to design appropriate error messages for this scenario so that users can (hopefully) dig themselves out of that hole on their own.
...