...
In Kafka, we currently support the following assignors:
Supported strategy | Feature | Drawback | |
RangeAssignor | Eager | Current default value. The 1st assignor we have | possible to generate heavily skewed assignments when the consumer topic subscriptions are not identical |
RoundRobinAssignor | Eager | Improvement for RangeAssignor to have balanced assignment | Didn't consider the overheads of reassignment |
StickyAssignor | Eager | Improvement for RoundRobinAssignor/RangeAssignor to preserve the existing assignments to reduce some of the overheads of a reassignment | Will have stop-the-world issue when doing rebalance |
CooperativeStickyAssignor | Eager, Cooperative | To be default value in 3.0 as in this KIP described, by having multiple rounds of rebalance to avoid the stop-the-world issue as described in KIP-429 |
As above table showed, since we already introduced a better assignor strategy, we should change the default assignor now.
Public Interfaces
With this KIP, the default value of partition.assignment.strategy
changes from "RangeAssignor"
to "CooperativeStickyAssignor"
...
this will require users to follow the upgrade path laid out in KIP-429 to safely perform a rolling upgrade.
From the user's perspective, the upgrade path of leveraging new protocols is similar to switching to a new assignor. For example, assuming the current version of Kafka consumer is 2.2 and "range" assignor is specified in the config (or no assignor is configured, which is identical as the RangeAssignor is the default below 3.0). The upgrade path would be:
...
The key point behind this two rolling bounce is that, we want to avoid the situation where leader is on old byte-code and only recognize "eager", but due to compatibility would still be able to deserialize the new protocol data from newer versioned members, and hence just go ahead and do the assignment while new versioned members did not revoke their partitions before joining the group. Note the difference with KIP-415 here: since on consumer we do not have the luxury to leverage on list of built-in assignors since it is user-customizable and hence would be black box to the consumer coordinator, we'd need two rolling bounces instead of one rolling bounce to complete the upgrade, whereas Connect only need one rolling bounce.
...