...
- To increase or reduce the size of the voter set
- To change quorum membership in the case of failures (e.g. disaster recovery)
- To switch between dedicated deployment and mixed deployment easily, with making controller nodes standalone or integrate with data nodes
Zookeeper itself began supporting dynamic reconfiguration in 3.5, so this is also about reaching parity with the capabilities Kafka has today by leveraging Zookeeper. Finally, dynamic reassignment is also a prerequisite to making Raft-based replication available to local topic partitions.
...
| Code Block |
|---|
{
"type": "data",
"name": "QuorumStateMessage",
"validVersions": "0",
"flexibleVersions": "0+",
"fields": [
{"name": "ClusterId", "type": "string", "versions": "0+"},
{"name": "LeaderId", "type": "int32", "versions": "0+"},
{"name": "LeaderEpoch", "type": "int32", "versions": "0+"},
{"name": "VotedId", "type": "int32", "versions": "0+"},
{"name": "AppliedOffset", "type": "int64", "versions": "0+"},
{"name": "CurrentVoters", "type": "[]Voter", "versions": "0+", "fields": [
{"name": "VoterId", "type": "int32", "versions": "0+"}
]},
{"name": "TargetVoters", "type": "[]Voter", "versions": "0+", "nullableVersions": "0+", "fields": [
{"name": "VoterId", "type": "int32", "versions": "0+"}
]}
]
} |
Discussion: Replication Progress Timeout for Zombie Leader
There's one caveat of the pull-based model: say a new leader has been elected with a new epoch and everyone has learned about it except the old leader (e.g. that leader was not in the voters anymore and hence not receiving the BeginQuorumEpoch as well), then that old leader would not be notified by anyone about the new leader / epoch and become a pure "zombie leader".
To resolve this issue, we will piggy-back on the "quorum.fetch.timeout.ms" config, such that if the leader did not receive Fetch requests from a majority of the quorum for that amount of time, it would start sending Metadata request to random nodes in the cluster to understand the latest quorum. If it couldn't connect to any known voter, the old leader shall reset the connection information and send out DiscoverBrokers. And if the returned response includes a newer epoch leader, this zombie leader would step down and becomes an observer; and if it realized that it is still within the current quorum's voter list, it would start fetching from that leader. Note that the node will remain a leader until it finds that it has been supplanted by another voter.
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.