Versions Compared

Old Version 48

changes.mady.by.user Calvin Liu

Saved on

compared with

New Version Current

changes.mady.by.user Calvin Liu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Add the ELR behavior related to the min.insync.replicas config

...

  1. The High Watermark will only advance if all the messages below it have been replicated to at least min.insync.replicas ISR members.

  2. The consumer is affected when consuming the ack=1 and ack=0 messages. When there is only 1 replica(min ISR=2), the HWM advance is blocked, so the incoming ack=0/1 messages are not visible to the consumers. Users can avoid the side effect by updating the min.insync.replicas to 1 for their ack=0/1 topics.

  3. Compared to the current model, the proposed design has availability trade-offs:

    1. If the network partitioning only affects the heartbeats between a follower and the controller, the controller will kick it out of ISR. If losing this replica makes the ISR under min ISR, the HWM advancement will be blocked unnecessarily because we require the ISR to have at least min ISR members. However, it is not a regression compared to the current system at this point. But later when the network partitioning finishes, the current leader will put the follower into the pending ISR(aka "maximum ISR") and continue moving forward while in the proposed world, the leader needs to wait for the controller to ack the ISR change.

    2. Electing a leader from ELR may mean choosing a degraded broker. Degraded means the broker can have a poor performance in replication due to common reasons like networking or disk IO, but it is alive. It can also be the reason why it fails out of ISR in the first place. This is a trade-off between availability and durability.

  4. The unclean leader election will be replaced by the unclean recovery.

  5. For fsync users, the ELR can be beneficial to have more choices when the last known leader is fenced. It is worth mentioning what to expect when ISR and ELR are both empty. We assume fsync users adopt the unclean.leader.election.enable as false.
    1. If the KIP has been fully implemented. During the unclean recovery, the controller will elect a leader when all the LastKnownElr members have replied.
    2. If only the ELR is implemented, the LastKnownLeader is preferred when ELR and ISR are both empty.
  6. Because the ELR behavior is closely bound to the config min.insync.replicas, the following behavior is introduced:
    1. When ELR is enabled, it will add min.insync.replicas config at the cluster-level, with the value of the static config. The removal of min.insync.replicas at the cluster level is disabled.
    2. When ELR is enabled, any existing broker level min.insync.replicas config will be removed. The update of min.insync.replicas at the broker-level is disabled.
    3. If the cluster-level min.insync.replicas is updated, all the ELR states are removed.
    4. If the topic-level min.insync.replicas is updated, the ELR state for this topic is removed.

Ack=0/1 comparison

Comparing the current ISR model with the proposed design

...