This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • KIP-352: Make leader aware of partition reassignment

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. While a reassignment is in progress, the new replicas are trying to catch up. The broker considers these partitions under-replicated even if the desired replication factor is always satisfied. This is misleading and makes URP metrics difficult to alert on.
  2. During a reassignment, users often attempt to throttle the reassignment traffic in order to avoid negatively impacting the cluster. The throttle applies to any replica which is not in sync, but there is no way for the leader to distinguish the replicas which are being reassigned.  If a replica falls out of sync, then suddenly it gets hit with the throttle. Not only does this make the replica slower to catch back up, it increases the amount of traffic which is subject to the throttle which could make the reassignment go even slower.
  3. As replicas are catching up, they are added to the ISR. Depending on the rate that the replicas catch up, there may be a non-trivial amount of time during which the ISR is larger than the desired replication factor. This can negatively impact end-to-end latency since replication must await all members of the ISR. Perhaps even worse, it makes determining the correct throttle even trickier because all ISR traffic skips the throttle.

...