Page History

...

• 10th Aug 2016: Switched from a delay-based approach, which uses dedicated throttled fetcher threads, to an inclusion-based approach, which puts throttled and unthrottled replicas in the same request/response
• 25th Sept 2016: Split throttled replica list into two properties. One for leader side. One for follower side.

Motivation

Currently data intensive admin operations like rebalancing partitions, adding a broker, removing a broker or bootstrapping a new machine create an unbounded load on inter-cluster traffic. This affects clients interacting with the cluster when a data movement occurs.

...

The tool, kafka-reassign-partitions.sh, calculates a mapping of topic -> partition-replica for each replica that is either (a) a move origin or (b) a move destination. The union of these are added to the topic level config by the script.. A leader throttle is applied to all existing replicas. A follower throttle is applied to replica that are being created as part of the reassignment process. 

When the tool completes all configs are removed from Zookeeper.  

...

...

Broker-level dynamic config,  unit=B/sWildcard support is also provided for setting a throttle to all replicas:

Wildcard support is also provided for setting a throttle to all replicas:

And And to set a ubiquitous throttle value to all brokers:

...

//Sample configuration for throttled replicas
{
 "version":1,
 "config": {
  "quota.leader.replication.throttled.replicas":"0:0,0:1,0:2,1:0,1:1,1:2"

 }
}

//Sample configuration for throttled replicas
{
 "version":1,
 "config": {
  "quota.follower.replication.throttled.throttled-replicas":"0:0,0:1,0:2,1:0,1:1,1:2"

 }
}

//Sample configuration for replication-quota
{
 "version":1,
 "config": {
  "replication-quota":"1000000"
 }
}

...