PLEASE SAVE YOUR WORK! THIS WIKI IS GOING DOWN SHORTLY FOR AN UPGRADE. WILL BE DOWN FOR APPROX 3 HOURS
Current state: ["Under Discussion"
Discussion thread: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
This KIP is related to KIP-4, KAFKA-1367 and KAFKA-2406. In the future, all the Kafka clients and admin tools will not access zookeeper directly. Therefore KAFKA-1367 was introduced to propagate the ISR change to all the brokers to sync up topic metadata on brokers. The approach is to create a ZK path to notify controller update ISR and send UpdateMetadataRequest to brokers. The issue is during a broker bounce or failure, the amount of ISR change are huge and overwhelm the controller. This KIP is to introduce a new config to batch the ISR update and throttle the ISR propagation rate.
Default value for the config will be set to 5 seconds.
Add a new config IsrPropagateIntervalMs to KafkaConfig. The broker will only propagate ISR change at most every IsrPropagateIntervalMs. The propagation will create a sequential ZK path in zookeeper /isr_change_notification with data of all the partition whose ISR has changed since last propagation.
Compatibility, Deprecation, and Migration Plan
This is a backward compatible change.