Child pages
  • KIP-29 - Add an IsrPropagateIntervalMs configuration to KafkaConfig
Skip to end of metadata
Go to start of metadata

Status

Current state["Under Discussion"

Discussion thread: here

JIRA: KAFKA-2406

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP is related to KIP-4, KAFKA-1367 and KAFKA-2406. In the future, all the Kafka clients and admin tools will not access zookeeper directly. Therefore KAFKA-1367 was introduced to propagate the ISR change to all the brokers to sync up topic metadata on brokers. The approach is to create a ZK path to notify controller update ISR and send UpdateMetadataRequest to brokers. The issue is during a broker bounce or failure, the amount of ISR change are huge and overwhelm the controller. This KIP is to introduce a new config to batch the ISR update and throttle the ISR propagation rate.

Public Interfaces

 

Default value for the config will be set to 5 seconds.

Proposed Changes

Add a new config IsrPropagateIntervalMs to KafkaConfig. The broker will only propagate ISR change at most every IsrPropagateIntervalMs. The propagation will create a sequential ZK path in zookeeper /isr_change_notification with data of all the partition whose ISR has changed since last propagation.

Compatibility, Deprecation, and Migration Plan

This is a backward compatible change.

Rejected Alternatives

None

  • No labels