...
The client left as is risks imposing heavy load on ZooKeeper. ZooKeeper itself only has a coarse-grained throttling mechanism in place through its zookeeper.globalOutstandingLimit
config which defaults to 1000. This config is insufficient for several reasons:1.
- the limit is meant to protect ZooKeeper from memory pressure associated with a backlog of requests.
...
- the limit is applied across all connections. Even with this config, one misbehaved client will affect the other clients.
We need a throttling mechanism in the client-side to give administrators control over Kafka's impact on ZooKeeper, and for this we propose a new broker config called zookeeper.max.in.flight.requests
.
...