WARN: This is an obsolete design. The design that's implemented in Kafka 0.9.0 is described in this wiki.
This page describes the algorithmic workflow of a lightweight consumer which removes the rebalance logic to the broker-side coordinator and the ZK dependency. It is based on the detailed design of the consumer coordinator described here.
This page is used for people to write the consumer client interface with other languages.
A consumer will have the following key components:
- A coordinator connector which keeps a socket channel with the current coordinator
- A list of fetchers used for sending fetch requests to brokers
And it keeps the following information:
- Current server cluster info, including a list of current brokers with their address and the current coordinator id
- Current consumed partition info, including the topic, broker-partition id, fetched offset, consumed offset, fetch size
Upon startup, a consumer will do the following:
Upon startup, the consumer's coordinator connector will do the following: