Current state: Accepted
Discussion thread: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
This KIP aims to enforce
__consumer_offsets auto topic creation.
Kafka brokers have a config called
offsets.topic.replication.factor that specify the replication factor for the
__consumer_offsets topic. The problem is that this config isn't being enforced upon auto topic creation. If an attempt to auto create the internal topic is made when there are fewer brokers than
offsets.topic.replication.factor, the topic ends up getting created anyway with the current number of live brokers. The current behavior is pretty surprising when you have clients or tooling running as the cluster is getting setup. Even if your cluster ends up being huge, you'll find out much later that
__consumer_offsets was setup with no replication.
The cluster not meeting the
offsets.topic.replication.factor requirement on the internal topic is another way of saying the cluster isn't fully setup yet. The right behavior should be for
offsets.topic.replication.factor to be enforced.
Rationale for the prior behavior can be found in KAFKA-1864 - Revisit defaults for the internal offsets topic Resolved
offsets.topic.replication.factor to 1 in
config/server.properties to maintain existing single-broker quickstart behavior. Note that this does not change the default
offsets.topic.replication.factor value of 3 in
Internal topic creation can happen in five paths:
- By a broker upon
- By a broker from
MetadataRequestquerying the internal topic even if
- By a user when using
- By a user when running
- By a broker through
- By a user directly writing to the topic znode in zookeeper.
Consequences of this KIP:
- will now fail topic creation of the internal topic with
offsets.topic.replication.factorrequirement is met.
- will now fail topic creation of the internal topic and retain existing behavior of failing topic creation with INVALID_REPLICATION_FACTOR until the offsets.topic.replication.factor requirement is met.
- will retain existing behavior.
AdminUtilscompares cluster size vs. replication factor comparison and throws an
InvalidReplicationFactorExceptionif the manually specified replication factor isn't met. If the replication factor is met, it creates the topic, ignoring the broker's
- Same as 3.
- Same as 3.
CreateTopicsResponseincluding an internal topic will return
INVALID_REPLICATION_FACTORif the manually specified replication factor isn't met.
- is unrelated, as the zookeeper write will not receive any error from kafka.
Compatibility, Deprecation, and Migration Plan
This is a bug fix KIP impacting the setup of new clusters. Users setting up a cluster should keep in mind that the
__consumer_offsets topic will not be created until their cluster satisfies the
One rejected alternative was to push
__consumer_offsets topic creation logic out of the brokers and into the
KafkaController. Since the
KafkaController can detect broker membership changes through zookeeper, it can create the
__consumer_offets topic as soon as the
offsets.topic.replication.factor is met. While doable, it is more complicated as it would add even more logic to keep track of in the already complicated
KafkaController and would additionally require
KafkaApis to instead lookup cluster readiness based on its
MetadataCache when responding to