Kafka data structures in Zookeeper
Skip to end of metadata
Go to start of metadata

0.8:

1. Topic registration info:
/brokers/topics/[topic] :

2. Partition state info:
/brokers/topics/[topic]/partitions/[partitionId]/state

3. Broker registration info:
/brokers/ids/[brokerId]

4. Controller epoch: 

/controller_epoch -> int (epoch)

5. Controller registration:

/controller -> int (broker id of the controller)

6. Consumer registration:

/consumers/[groupId]/ids/[consumerId]

7. Consumer owner:

/consumers/[groupId]/owners/[topic]/[partitionId] -> string (consumerId)

8. Consumer offset:

/consumers/[groupId]/offsets/[topic]/[partitionId] -> long (offset)

9. Re-assign partitions

/admin/reassign_partitions

10. Preferred replication election

/admin/preferred_replica_election

11. Delete topics
/admin/delete_topics

0.8.1

Topic Configuration

/config/topics/[topic_name]

Example

/config/changes/[config_change_x] -> "topic_name"

Contains the name of the topic that changed.

Labels
  • No labels
  1. Can we cover the singleton entries too? It would be good to have a single document that covers all the zk entries and standardizes those names too.

    Some comments:

    • The partition assignment uses the partition as a string, it should be an int.
    • I suspect leaderAndIsr is not the best name, since if you add a third thing X it needs to be renamed to leaderAndIsrAndX. Maybe /brokers/topics/test/partitions/partition-id/replicaInfo
    • I think underscores in json are more standard than camel case
    • We should figure out whether we are doing paths with underscores, dashes, or camel case
    1. Maybe instead of singleton values, we wrap the value with a simple single-element JSON structure like {"what_this_is": x}

      So for offset management, instead of '/consumers/my-group/offsets/some-topic/0' -> '42' we have '/consumers/my-group/offsets/some-topic/0' -> '{"offset": 42}'

  2. Thanks for the comments. Made another pass:

    • standardized on underscores for both path and values
    • changed leaderAndIsr to state
    • added all paths there including singleton values.

    Things that I didn't change according to the comments:

    • The partition assignment still uses the partition as a string since in JSON, map keys can only be strings.
    • I left all singleton values as they are, instead of wrapping them in JSON. I prefer to keep things simple. If we need to put more values there, we likely need a new path since the original path name may no longer be appropriate.
  3. Perhaps /controllerEpoch should change to /controller_epoch.

  4. We also need to include the zookeeper path format changes for the following -

    1. Reassigned partitions path
    2. Preferred replica election path

  5. Anonymous

    I just wanted to comment on your blog and say I really enjoyed reading your blog here. It was very informative and I also digg the way you write! Keep it up and I'll be back soon to find out more mate.Metal Detectors