This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • Kafka mirroring (MirrorMaker)

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: concrete .sh invocations instead of generic run-class

...

Setting up a mirror is easy - simply start up the mirror-maker processes after bringing up the target cluster. At minimum, the mirror maker takes one or more consumer configurations, a producer configuration and either a whitelist or a blacklist. You need to point the consumer to the source cluster's ZooKeeper, and the producer to the mirror cluster's ZooKeeper (or use the broker.list parameter).

Code Block
bin/kafka-runmirror-classmaker.sh kafka.tools.MirrorMaker --consumer.config sourceCluster1Consumer.config --consumer.config sourceCluster2Consumer.config --num.streams 2 --producer.config targetClusterProducer.config --whitelist=".*"

...

The consumer offset checker tool is useful to gauge how well your mirror is keeping up with the source cluster. Note that the --zkconnect argument should point to the source cluster's ZooKeeper (DC1 in this scenario). Also, if the topic is not specified, then the tool prints information for all topics under the given consumer group. For example:

Code Block
bin/kafka-runconsumer-classoffset-checker.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect dc1-zookeeper:2181 --topic test-topic
Group           Topic                          Pid Offset          logSize         Lag             Owner
KafkaMirror     test-topic                     0   5               5               0               none
KafkaMirror     test-topic                     1   3               4               1               none
KafkaMirror     test-topic                     2   6               9               3               none