The existing system tests consist of a small number of heavily parameterized test suites. As part of KIP-25, it is necessary to have a clear understanding of what functionality is currently present, and what of that functionality needs to be ported.

As of this writing, there are five test suites within system_tests. The replacement proposal is summarized as follows.

 Tracking JIRA:









Written in bash, no longer runs.




Writting in bash, no longer runs. will provide superset


Y provides a set of representative examples



(has 2 test cases) Use new consumer in port



Fairly straight port


See below for more notes on the current state of each existing system test suite.

Parent JIRA of the existing system tests:


Test cases and associated parametrizations:



Brief description:

Written in bash, these tests validate an end-to-end setup which includes two broker clusters, mirror maker, producer and consumer.

Randomly selected source brokers, target brokers, and mirror makers are shut down cleanly while messages are produced in the background.

This test does not run - the calls to the producer performance and mirror maker command line tools have not been updated to match changes on the java side.


don’t port, this is covered in replication and mirror maker test suites


Brief description:

Written in bash, this suite has two wrappers around which produce 2e6 messages, but no regression checks on actual performance numbers. Validates size of log agains offset (relevant in kafka 0.7 when log offsets were specified in bytes). Both tests here no longer function.

Proposal: more than covers this.


Concurrently bounce leader, controller, or follower with hard, clean, or soft “kill” while producing with ProducerPerformance in the background. Validates that produced messages are consumed, and compares checksums of per-topicPartition log segments across replicas.

Note: Mixed in here is another type of test called “log retention” (test cases 4001 through 4018). These tests display some information about smallest offsets of log segments but do no validation.


Drop log retention test cases and port specific subset of replication tests.


3 brokers, 3 topics, 3 partitions, 3 replicas on each topic

min.insync.replicas == 2

ack -1

failures: [leader, follower, controller] X [clean, hard, soft]

One test cases with with ack 1

One test case with compression toggled on


How does message validation work in old system tests?

Loads message ids from producer log and consumer log, checks for messages produced but not consumed (dropped messages). If producer ack == 1, check whether message loss is within acceptable threshold. Otherwise dropped messages fail this validation step.

How does broker validation work in old system tests?

Validate broker log checksum in source and target brokers.

How does log offset checksum validation work?

foreach topicPartition, for each replica:

compute checksum of broker logs

validate that checksums are identical across replicas


in the face of consumer failure (offset_management)

Brief description:

These tests validate tracking of consumer offsets within the __consumer_offsets topic by bouncing console consumers *and* the leaders of (partitions of) the __consumer_offsets topic.


Port something quite similar to this, but use new consumer.

Note that this may break initially while the new consumer is still under heavy development.

What do these tests do?


Set up a Zk and Kafka cluster.

Produce messages to a multiple topics - various partition counts.

Start one or more consumer groups to read various subsets of above topics.

Offsets stored in Kafka “__consumer_offsets” topic. 3 replicas, 2 partitions


stop leaders of __consumer_offsets

bounceConsumers = True/False

(re)start leaders of __consumer_offsets


Verify that there are no duplicate messages or lost messages on any consumer group.

Producer dimensions : mode:sync, acks:-1, comp:0

how does validation work here?

kafka_system_test_utils.validate_data_matched_in_multi_topics_from_single_consumer_producer(self.systemTestEnv, self.testcaseEnv, replicationUtils)

This is the same as other message validation

7001 uses one producer and one consumer

7002 one producer, 2 consumer groups, first with 1 consumer, second with 4 consumers


Brief description:

Propagate data between two broker clusters with one or more mirror maker processes while periodically bouncing mirror maker. Validate messages and broker checksums.

Original JIRA(s):

What do these tests do?


Start up source broker cluster and target broker cluster

Start up producer (using

Start up 1 or 2 mirror maker instances, both with the same source and target clusters.

(consumers are in same consumer group)


Bounce (or not) mirror maker processes.

Consume messages from the target cluster.


Validate messages

How does data message validation work?

Loads message ids from producer log and consumer log, checks for messages produced but not consumed (dropped messages). If producer ack == 1, check whether message loss is within acceptable threshold. Otherwise dropped messages fail this validation step.

Validate broker log checksum:

Same as replication tests


Yes, but add more failure modes.

Just run with ack == -1

  • No labels