This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • Kafka system testcases
Skip to end of metadata
Go to start of metadata

Description of testcases in Kafka system test

Replication Basic
1 Topic, 1 Partition
(Kafka Replication Test Plan C.1) 

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0001

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

None

Sync

-1

0

testcase_0002

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

None

Sync

-1

1

testcase_0003

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

None

Sync

1

1

testcase_0004

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

None

Async

-1

1

testcase_0005

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

None

Async

1

1

testcase_0006

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 10240

None

Sync

-1

1

testcase_0007

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 10240

None

Async

-1

1

testcase_0008

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 10240

None

Sync

1

1

testcase_0009

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 10240

None

Async

1

1

testcase_0010

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 1048576

None

Async

1

1

testcase_0011

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 1048576
Auto Create Topic : true

None

Async

1

1

Replication Basic
2 Topics, 3 Partitions
Kafka Replication Test Plan C.1

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0021

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 102400

None

Sync

-1

0

testcase_0022

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000

None

Sync

1

0

testcase_0023

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000

None

Async

1

1

testcase_0024

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000
Auto Create Topic: true

None

Async

1

1

Replication with Leader Failure
1 Topic, 1 Partition
(Kafka Replication Test Plan C.3)

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0101

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -15

Sync

-1

0

testcase_0102

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -15

Sync

-1

1

testcase_0103

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -15

Sync

1

1

testcase_0104

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -15

Async

-1

1

testcase_0105

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -15

Async

1

1

testcase_0106

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 10240

kill -15

Sync

-1

1

testcase_0107

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 10240

kill -15

Async

-1

1

testcase_0108

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 10240

kill -15

Sync

1

1

testcase_0109

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 10240

kill -15

Async

1

1

testcase_0110

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 1048576

kill -15

Async

1

1

Replication with Leader Failure
1 Topic, 3 Partition
(Kafka Replication Test Plan C.3)

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0111

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

-1

0

testcase_0112

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

-1

0

testcase_0113

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

1

0

testcase_0114

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

1

0

testcase_0115

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

-1

1

testcase_0116

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

-1

1

testcase_0117

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

1

1

testcase_0118

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

1

1

testcase_0119

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400
Auto Create Topic: true

kill -15

Async

1

1

Replication with Leader Failure
2 Topics, 3 Partitions
(Kafka Replication Test Plan C.3)

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0121

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

-1

0

testcase_0122

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000

kill -15

Sync

1

0

testcase_0123

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000

kill -15

Async

1

1

testcase_0124

(set log.index.interval.bytes to be slightly
smaller than message size to force the
indexing to be performed for each message)

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000

kill -15

Sync

-1

0

testcase_0125

(set log.index.interval.bytes to be slightly
smaller than message size to force the
indexing to be performed for each message)

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000

kill -15

Sync

1

0

testcase_0126

(set log.index.interval.bytes to be slightly
smaller than message size to force the
indexing to be performed for each message)

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000

kill -15

Sync

-1

1

testcase_0127

(set log.index.interval.bytes to be slightly
smaller than message size to force the
indexing to be performed for each message)

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000

kill -15

Sync

1

1

testcase_0128

No. of topic : 2
No. of partition : 3
Replica factor : 3
Log segment size : 512000
Auto Create Topic: true

kill -15

Async

-1

1

Replication with Leader Failure
2 Topics, 3 Partitions
Replica Factor 2 in a 3-broker cluster)
(Kafka Replication Test Plan C.3)

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0131

No. of topic : 2
No. of partition : 3
Replica factor : 2
Log segment size : 102400

kill -15

Sync

-1

0

testcase_0132

No. of topic : 2
No. of partition : 3
Replica factor : 2
Log segment size : 512000

kill -15

Sync

1

0

testcase_0133

No. of topic : 2
No. of partition : 3
Replica factor : 2
Log segment size : 512000

kill -15

Async

1

1

testcase_0134

No. of topic : 2
No. of partition : 3
Replica factor : 2
Log segment size : 102400
Auto Create Topic:
true

kill -15

Async

-1

1

Replication with Leader Hard Failure
1 Topic, 3 Partition
(Kafka Replication Test Plan C.3)

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0151

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -9

Sync

-1

0

testcase_0152

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -9

Async

-1

0

testcase_0153

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -9

Sync

1

0

testcase_0154

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -9

Async

1

0

testcase_0155

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -9

Sync

-1

1

testcase_0156

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -9

Async

-1

1

testcase_0157

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -9

Sync

1

1

testcase_0158

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -9

Async

1

1

testcase_159

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400
Auto Create Topic: true

kill -9

Async

-1

1

Replication with Controller Failure
1 Topic, 3 Partition
(Kafka Replication Test Plan C.3)

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0201

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

-1

0

testcase_0202

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

-1

0

testcase_0203

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

1

0

testcase_0204

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

1

0

testcase_0205

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

-1

1

testcase_0206

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

-1

1

testcase_0207

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

1

1

testcase_0208

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

1

1

testcase_0209

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400
Auto Create Topic:
true

kill -15

Async

-1

1

Replication with Follower Failure
1 Topic, 3 Partition
(Kafka Replication Test Plan C.3)

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0251

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

-1

0

testcase_0252

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

-1

0

testcase_0253

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

1

0

testcase_0254

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

1

0

testcase_0255

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

-1

1

testcase_0256

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

-1

1

testcase_0257

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Sync

1

1

testcase_0258

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -15

Async

1

1

testcase_0259

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400
Auto Create Topic:
true

kill -15

Async

-1

1

Replication with Simulated Leader GC Pause
(SIGSTOP => SIGCONT)
1 Topic, 3 Partition
(Kafka Replication Test Plan C.3)

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_0301

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -SIGSTOP
kill -SIGCONT

Sync

-1

0

testcase_0302

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -SIGSTOP
kill -SIGCONT

Async

-1

0

testcase_0303

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -SIGSTOP
kill -SIGCONT

Sync

1

0

testcase_0304

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -SIGSTOP
kill -SIGCONT

Async

1

0

testcase_0305

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -SIGSTOP
kill -SIGCONT

Sync

-1

1

testcase_0306

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -SIGSTOP
kill -SIGCONT

Async

-1

1

testcase_0307

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -SIGSTOP
kill -SIGCONT

Sync

1

1

testcase_0308

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400

kill -SIGSTOP
kill -SIGCONT

Async

1

1

testcase_0309

No. of topic : 1
No. of partition : 3
Replica factor : 3
Log segment size : 102400
Auto Create Topic: true

kill -SIGSTOP
kill -SIGCONT

Async

-1

1

Log Retention Testing (Size)
2 Topics, 2 Partitions
Replica Factor = No. of brokers

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_4001

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 102400

kill -15

Sync

-1

0

testcase_4002

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 102400

kill -15

Sync

1

0

testcase_4003

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 102400

kill -15

Sync

-1

1

testcase_4004

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 102400

kill -15

Sync

1

1

testcase_4005

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 102400

kill -15

Async

-1

0

testcase_4006

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 102400

kill -15

Async

1

0

testcase_4007

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 102400

kill -15

Async

-1

1

testcase_4008

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 102400

kill -15

Async

1

1

Log Retention Testing (Size)
2 Topics, 2 Partitions
Replica Factor < No. of brokers

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_4011

No. of topic : 2
No. of partition : 2
Replica factor : 2
Log segment size : 102400

kill -15

Sync

-1

0

testcase_4012

No. of topic : 2
No. of partition : 2
Replica factor : 2
Log segment size : 102400

kill -15

Sync

1

0

testcase_4013

No. of topic : 2
No. of partition : 2
Replica factor : 2
Log segment size : 102400

kill -15

Sync

-1

1

testcase_4014

No. of topic : 2
No. of partition : 2
Replica factor : 2
Log segment size : 102400

kill -15

Sync

1

1

testcase_4015

No. of topic : 2
No. of partition : 2
Replica factor : 2
Log segment size : 102400

kill -15

Async

-1

0

testcase_4016

No. of topic : 2
No. of partition : 2
Replica factor : 2
Log segment size : 102400

kill -15

Async

1

0

testcase_4017

No. of topic : 2
No. of partition : 2
Replica factor : 2
Log segment size : 102400

kill -15

Async

-1

1

testcase_4018

No. of topic : 2
No. of partition : 2
Replica factor : 2
Log segment size : 102400

kill -15

Async

1

1

Mirror Maker

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_5001

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

None

Sync

-1

0

testcase_5002

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -SIGTERM

Sync

-1

0

testcase_5003

(2 instances of Mirror Maker with bouncing)

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -SIGTERM

Async

-1

1

testcase_5004

(2 instances of Mirror Maker with bouncing)

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -SIGTERM

Async

1

1

testcase_5005

(2 instances of Mirror Maker with bouncing)

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 10240

kill -SIGTERM

Async

-1

1

testcase_5006

(2 instances of Mirror Maker with bouncing)

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 10240

kill -SIGTERM

Async

1

1

Migration Tool

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_9001

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

None

Sync

-1

0

testcase_9003

(2 instances of Migration tool with bouncing)

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -SIGTERM

Async

-1

1

testcase_9004

(2 instances of Migration tool with bouncing)

No. of topic : 1
No. of partition : 1
Replica factor : 3
Log segment size : 10240

kill -SIGTERM

Async

1

1

testcase_9005

(2 instances of Migration tool with bouncing)

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 10240

kill -SIGTERM

Async

-1

1

testcase_9006

(2 instances of Migration tool with bouncing)

No. of topic : 2
No. of partition : 2
Replica factor : 3
Log segment size : 10240

kill -SIGTERM

Async

1

1

Others

 

 

 

 

 

Test Case ID

Test Parameters

Failure

Sync/Async

acks

comp

testcase_9051

No. of topics: 300
No. of partitions: 4
No. of brokers: 8
Replica factor: 3

kill -SIGTERM

Sync

-1

0

testcase_9061

1. Testing "Clean Shutdown" by rolling bouncing brokers

No. of topics: 20
No. of partitions: 2
No. of brokers: 4
Replica factor: 2

kill -SIGTERM

Async

-1

1

testcase_9081

1. Stop one broker long enough (a few min)
to make a lot of log segments truncated
in that replica.
2. Start that broker again to observe its
behaviour in catching up.

No. of topic : 2
No. of partition : 2
Replica factor : 2
Log segment size : 10240

None

Async

-1

1

FAQ

How Do I Run A Quick Sanity Test?

The Sanity test shell script will execute the following 2 test cases:
  * testcase_0001 (basic test without Broker failure)
  * testcase_1 (basic test with Leader failure)

In <kafka_home>/system_test
$ ./run_sanity.sh

How Do I Run A Full Regression Test?

  • To execute System Test:

    <kafka_home>/system_test/ $ python –u –B system_test_runner.py 2>&1 | tee system_test_output_`date +%s`.log
    
    -u : python interpreter argument for unbuffering
    -B : python interpreter argument not to generate binary file
  • Notes
    1. A template for a full regression test with all testcases is provided as system_test/testcase_to_run_all.json. Please copy this file into testcase_to_run.json to execute all test cases.
    2. Otherwise, you may edit system_test/testcase_to_run.json to add / remove test cases to run
    3. Edit system_test/logging.conf to turn ON/OFF System Test debug messages (NOT Kafka code log4j messages)
    4. If you are running the System Test in your local box, there is no need to make any changes to the cluster_config.json
    5. You may change any properties' values inside <xxxx_testsuite>/testcase_xxxx/testcase_xxxx_properties.json to see different behavior. (eg. Changing "bounce_broker": "false" => "true" OR "replica_factor": "3" => "2", etc)
  • No labels