This tutorial will show you how to set up a test cluster and run Kafka system tests on your local machine. I'll assume here that you have a clone of Kafka trunk.

This is broken down into a series of short-ish videos. A summary of the commands run is below each video clip, so you can skip any video clip if you want to.

 


Ensure Vagrant and Virtualbox are installed

We'll need both Vagrant and Virtualbox to create a test cluster

$ # Easy to check with these commands:
$ vagrant --version  # We want vagrant >= 1.6.4
$ vboxmanage --version

To install Vagrant, go here

To install Virtualbox, go here

 


Bootstrap Vagrant for system testing 

$ # This does things like installing necessary Vagrant plugins, creating a Vagrantfile.local, and creating a reusable Vagrant base box
$ tests/bootstrap-test-env.sh

 


Bring up the test cluster

This part is fairly simple, but can take a while, so bring a good book.

$ # If you skipped the previous step, make sure you have a Vagrantfile.local in your Kafka directory
$ cp vagrant/system-test-Vagrantfile.local Vagrantfile.local
$ 
$ # Update Vagrantfile.local, and set num_workers = 10
$ # If you packaged a box,        set base_box = "kafkatest-worker"
$
$ vagrant up # This will bring up your cluster
$
$ # Other useful vagrant commands
$ vagrant status 		# See how many vms you have, whether they're running, etc.	
$ vagrant ssh <vm_name> # ssh into a vm
$ vagrant hostmanager   # update /etc/hosts on each vm
$ vagrant halt 			# shut down without destroying
$ vagrant destroy -f 	# destroy the vms
 

 


Run the tests

Here I'll install ducktape, introduce a few useful ducktape commands, show how to actually run the tests, and give an overview of test output.

$ # Install ducktape:
$ pip install ducktape
$
$ cd tests 
$
$ ducktape --version # Good way to check if ducktape is properly installed
$ ducktape <path> --collect-only  # See which tests ducktape discovers underneath <path>
$
$ # Run "test_console_consumer.py" with the debug flag turned on
$ ducktape kafkatest/sanity_checks/test_console_consumer.py --debug 
$
$ # Run all tests
$ ducktape kafkatest/tests
 

 

 

 

 

 

  • No labels

1 Comment

  1. Are these instructions supposed to work on a VM? I've set up an Ubuntu VM with Vagrant and VirtualBox and have followed these steps. When I try vagrant up I get this error:

     

    Bringing machine 'worker1' up with 'virtualbox' provider...
    Bringing machine 'worker2' up with 'virtualbox' provider...
    Bringing machine 'worker3' up with 'virtualbox' provider...
    Bringing machine 'worker4' up with 'virtualbox' provider...
    Bringing machine 'worker5' up with 'virtualbox' provider...
    Bringing machine 'worker6' up with 'virtualbox' provider...
    Bringing machine 'worker7' up with 'virtualbox' provider...
    Bringing machine 'worker8' up with 'virtualbox' provider...
    Bringing machine 'worker9' up with 'virtualbox' provider...
    Bringing machine 'worker10' up with 'virtualbox' provider...

    ==> worker1: Box 'kafkatest-worker' could not be found. Attempting to find and install...
        worker1: Box Provider: virtualbox

       worker1: Box Version: >= 0

    ==> worker1: Box file was not detected as metadata. Adding it directly...

    ==> worker1: Adding box 'kafkatest-worker' (v0) for provider: virtualbox
        worker1: Downloading: kafkatest-worker

    An error occurred while downloading the remote file. The error
    message, if any, is reproduced below. Please fix this error and try

    again. 

    Couldn't open file /home/ubuntu/kafka/kafkatest-worker