Apache CloudStack's DevCloud appliance is used today to deploy a basic zone CloudStack environment. Within this DevCloud appliance we run basic integration tests to ensure that essential functionality isn't broken by the rapid checkin process. This page lists the details on how these workers are setup with our continuous integration system.
The XCP-based DevCloud is capable of deploying a basic zone CloudStack environment. We've patched this appliance with a clone of the incubator-CloudStack ASF repo and our test utilities (marvin, cloudmonkey) and created a 'testcloud' appliance. The following workflow explains the way tests are run within the appliance:
The testcloud appliance contains an init script at /etc/init.d/DevCloud that will start up a testrunner process when the appliance boots. Following this the testrunner will perform the necessary setup to prepare a basic zone CloudStack management server.
The host machine is a Ubuntu server with VirtualBox 4.2 installed with guest additions. VirtualBox performs DHCP for the 'testcloud' workers that it spins up. The testcloud image itself is cloned into multiple test workers (currently 5). A simple scheduler will pick up an idle worker VM on trigger from jenkins master that is polling for commits on the git:repo. Once picked up each worker will perform all the tests and post the results back to the gateway/DHCP which in our case is the host machine running VirtualBox. A jenkins slave runs in headless mode on the host machine and posts results back to jenkins master.
After the testworker has done its job and posted its results back to jenkins we wait for a timeout (1800s) to release it back to the pool of workers. To clean up the image of any remnants from the previous test run the worker is restored to a base snapshot keeping the environment clean for the subsequent run.
When we change mvn instructions for building CloudStack and/or the tests that need to run the testrunner process will fetch the latest version of itself from a github repo running DevCloud-ci. This is useful so we don't change the image of the testcloud appliance when our build process changes.
All the code is hosted on github. Feel free to fork it, play around with it and contribute to making it an effective tool for automating our tests. Comments and Suggestions welcome on the CloudStackemail@example.com lists
The jenkins job is a vanilla job enabled with JUnit reports. E-mail will be enabled to CloudStackfirstname.lastname@example.org
The setup is a single machine with 4 core CPU, 8GB RAM and 500GB of hard disk space. Installed with Ubuntu 12.04 LTS - Server Edition.
Following additional packages are installed: VirtualBox 4.2.4, VirtualBox 4.2.4 - Extension Pack, VirtualBox Guest additions
1. create a jenkins user on this machine that will run a jenkins-slave service which will interact with jenkins master to trigger the test runs and send test reports. you'll need access to add nodes and create jobs on the jenkins CI. once the node is setup and the slave is connected you are ready to schedule jobs.
2. modify the default network in VirtualBox to perform DHCP as follows and enable the DHCP server
3. wget the testcloud appliance to your server. The appliance is pre-seeded with the init script that will launch the test runner and configure CloudStack. import the appliance into VirtualBox to change the default credentials in the script to suit those of your jenkins user.
4. start the VM and login to DevCloud
and change the password in the /etc/init.d/DevCloud init script
and then reinitialize the system-V service
5. stop the VM and export it as an appliance out of which we will create further test workers.
6. import the appliance in to virtual box again but creating as many workers as you need.
7. create a base snapshot that will be restored to after the testworker VM has finished running a test. This is to restore the appliance to its clean state for the next test run
8. your workers are now ready to be used and connected to the jenkins job. The jenkins scheduler script is available on the github repo as ci/scripts/schedulerun.sh
1. Speed up the mvn clean install step with a local nexus proxy/maven cache. Currently this eats up around 5m
2. The logs should continuously write to a remote syslog appender on the virtual box gateway. We shouldn't depend on test worker process to complete each time.
3. Draft a proposal for moving this to builds.a.o
WIP: We intend to do the same for a KVM based DevCloud environment to so as to test any functionality that is essential for advanced zone CloudStack deployments.