"My company has a xyz feature for CloudStack. Can someone help me test it?"
This page is a proposal for creating a unified design for QA testing and automation. It should be looked at as a blueprint to follow for a QA set-up (think QACloud like DevCloud)
Any comments / suggestions will be much appreciated.
It is important to note that keeping it simple and easy to replicate is one of the main goals of this exercise. Thus, it may not cater to every possible use case.
The code is based on Prasanna's effort on similar lines (big thanks to Prasanna!), and in future should enable the creation of test-beds for unified testing.
To ensure the master branch is relatively stable almost always, the proposal is to create a "staging" branch where all check-ins shall go. Once the BVTs pass against the staging branch, the commits may be merged with the master branch. Ideally, the commit process should be automated.
Every single set-up would contain
At the heart is a VM that serves as a Jenkins slave, and contains a set of tools to allow one to run QA tests on a clean environment. These tools, hosted on the VM (lets call it "QACloud"), are :
Tool / Software
Provision baremetal hosts with hypervisor
Cobbler integrates well with Puppet and provides useful helper scripts
Configure management server with mysql etc
DHCP + DNS
Used to provision hosts
Testing infrastructure uses isolated network. Some tasks, like yum install, need internet access
As NFS secondary storage
Packaged systemVM and builtin templates for quicker download
Interact with baremetal machines through IPMI network
Deployment and testing framework for CloudStack
The whole code is Python based making it easier to package as needed in future.
A typical deployment would have the QACloud VM also act as a Jenkins slave for builds and launching test suite
There are two steps in the QA process :
Typically, the setting up of the Cloud is a separate Jenkins job, upon successful run of which we launch a test suite in a different job. Note that, the code contains basic elements for you to set-up a single Jenkins job for both the steps, though it may be very slow (Since BVTs are run sequentially in this case to avoid Marvin issues)
Follow the instructions here to set-up the infrastructure from scratch. This is a more involved task and needs familiarity with the CI tools.
It lists the set of commands / scripts used to set-up Cobbler, Puppet manifests etc. The sample configuration for DNSMASQ is provided and allow one to easily change to suit their needs.
The configuration would look something like the below after things are correctly set-up, after which you should proceed to adding Jenkins jobs as described in next section.
Set-up Jenkins job for infrastructure provisioning. This is a simple jenkins job that packages CloudStack RPMs and launches a tester.py python script. The following configuration may be used :
echo export M2_HOME=/usr/local/apache-maven-3.1.0 echo export M2=/usr/local/apache-maven-3.1.0/bin echo export PATH=/usr/local/apache-maven-3.1.0/bin:$PATH PACKAGE_NAME="CloudStack_Auto-rhel6.3" cd $WORKSPACE rm -rf dist/rpmbuild/* cd packaging/centos63 ./package.sh cd ../.. tempdir=`mktemp -d` mkdir -p "$tempdir" cp dist/rpmbuild/RPMS/x86_64/*.rpm $tempdir/ createrepo $tempdir/mv $tempdir $PACKAGE_NAME tar -cvzf $PACKAGE_NAME.tar.gz $PACKAGE_NAME cd /root/cloud-autodeploypython2.7 /root/cloud-autodeploy/tester.py $WORKSPACE xen
Set up Jenkins job for launching BVT suite (TODO : Insert details from QA here)
Set up Jenkins post build task to launch continuous build and provide Git cherry pick info by adding below to post-build shell script task
python2.7 /root/cloud-autodeploy/cherryPickCommits.py <JenkinsUrl> <JenkinsUser> <JenkinsPassword> python2.7 /root/cloud-autodeploy/kickContinuousBuild.py <JenkinsUrl> <JenkinsUser> <JenkinsPassword> <Jenkins Job Name>
This part relates to the overall goal of keeping the master branch relatively stable, and make cherry-picking of commits an easier process rather than someone monitoring a mailing list.
Thus, all BVTs run against a "staging" branch. It is important to note that "master" may not just be the repository master branch, but a term used to refer to any branch you may want to keep stable so as to create a RC out of.
The flow here is :
Thus, the flow ensures that only clean commits make its way to master branch. This should enable us to go one step further in making master almost always stable.
The obvious question to ask is : how do we ensure commits pass successfully across all hypervisor configuration types?
This goes back to providing a unified design to set-up infrastructure, and tie all configurations together using Jenkins job. At the end of the Jenkins job, the above flow shall be kicked off.
1. Integrate across multiple hypervisor runs
2. Better scheduling using a pool of hosts
4. Providing hooks so one can customize actions to be taken in case of failures.
1. The VM is pretty big. Where do we host it? People can set up using the provided instructions too
2. Have a Hackathon to improve this / set up from scratch?
3. To keep template size manageable, the VM has ~50GB of space for NFS. This may run out quickly. Add instructions to add a new volume to VM to serve as NFS?