This document is for you to quickly setup CI jobs to ensure the quality of your own customized Bigtop distribution.
Notice that, the document assume you already know well how to use Jenkins, hence the instructions are brief.
If you have questions, feel free to ask on Bigtop mailing list. 
Setup a Jenkins master
If you already have a managed Jenkins/Hudson, or you prefer your all installation, go to Setup Bigtop packages build matrix
To setup a Jenkins master by leveraging Docker, do the following:
# create jenkins user on host machine with uid=1000 to map the jenkins uid inside jenkins image sudo adduser jenkins -u 1000 sudo yum install -y docker git sudo su - jenkins -c "git config --global user.email \"jenkins@bigtop.apache.org\"" sudo su - jenkins -c "git config --global user.name \"jenkins\"" sudo usermod -a -G docker jenkins sudo service docker start sudo su - jenkins -c "docker run -d --name jenkins-master -p 80:8080 -v `pwd`:/var/jenkins_home jenkins"
And the needed Plugin(s):
- git plugin
Setup Bigtop packages build matrix
Create a new job:
New Item -> Multi-configuration project
In Source Code Management section, fill in your repo and the branch, for example:
Repository URL: https://git-wip-us.apache.org/repos/asf/bigtop.gitBranch Specifier: master
In the Configuration Matrix section, add a user defined axis with following name and values:
Name: BUILD_ENVIRONMENTSValues: centos-6 centos-7 fedora-20 ubuntu-14.04 debian-8 opensuse-13.2
Add another user defined axis:
Name: COMPONENTS Values: bigtop-groovy bigtop-jsvc bigtop-tomcat bigtop-utils crunch datafu flume giraph hadoop hama hbase hive hue ignite-hadoop kafka kite mahout oozie phoenix pig solr spark sqoop sqoop2 tachyon tez ycsb zookeeper
Finally, add a shell build step with the following script:
docker run --rm -v `pwd`:/ws --workdir /ws -e COMPONENTS=$COMPONENTS bigtop/slaves:trunk-$BUILD_ENVIRONMENTS \ bash -l -c 'A=rpm ; type -p dpkg && A=deb ; ./gradlew allclean ; ./gradlew -Dbuildnodeps=true $COMPONENTS-$A'
The result will be looked like this:
http://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
However, do aware that full matrix build of packages is time consuming, and they need roughly 50GB disk space for each build.
Setup Bigtop deployment & smoke test job
To setup a deployment and smoke test job for centos 6 OS, use following shell script:
# setup configuration file
cat > bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig_Bigtop-Docker-provisioner-centos-6.yaml <<-__EOT__
docker:
memory_size: "4096"
image: "bigtop/deploy:centos-6"
boot2docker:
memory_size: "4096"
number_cpus: "2"
repo: "http://bigtop-repos.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64"
distro: centos
components: [hadoop, yarn, spark]
namenode_ui_port: "54070"
yarn_ui_port: "8488"
hbase_ui_port: "64010"
enable_local_repo: false
smoke_test_components: [mapreduce, pig, spark]
jdk: "java-1.7.0-openjdk-devel.x86_64"
__EOT__
# destroy previous cluster
./gradlew docker-provisioner-destroy
# provision
./gradlew -Pconfig=vagrantconfig_Bigtop-Docker-provisioner-centos-6.yaml -Pnum_instances=3 -Prun_smoke_tests=true docker-provisioner
# destroy provisinoed cluster
./gradlew docker-provisioner-destroy
The result will be looked like this:
http://ci.bigtop.apache.org/view/Deployment/job/Bigtop-Docker-provisioner-centos-6/
You can replace the repo by your own repo so that your packages can be tested using Bigtop deployment recipes.
The components and smoke_test_components are also configurable, choose what you'd like to deploy and test as what you want.
We're going to switch to docker-compose instead of Vagrant in Bigtop Docker Provisioner so that the Provisioner can serve users w/o the bigtop/deploy image limitation.