Environment
- CentOS 6 machines running on VM/Docker/Bare Metals, etc
Settings
Hadoop+Spark Stack
hadoop_cluster_node::cluster_components: - hadoop - spark
Bigtop 1.2.0 CentOS 6 repo:
bigtop::bigtop_repo_uri: "http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/centos/6/x86_64"
Check out available OS from Bigtop Installable binary artifacts
Deploy
There's only ONE thing you need to do manually:
- Replace bigtop1.apache.org to a hostname of your master node
The deployment script(execute on EVERY node in your cluster):
# Install Puppet sudo rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm sudo yum -y install puppet sudo puppet module install puppetlabs-stdlib # Install Bigtop Puppet sudo git clone https://github.com/apache/bigtop.git /bigtop-home sudo (cd /bigtop-home; git checkout branch-1.2) sudo cp -r /bigtop-home/bigtop-deploy/puppet/hieradata/ /etc/puppet/ sudo cp /bigtop-home/bigtop-deploy/puppet/hiera.yaml /etc/puppet/ # Configuration sudo su root -c "cat > /etc/puppet/hieradata/site.yaml << EOF --- bigtop::hadoop_head_node: "bigtop1.apache.org" hadoop::hadoop_storage_dirs: - /data/1 - /data/2 hadoop_cluster_node::cluster_components: - hadoop - spark bigtop::jdk_package_name: "java-1.8.0-openjdk-devel.x86_64" bigtop::bigtop_repo_uri: "http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/centos/6/x86_64" EOF " # Deploy sudo puppet apply --parser future --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules /bigtop-home/bigtop-deploy/puppet/manifests
DONE!
To customize your own Big Data Stack:
hadoop_cluster_node::cluster_components: - hadoop - spark - hbase ...