This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Page tree
Skip to end of metadata
Go to start of metadata

The new Bigtop integration test framework aims to provide a one stop seamlessly integrated Hadoop ecosystem build and test solution to maximize the user experience . With Integration Test Framework 2.0, everything is under your control at a single entry of Gradle command.

# Build
./gradlew hadoop-pkg-ind
# Deploy
./gradlew docker-provisioner

Experience it by trying with following examples!

Prerequisites

Build

To build Hadoop ecosystem packages, just do:

./gradlew COMPONENT-pkg-ind

Example:

./gradlew hadoop-pkg-ind (default to build on CentOS 7 Docker image)
./gradlew spark-pkg-ind -POS=ubuntu-16.04
./gradlew kafka-pkg-ind -POS=fedor-26

Properties:

-POS=[centos-7|fedora-26|debian-9|ubuntu-16.04|opensuse-42.3] 
-Pprefix=[trunk|1.2.1|1.2.0|1.1.0|...]
-Pnexus

If you'd like to create a repository(yum or apt) at local, just do:

./gradlew repo-ind

Note that the build system does not know which system(yum or apt) you'd like to build. But you can hint it with -POS property.

./gradlew repo-ind (default to build on CentOS 7 Docker image)
./gradlew repo-ind -POS=ubuntu-16.04
./gradlew repo-ind -POS=fedor-26

Typically we'll just do it together along with the build:

./gradlew hadoop-pkg-ind repo-ind -POS=fedora-26

Deploy and Test

To deploy a Bigtop stack inside docker containers using Bigtop Docker Provisioner, just do:

./gradlew docker-provisioner

Example:

Create a 3 node cluster:
  $ ./gradlew docker-provisioner -Pnum_instances=3 
Run smoke tests on 3 node cluster:
  $ ./gradlew docker-provisioner -Pnum_instances=3 -Prun_smoke_tests 
Specify custom configuration file:
  $ ./gradlew docker-provisioner -Pconfig=config_debian-9.yaml 

Properties:

-Pconfig=[CONFIG_FILE] (located under provisioner/docker/)
-Penable_local_repo
-Pimage=[DOCKER_IMAGE] (overwrites -POS and -Pprefix)
-Pnum_instances=[NUM_INSTANCES]
-Pnexus=[NEXUS_URL] (NEXUS_URL is optional)
-POS=[centos-7|fedora-26|debian-9|ubuntu-16.04|opensuse-42.3]
-Pprefix=[trunk|1.2.1|1.2.0|1.1.0|...]
-Prun_smoke_tests (run test components defined in config file)
-Psmoke_tests=[COMPONENTS]
-Pstack=[COMPONENTS]

For detailed explanation, please refer to the helper message:
  $ provisioner/docker/docker-hadoop.sh --help

Typically what we want to do with provisioner is to deploy a cluster and then run smoke tests. The following example deploys Spark on Hadoop and runs Spark smoke test:

./gradlew docker-provisioner -Pconfig=config_ubuntu-16.04.yaml -Pstack="hdfs,yarn,spark" -Psmoke_tests="spark"

To be noticed that, anything not specified is default to what specified in the config file. If no config file specified, default to config_centos-7.yaml

Another typical case is the Build → Deploy → Test lifecycle. You can do it in one command like this:

./gradlew spark-pkg-ind repo-ind docker-provisioner -POS=fedora-26 -Pnum_instances=3 -Penable_local_repo -Pstack="hdfs,yarn,spark" -Psmoke_tests="spark"

Build from branch and commit hash

Another developer friendly feature added is to build packages from a git branch, or even a commit hash. To do so, refer to the document here:

https://github.com/apache/bigtop#for-developers-building-a-component-from-git-repository

  • No labels