Testing with Smalltop Benchmarking Framework
Skip to end of metadata
Go to start of metadata

Smalltop - The DConE Benchmarking Framework

How it works

Smalltop is derived from Apache Bigtop, originally designed to test Hadoop.
This framework is used to run performance benchmarks on the Distributed Coordination Engine (DConE). It is capable of:

  1. Deploying all nodes with Puppet
  2. Downloading the desired DConE JAR file for testing
  3. Building and driving tests with Bigtop
  4. Creating test reports

Node deployment with Puppet

Puppet allows us to declare a node configuration state. This is ideal for DConE node deployment because it guarantees that the final configuration state of a node is reached.

Node deployment can be performed by using this command:

  • pdsh -R exec -w ssh:root@10.1.5.[xxx-xxx] puppet apply Smalltop/bigtop-deploy/puppet/manifests/site.pp

"xxx-xxx" are the nodes you want to configure for benchmarking. It is also assumed that these nodes contain the proper installations of Java, Ant, Maven, Puppet and Subversion.

The site.pp and dcone.pp are responsible for DConE node deployment. The configurations perform the following:

  1. Cleans nodes by stopping all Java processes and removing previously created directories from deployment
  2. Creates the necessary directories used for node deployment
  3. Copies and updates application properties files to directories
  4. Performs induction on nodes
  5. Creates and validates membership of nodes
  6. Assigns uuidProposers to begin proposing

Writing tests for Smalltop

Tests can be written in Groovy and saved under test-artifacts/src/groovy/org.apache.bigtop.itest/funny. The test driver method must contain @Test. Other junit test functions can be used such as @BeforeClass and @AfteClass, which execute before and after the test is performed.

Building tests with Bigtop

Testing is divided into two parts, test-artifacts and test-execution. Test-artifacts contain the performance tests that are written in Groovy script. Test-execution provide the instructions of which artifacts to test. Smalltop will automatically build and run all the tests contained in the directories specified in test-execution that contain junit tests (@Test).

The following commands must be used to build the DConE tests:

  • mvn -f bigtop-tests/test-execution/smokes/funny/pom.xml clean
  • mvn -f bigtop-tests/test-artifacts/funny/pom.xml clean install
  • mvn -f bigtop-tests/test-execution/smokes/funny/pom.xml verify -Denforcer.skip=true

In this case, we are kicking off the Funny test, which is the first test written for Smalltop. The TestFunny.groovy file found under test-artifacts/funny contains the driving method that kicks off the test. In the current Funny test, for each node we want to perform benchmarking on, we instantiate DconeThread objects that will consume DConE output data and create reports. This is just one example of a benchmark test.

Test reporting

There are a variety of other classes contained within test-artifacts such as DataCollector.groovy and ReportCreator.groovy. These classes are used for data reporting. For instance, DataCollector, when instantiated, will gather the elapsed time and heap usage while agreements are being performed. DataCollector will return a String, which can then be passed into ReportCreator.groovy to write the string to a data report file.