This page will describe how to write new Derby performance tests within the Junit framework that is already available. For more details of the test harness related to Junit testing, please see details on DerbyJunitTesting page.

JDBCPerfTestCase class

As part of DERBY-1970, an initial framework for adding performance tests for Derby has been added. This is

  • org.apache.derbyTesting.junit.JDBCPerfTestCase

The JDBCPerfTestCase extends the functionality provided by TestCase. JDBCPerfTestCase extends BaseJDBCTestCase. The following functionality is added in JDBCPerfTestCase

  • Time the execution of the test fixture
  • Add mechanism to run a test fixture for I iterations
  • Adds mechanism to run a test including setup, test fixture, tearDown for R times.

For a standard JUnit test, the method fixture is run once.

 
{
     setUp run once
     fixture method is run once
     tearDown run once
}

For JDBCPerfTestCase with R repeats and I iterations

This cycle is repeated R times

 
{
    setUp run once
    fixture method is run I times(and timed)
    tearDown is run once
}

Location of performance tests

The base directory for adding performance tests is org/apache/derbyTesting/perf/
For simple individual tests, please add them to org/apache/derbyTesting/perf/basic Depending on what area the performance test is supposed to cover, you can also add subpackages under basic. For e.g, if the performance test is testing some area in lobs, then create a sub directory lobs and add a test there.

For example, see the ValuesTest in org/apache/derbyTesting/perf/basic/jdbc

Discussion on the list about location of performance tests http://www.nabble.com/Location-for-performance-tests.-tf2483748.html#a6927831

Writing a performance Junit Derby test

To write a performance JUnit test that uses JDBC, make your test class extend JDBCPerfTestCase. Add the test fixtures that you want to measure performance for, optionally write your setUp and tearDown methods and finally the suite method that returns a suite of tests to run. In the suite method, add the number of iterations and repeats that you want the to run the test. If you do not specify the iterations and repeats for the test, then by default , the test is run for one iteration and the entire cycle is run once.

For an example, see the tests in org/apache/derbyTesting/perf/basic/jdbc

If a particular test fixture is to be run for I iterations and R repeats, some careful consideration is required to ensure that, there are no unwanted side effects. For example, if the test method tries to insert data into a table with primary key from one range, then it must be able to handle that the particular test fixture will be able to run for I iterations.

Add the suite of tests for the particular test class, also to the _Suite.suite() method in org/apache/derbyTesting/perf/basic

Since JDBCPerfTestCase extends the BaseJDBCTestCase, please use the standard practises for setting up connections, etc as described on the DerbyJunitTesting page. For instance the method getConnection to obtain a connection to the default database, so there is no need for your own test class to have a Connection field.

If you are missing something, ask on derby-dev or create a sub-task/link a Jira issue to DERBY-1970

Running Tests

Running tests using Junit directly

Tests can be run using the Junit 3.8 TestRunners. The classpath needs to include the junit.jar and the Derby classes. Most of Derby's JUnit test cases and suites run successfully using the TestRunners directly, as that is the eventual goal.

Also note that the top level suite org.apache.derbyTesting.functionTests.suites.All does not run the performance suite.

The class name of the test passed to the runners can be an individual test case or one one of the Derby suites, see examples.

(warning) No system properties are required when running directly with TestRunners

batch run

# Single test case
> java -cp '../../tools/java/junit.jar:../../classes' junit.textui.TestRunner
        org.apache.derbyTesting.perf.basic.jdbc.ValuesTest
...........................

# The basic package _Suite
java junit.textui.TestRunner org.apache.derbyTesting.perf.basic._Suite

Adding performance tests to Derby

Goal is to add performance tests to Derby so that anyone in the community can run these tests easily and provide numbers, identify areas for improvement, be encouraged to work on performance issues and increase the quality of derby.

This would gain all the benefits of JUnit, such as running tests from ant, integration with IDEs, ability to hook into other JUnit suites,easier understanding of how Derby tests are run etc.

Features and Improvements for the performance junit framework

Please add new items to the end so that the numbering of items is kept consistent.

  1. Improve reporting for the JDBCPerfTestCase..
    Extend the TestResult, and enhance reporting detail in JDBCPerfTestCase to gather the performance results in xml format for flexibility and ease to generate cool reports using xsl. One idea is to use the java.beans.XMLEncoder to write out XML output to a file.
  2. A way to have a suite run all tests in client/server mode and a suite that runs all tests in embedded mode.
  3. Each test suite runs as part of one jvm run. Need a way to handle running the test as separate runs.
  4. Ensure that the performance tests are not broken, by adding some or parts of the tests to functional tests. Need to analyze the benefit of doing this and the time overhead of adding them to derbyall. Since performance tests are written as junit tests, it would be easy to plug them in for functional tests.
  5. A way to add multi-user performance tests.
  6. Add tools to generate comparison numbers between different build runs.
  • No labels