Test Configurations

This is about test configurations in a pure JUnit setup (the eventual goal).

Goals

• Ideally running the Derby tests should allow multiple configurations to exist concurrently, for example to parallelize test runs or stress test by kicking off multiple runs within the same JVM.
• The default way for running tests must not require any system property or other setup that would make it harder for tests to run elsewhere as standard JUnit tests.
  

Primary Configurations

Derby's JUnit tests will be self-contained for the primary configurations which are:

• embedded
• derby client
  

This means that each test's suite() method will return a suite that runs test fixtures in all the primary configurations as required. This is option B in this derby-dev discussion: http://mail-archives.apache.org/mod_mbox/db-derby-dev/200609.mbox/%3c451C1CDF.7050706@apache.org%3e

Tests can fall into any of the following categories:

• embedded & client - runs fixtures in both configurations, some individual fixtures with a test class may only run in a single configuration due to items not supported in a configuration, bugs in a configuration or no value in running the fixture in that configuration. Typically tests that are aimed at testing JDBC methods or objects (e.g. functional test packages jdbcapi and jdbc4) will run in both configurations so that both drivers are tested.
• embedded only - Typically SQL language tests (e.g. function test package lang) can run only in embedded since they are testing execution of SQL statements within the embedded engine. For example testing of server side routines, DDL, triggers or GRANT/REVOKE would not gain additional test coverage by running with the client driver. SQL language tests that relate to data types most likely will benefit from running with the client as well to ensure testing of transmission of the datatype across the DRDA protocol. Test of the storage sub-system (store package) most likely will not need to run with the client configuration.
• client only - E.g. specific testing of a client data source.
  

Whenever test fixtures are added to a suite using the reference to the class (e.g. suite.addSuite(MyTest.class)) then the order of execution of the fixtures is not defined and may vary across different platforms. Thus this requires that the fixtures be independent of each other, which is a good practice to follow. Remember that the setUp() and tearDown() methods will be run for each fixture. If a test needs ordering among fixtures then adding the fixtures explicitly will preserve the order.

    // adds all fixtures with no defined order of execution
    suite.addSuite(MyTest.class);

    // again, no order defined as MyTest.class is used
    return TestConfiguration.defaultSuite(MyTest.class);

    // adds fixtures that will executed in order testA first, then testB, and last testC
    suite.add(new MyOrderedTest("testA"));
    suite.add(new MyOrderedTest("testB"));
    suite.add(new MyOrderedTest("testC"));

Here are examples of how to write the suite() method for the test class MyTest for various combinations. Any tests that are added into a suite without any client server decorator will run the fixtures only as embedded.

It's also useful to look at actual test classes to see how various other situations are handled.

All test fixtures running as embedded and client

Note this default suite setup includes a CleanDatabaseTestSetup.

public static Test suite() {
   return TestConfiguration.defaultSuite(MyTest.class);
}

All fixtures running as embedded and client but some based upon JDBC level

Note manually added CleanDatabaseTestSetup.

public static Test suite() {
    TestSuite suite = new TestSuite();

    // run as embedded
    suite.addTest(baseSuite());

    // run as client server
    suite.addTest(TestConfiguration.clientServerDecorator(baseSuite()));

    return new CleanDatabaseTestSetup(suite);
}
private static Test baseSuite() {
    TestSuite suite = new TestSuite(MyTest.class)
    if (JDBC.vmSupportsJDBC3()) {
          suite.addTest(new MyTestCase("jdbc3SomeTestCase"));
    }
    return suite;
}

All test fixtures running only as embedded

Note this setup does not include a CleanDatabaseTestSetup.

public static Test suite() {
   return TestConfiguration.embeddedSuite(MyTest.class);
}

All test fixtures running only as client

Note this setup does not include a CleanDatabaseTestSetup.

public static Test suite() {
   return TestConfiguration.clientServerSuite(MyTest.class);
}

Suites

Top level suites have two roles:

• To allow a set of tests to be run, e.g. all JDBC api tests (tests.jdbc._Suite), all SQL language tests (tests.lang._Suite), all tests (suites.All)
• To run a number of tests (or suites) in a secondary configuration, e.g. all language tests with encryption.
  

Top level suites should not contain magic that requires the use of a top-level suite to run an individual test.

System Properties

Derby's JUnit infrastructure classes to run tests may setup various system properties. Tests that set specific system properties are not listed here.

Folders

The tests only use folders below ${user.dir}, thus all paths in the table are relative to ${user.dir}

Default Configuration (embedded)

The default configuration is:

• Embedded driver
• derby.system.home=${user.dir}/system
• database name wombat
• user APP
  

In general tests should not rely on the user name or the database name, this will allow tests to run in multiple configurations without changes.

Derby Client Configuration

Setup by decorator created by TestConfiguration.clientServerDecorator(Test) or TestConfiguration.clientServerSuite(Class) static methods. Inherits configuration setup from current configuration but:

• Uses client driver
• Starts network server in its setUp method, server output in ${user.dir}/logs/serverConsoleOutput.log in append mode.
• Stops network server in its tearDown method.