This page will describe how to run Derby JUnit tests, and how to write new JUnit tests for Derby. All features of the test harness related to JUnit testing should be described here as well.

Base JUnit test classes

After some discussion on derby-dev, a set of base classes to be used when writing JUnit tests were created. These are:

  • org.apache.derbyTesting.junit.BaseTestCase
  • org.apache.derbyTesting.junit.BaseJDBCTestCase
  • org.apache.derbyTesting.junit.BaseJDBCTestSetup
  • org.apache.derbyTesting.junit.TestConfiguration
  • org.apache.derbyTesting.junit.JDBCClient
  • org.apache.derbyTesting.functionTests.util.JDBC

(info) These classes were moved to their current package from org.apache.derbyTesting.functionTests.util as they can apply to more than functional tests.

(warning) Note that the class org.apache.derbyTesting.functionTests.util.DerbyJUnitTest is deprecated.

To write a JUnit test that uses JDBC, make your test class extend BaseJDBCTestCase. This class is subclass of TestCase in JUnit framework and you can add test methods, optionally write your setUp and tearDown methods and finally the suite method that returns a suite of tests to run (typically all tests in the test class). BaseJDBCTestCase provides a set of often used functionality, please look at what is available before writing your test. For instance the method getConnection to obtain a connection to the default database, so there is no need for your own test class to have a Connection field. If you are missing something, ask on derby-dev or create a sub-task/link a Jira issue to DERBY-1122.

BaseTestCase is to be used for JUnit tests not using the JDBC API.

BaseJDBCTestSetup is the base class for test decorators that use JDBC. Look at the utility methods that this class provides when writing a JDBC test that extends it.

TestConfiguration holds information about the test configuration/environment. It is responsible for parsing information passed along from the test harness.

JDBCClient is an enumeration of valid JDBC clients the test harness can be run with.

JDBC holds test utility methods related to JDBC objects.

Remember that the methods setUp and tearDown are run before and after each test method (fixture) in your class. If you want to run some code only once before or after all test methods in your class, you can consider using a BaseJDBCTestSetup or TestSetup test decorator.

Information about JUnit can be found at http://www.junit.org The usage of JUnit in Derby is currently based on JUnit 3.8.x.

This wiki also has an JUnit Introduction page.

The suite() method

The suite() method is a static method returning the tests to be run. We use it to gain more flexibility in which test methods are executed and the environment they are executed in. For instance, the use of the suite() method allows us to use test decorators (the most common case is setting up and populating a common database used by all tests in the suite).

By default, all test methods whose name starts with test are added to the suite, typically with new TestSuite(MyDerbyTests.class, "My suite name"). If there are tests that should only be run in certain scenarios, logic for doing this should be placed in suite().

In general, all tests should be written to be able to run under any framework. Tests that need to be written for a specific framework should be placed in a separate test class and have their suite() method return an empty suite when running in a different framework.

A test must be self-contained about which frameworks it can be run in. This avoids the situation with the old Derby harness where discovering if a test is being run in a framework and/or environment is very painful.

Tests that require a minimum level of JDBC support use the vmSupportsJDBC2, vmSupportsJDBC3, vmSupportsJDBC4 and/or vmSupportsJSR169 methods in the JDBC class. The style for using these methods should be "postive" and not "negative", e.g. "this test requires JDBC 3", not "this test doesn't run on JDBC 2 or JSR 169". This mechanism is to be used instead of any skip facility in the old test harness as it allows the tests to run correctly from standalone JUnit test runners.

If a test has no test cases to run in a certain configuration or VM environment then it should just return an empty suite.

See the test configuration page for more details about how to write suite() methods and the definition of primary configurations.

Package Level _Suite

A single suite defined by a class _Suite in a package should exist for all the JUnit test classes in that package. The special _Suite class has a suite() method that returns all the tests in the package by invoking the suite() method of each test class and adding the result to an instance of TestSuite.

For example, for the package org.apache.derbyTesting.functionTests.tests.jdbcapi a class org.apache.derbyTesting.functionTests.tests.jdbcapi._Suite would exist that runs all the tests (usually defined in subsuites) in that package.

See DerbyJunitTestConfiguration for details on top level suites.

Configurations

See DerbyJunitTestConfiguration

Usage of SQL states in JUnit tests

In your tests, you might want to check for a specific SQL state when an SQLException is thrown. The recommended way to handle this, is to use the BaseJDBCTestCase.assertSQLState(String, String, SQLException) and either pass in a constant from org.apache.derby.functionTesting.util.SQLStateConstants, or simply hardcode the value or use your own test-local constant. Doing this will assure that tests break if someone changes the SQL state for an exception in the internal class, and raises the awareness of SQL state changes in the community.

Please do not reference the internal Derby class that define SQL states. This class is not part of Derby's public interface, and should thus not be used to check for expected SQL states.

Running Tests

Which Tests

The page lists which set of tests can be run against Derby - DerbyTopLevelJunitTests.

Running tests using Junit directly

Tests can be run using the Junit 3.8 TestRunners. The classpath needs to include the junit.jar and the Derby classes (or jars). Most of Derby's JUnit test cases and suites run successfully using the TestRunners directly, as that is the eventual goal.

DERBY-1952 was opened to start removing JUnit tests from the old harness and instead only support running them directly using JUnit test runners. The top level suite
org.apache.derbyTesting.functionTests.suites.All successfully runs most of the JUnit tests standalone and those tests are not run by the old test harness.

The class name of the test passed to the runners can be an individual test case or one one of the Derby suites, see examples below.

Note that by default, the tests will attempt to use the port 1527 as this is the default port for Derby. Also, when running suites.All, a maximum of 20 ports will be used starting on 1527. No other ports will be used other than 1527+20.

To run the suites or individual tests with a different port, the derby.tests.basePort property must be specified. In this case, the ten ports that suites.All requires to run will start on basePort rather than on 1527.

(warning) Some JUnit tests may require running in the old harness for two reasons:

  • they depend on harness functionality that has not be added to the JUnit utility classes or
  • no-one has converted the test to use the new utiltiy methods

Such tests may fail in unpredictable ways when run by TestRunners direcly. One way to identify such tests is to see if they have any _app.properties or _derby.properties file, or they are listed in exclude files.

(warning) No system properties are required when running directly with TestRunners, the BaseTestCase class automatically installs a security manager with the correct policy file.

However, the following ad hoc system properties may be beneficial in specific cases:

  • derbyTesting.oldReleasePath : this property applies only to the upgradeTests suite, and then specifies the location where old release (derby.)jars can be found. More details can be found in the javadoc for the upgradeTests, but when running these tests regularly, or when running them behind a firewall, you should, respectively need to download the jars and specify the property. When the property is not specified, the (upgrade)tests will attempt to connect to the svn repository.
  • derby.tests.trace : when this value is set to true, output is generated that reflects each test and the time it took to run; this is useful when running with taciturn testrunners, such as the junit.textui.TestRunner (which otherwise only prints a '.' for each fixture). The default value is off.
  • derby.tests.debug : switches on verbose running. Any tests that have taken advantage of the method org.apache.derbyTesting.junit.TestConfiguration.println() will print additional statements when this is switched to true. Default value is off.

See some notes on running test runners on specific platforms including WCTME 5.7.

batch run

# Single test case
> java -cp '../../tools/java/junit.jar:../../classes' junit.textui.TestRunner
        org.apache.derbyTesting.functionTests.tests.jdbcapi.ProcedureTest
...........................
Time: 8.943

OK (27 tests)

# The jdbcapi package _Suite
java junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.jdbcapi._Suite

# All the test cases
java junit.textui.TestRunner org.apache.derbyTesting.functionTests.suites.All

GUI (swing) run

> java -cp '../../tools/java/junit.jar;../../classes' junit.swingui.TestRunner -noloading

Then supply the complete name of the test class in the top window, e.g. org.apache.derbyTesting.functionTests.tests.jdbcapi.ProcedureTest.
(warning) The use of the -noloading flag is required, which means that if you recompile Derby or the test classes you must restart the graphical test runners.

GUI (awt) run

> java -cp '../../tools/java/junit.jar;../../classes' junit.awtui.TestRunner -noloading

Then supply the complete name of the test class in the top window, e.g. org.apache.derbyTesting.functionTests.tests.jdbcapi.ProcedureTest.
(warning) The use of the -noloading flag is required, which means that if you recompile Derby or the test classes you must restart the graphical test runners.

Running tests using ant in a code line

The top-level build.xml file in the trunk codeline contains targets for running JUnit tests.

Target

Action

junit-single

Run a single test suite, specified by -Dderby.junit.testclass

junit-clean

Remove all the junitYYYYMMDD_hhss output folders from previous test runs

junit-all

Run all the JUnit tests. Runs the same set of tests as suites.All though it is broken down into per-package _Suite runs (thus the results may be different in some environments, especially when it comes to memory usage). May also run additional tests that require special setup or need to run in their own JVM such as the tests for checking the auto-loading of JDBC drivers. The report output is the XML format for ant's junit task.

junitreport

Run junit-all and produce an HTML report in the output folder

junit-all-codeline-jars

Run junit-all setting the classpath for the tests to be based upon the jar files build by the buildjars target. Note in this case the CLASSPATH environment variable is ignored. junit.jar should be in ant's lib folder.

The tests are run using the CLASSPATH of the environment which needs to include the Derby code you want to test and junit.jar.
JAVA_HOME needs to be set to the JVM to be tested and PATH must be set up so that the execution of java matches the virtual machine from JAVA_HOME.

If the ant property jdk16 is set then the Java SE 6/JDBC 4 tests will be run using the virtual machine at ${jdk16}/bin/java. Note this need not match the virtual machine defined in JAVA_HOME. If jdk16 is not set then the Java SE 6/JDBC 4 tests will not run, even if JAVA_HOME points to a Java SE 6 environment.

The output folder is created in the current directory with the name junitYYYYMMDD_hhss, e.g. junit_20070112_0813. The generated report can be viewed by opening the index.html file (located in the output folder) in a web browser.

Running a single JUnit test suite using ant in a code line

Here's an example to run a single test suite:

ant -Dderby.junit.testclass=org.apache.derbyTesting.functionTests.tests.lang.TriggerWhenClauseTest junit-single

Running tests using Eclipse

Eclipse is a good tool to run the junit tests. Once you have your code and the related jar files in eclipse env then using the java perspective in the package explorer right click on the classname of the test and select the "Run As" option. It will have Junit Test as one option. Click that and then you can view how the test progress in the junit tab. If a test fails you can click on the lines in the displayed stack trace and Eclipse will open up that java file in the editor at the correct line. Here is the screen shots for a passed.jpg and a failed.jpg.

Platform and JVM Specific Issues Running JUnit Tests

Various platform specific jvm issues and configuration requirements are tracked on the JunitVmIssues page. Please consult this page for information if you have trouble and update this page with new information as you discover it.

Running tests using the old Derby harness

(warning) All except two JUnit tests were as of January 2007 setup to run only as pure-JUnit tests, and it is expected that no new tests will be added that require the old harness, thus this section can be safely ignored for the most part. (warning)

The old Derby harness supports running JUnit tests directly and from its suite.runall files. The name of a Derby JUnit test is the test name using the suffix .junit instead of .java, e.g. somesuite/SomeTest.junit. In other words, the test type is junit. The file that implements a test is a java file, e.g. SomeTest.java. The test harness uses the last part of the name to decide how it should run the test. If the name ends with .java, the harness knows that the test is a java class with a main() method. If the name ends with .junit, the harness knows it is a java class which can be run by a JUnit test runner. No actual file exists with the .junit suffix.

The harness can be used to run individual JUnit tests or tests that are suites, in both cases the RunTest class is used. RunSuite is only for suites defined by the old harness setup (i.e. .runall files)

# Run the set of language JUnit tests
java org.apache.derbyTesting.functionTests.harness.RunTest lang/_Suite.junit

# Run one language JUnit test
java org.apache.derbyTesting.functionTests.harness.RunTest lang/GroupByExpressionTest.junit

(warning) JUnit tests run within the old harness can take advantage of the harness's facility to control the environment and when the test is run by using properties files, exclude etc. However, the practice is strongly discouraged as it makes it harder to run tests directly using JUnit test runners, which is the eventual goal. Most of the facilities provided by the old harness now exist in the JUnit utility classes in the org.apache.derbyTesting.junit package.

Running the XML Tests

The XML tests require a more advanced version of Xalan than the default version bundled with JDK 1.4. The XML tests silently exit if the required environment is not found.

To run the XML tests under JDK 1.4, you must do the following:

  • Download the latest version of Xalan (2.7.0 as of this writing)
  • Copy all of the downloaded jars into the jre/lib/endorsed directory of your JDK 1.4 installation. Those jar files are: serializer.jar, xalan.jar, xercesImpl.jar, xml-apis.jar, and xsltc.jar.

That's it! Now the XML tests should run for you under JDK 1.4.

To run the XML tests under a higher version of the JDK, you must do the following:

  • Download the latest version of Xalan as described above.
  • Wire the downloaded jar files into your CLASSPATH.

Running concurrent JUnit tests

It currently is possible to run concurrent JUnit tests. This can be done for standalone tests or for whole suites of tests, including the all-encompassing suites.All. For this to be possible, the base port for the tests must be defined for each run.

To do so, the property derby.tests.basePort must be specified upon the execution of the test.

It needs to be noted that the full range from the specified port through to basePort+20 MUST be free. Therefore, when running several tests make sure that the range [basePort;basePort+20] for all of the tests does not overlap.

Examples:

# The base port is 2230. Ports from 2230 through to 2240 (inclusive) will be used
java -Dderby.tests.basePort=2230 junit.textui.TestRunner org.apache.derbyTesting.functionTests.suites.All

# The base port is 2241. It does not overlap with the previous test.
java -Dderby.tests.basePort=2241 junit.textui.TestRunner org.apache.derbyTesting.functionTests.suites.All

OK!

# The base port is 2230. Like above, ports through 2240 will be used
java -Dderby.tests.basePort=2230 junit.textui.TestRunner org.apache.derbyTesting.functionTests.suites.All

# The base port is 2235 and falls within the range [2230;2240]. Unexpected behavior might arise.
java -Dderby.tests.basePort=2235 junit.textui.TestRunner org.apache.derbyTesting.functionTests.suites.All

NOT OK!

The first example will work perfectly, but to be absolutely safe, the ranges would be farther apart.

Running the LDAPAuthenticationTest JUnit test

The junit test LDAPAuthenticationTest requires an LDAP server configured in a very specific way, details of which are described in the test' source code. As it is by nature not possible to set up a secure server to be used by just anyone, the details for the LDAP server in the test are provided by properties. As not everyone can have access to an LDAP server, if the properties are not provided, the test will not be run. For further information please refer to the test itself; but here are the properties the test looks for:

  • derbyTesting.ldapServer=[ldapserver_ipaddress]

  • derbyTesting.ldapPort=[port for the ldap server]

  • derbyTesting.dnString=[dn string for the ldap server]

  • derbyTesting.ldapUser=[main ldap user]

  • derbyTesting.ldapPassword=[mail ldap user's password] Note that the test requires an additional user to be set up on the LDAP server - for details, see the test.

Optional delay for Network Server start time

Sometimes when running JUnit tests with jvm options that are known to slow things down significantly network server start timeouts can occur, for instance in derbynet.SecureServerTest. This is governed by the default wait time set in org.apache.derbyTesting.junit.NetworkServerTestSetup. To allow for one-off adjusting of the settings, without modifying the wait time in the test framework source, one can use the following property (10.5 and up) to overwrite the default:

  • derby.tests.networkServerStartTimeout=[int indicating wait time in seconds] This property will overwrite the default specified in class org.apache.derbyTesting.junit.NetworkServerTestSetup.

Switching the Derby harness over to JUnit

An ultimate goal would be for all Derby tests to be JUnit tests and all existing logic in the Derby test harness is replaced by JUnit code, such as test decorators to perform test setup and cleanup. A discussion was started on derby-dev. This would gain all the benefits of JUnit, such as running tests from ant, integration with IDEs, ability to hook into other JUnit suites, easier understanding of how Derby tests are run etc.

Click here on more information on the roadmap to completing the switch to JUnit.

  • No labels