Replacing functionality of the Derby test harness
Goals
- Derby's use of JUnit fits the standard pattern
- All tests pass when run as individual tests
- No setup required on the command line, can run tests directly from JUnit's TestRunners
- Standard mechanism to add suites (use of TestSuite class directly)
- Tests can be run from an ant target
- Tests/Suites can be added into other project's test suites with no additional knowledge or requirements
Conversion or writing JUnit tests FAQ
SQL tests
Harness based script tests (.sql scripts) can be converted in three ways:
- Continue running the test as a script using the ScriptTestCase class, see ConvertSqlScriptTestsToJunit
- Look at using the utility org.apache.derbyTesting.functionTests.util.SQLToJUnit attached to convert the old test automatically. More info is in DERBY-2151.
- Re-write the test by hand using standard JDBC calls and the utility methods and decorators available. The method org.apache.derbyTesting.junit.Utilities.showResultSet(ResultSet rs) can be used to print resultsets out in two dimensional array format for use by assertFullResultSet.
Decorators
A number of useful decorators and convenience methods that return decorators exist, this section shows which to use for certain situations. Remember that a decorator can wrap a set of test fixtures:
return new CleanDatabaseTestSetup(suite);
or a single test fixture
suite.addTest(new CleanDatabaseTestSetup(new MyJDBCTest("testJDBCStuff")));
Decorators can also be nested, some thought may be required to ensure the nesting results in the correct execution order:
return new CleanDatabaseTestSetup(
DatabasePropertyTestSetup.setLockTimeouts(suite, 2, 4));
Question |
Answer |
How to set system properties for a test? |
Setting properties on a database level is preferred as setting properties at a system level may interfere with other tests that could be run in parallel. However the SystemPropertyTestSetup decorator does exist to allow setting system properties, passing in the set of properties to set. |
How to set database properties for a test? |
Use the DatabasePropertyTestSetup decorator passing in the set of properties to set. |
How to reduce the lock timeout values for a test? |
Use the conveniencedecorator returned by DatabasePropertyTestSetup.setLockTimeouts or DatabasePropertyTestSetup directly. |
How to set BUILTIN authentication with a set of users |
Use the convenience decorator returned by DatabasePropertyTestSetup.builtinAuthentication, use a unique password token |
How to run as a different user? |
Use the decorator returned by TestConfiguration.changeUserDecorator |
How to run tests with Derby's client driver? |
Any test class's suite() method should return a suite that runs fixtures in embedded and client/server mode as required. TestConfiguration.defaultSuite() takes a Class object and returns a suite that runs all the test methods starting with |
Since the database is re-used when running tests how do I ensure the test starts out with a clean database? |
Use the CleanDatabaseTestSetup decorator. |
How to execute DDL at setUp that is dropped automatically at tearDown? |
Provide a class that extends CleanDatabaseTestSetup and implement the decorateSQL method. This can be achieved as an inner class, search for references to CleanDatabaseTestSetup for examples. |
How to run a test in SQL authorization mode? |
Use the utility method TestConfiguration.sqlAuthorizationDecorator. The simple DatabasePropertyTestSetup cannot be used directly since setting the SQL authorization mode cannot be undone. The returned decorator changes the default database to a different one that has SQL authorization mode set. This database, like the default database, is re-used by each call to TestConfiguration.sqlAuthorizationDecorator. |
Harness features
The test decorator pattern, a class extending TestSetup or the Derby specific BaseJDBCTestSetup seems the natural way to replace most of the functionality of the test harness.
Feature |
Decorator |
JIRA |
Comments |
Create & remove database |
yes |
|
|
Setup SecurityManager and policy file |
no & yes |
|
Messing around with this it seems the best way is to have BaseTestCase install a security manager with the default policy and have decorators to remove the security manager or change the policy file for individual tests. Then a large number of decorators are not needed to run the tests and the exception cases (no security manager, separate policy file) will stand out. Adding a class |
Cleanup database objects |
yes |
new - allows databases to be shared across tests to decrease test running time |
|
Determine JDK environment |
no |
DERBY-1638 for JDBC level |
E.g. isFoundation() etc., is JDK 1.3 |
Boot & shutdown network server |
yes |
At least within the same JVM can be a decorator |
|
Setup & remove extin, extout, extout folders |
yes |
(see DERBY-1656 ) |
|
Copy & remove support files |
yes |
|
In some cases files can be accessed directly as resources instead of copying them. See the test resource methods in |
Set & clear derby properties |
yes |
|
Replacement for the <test>_derby.properties, system properties implemented in class |
Shutdown database/server |
no |
|
|
Support running junit tests directly with a remote server |
no |
|
Folder Policy
It would be good to have some up-front policy about how folders are used, to provide consistency and help with good security manager testing. See:
Old Harness Suite status
Progress on removing tests from the old suites, with the assumption they are converted to JUnit tests.
Suite |
Initial Test Count |
Remaining Test Count (2007/02/15) |
2013/09/26 |
Comments |
demo |
2 |
2 |
0 |
see DERBY-2903 and DERBY-3294 |
derbylang |
148 |
91 (~70% .sql tests) |
37 (34 .sql) |
|
derbynetclientmats |
8 |
6 |
4 ( 1 junit running in old harness) |
ClientJunitTestConversion (derbynet/* only) |
derbynetmats |
45 |
42 |
0 |
ClientJunitTestConversion (derbynet/* only) |
derbytools |
15 |
15 |
2 |
|
encodingTests |
4 |
4 |
1 |
|
encryption |
3 |
3 |
2 |
|
encryptionAES |
2 |
2 |
0 |
|
encryptionAll |
8 |
8 |
5 |
DERBY-1001, DERBY-3711. For an encryption Decorator (among other things), see DERBY-1952. |
encryptionBlowfish |
3 |
3 |
1 |
|
encryptionCFB |
3 |
3 |
1 |
|
encryptionDES |
3 |
3 |
1 |
|
encryptionECB |
3 |
3 |
1 |
|
encryptionOFB |
3 |
3 |
1 |
|
i18nTest |
7 |
7 |
0 (removed) |
|
j9derbynetmats |
30 |
0(ignore) |
|
old suite for running on a specific jvm |
jdbc20 |
10 |
5 |
0 (removed) |
|
jdbc40 |
8 |
4 |
0(removed) |
|
jdbcapi |
22 |
15 |
0 (removed) |
|
jdbcxa40 |
2 |
0 (removed) |
|
|
jdk14 |
9 |
7 |
0 (removed) |
|
largeDataTests |
1 |
1 |
0 (removed) |
Not in derbyall |
multi |
1 |
1 |
0 (removed) |
|
nist |
125 |
0 (removed) |
|
|
propertyinfo |
1 |
1 |
1 |
|
simpledemo |
1 |
1 |
1 |
|
storemats |
6 |
6 |
1 |
|
storemore |
38 |
38 |
16 |
|
storerecovery |
13 |
13 |
9 |
|
storetests |
8 |
8 |
5 |
|
storeunit |
34 |
34 |
34 |
|
unit |
4 |
4 |
4 |
|
upgrade |
1 |
1 |
0 (removed) |
|
xa |
11 |
11 |
5 |
|
xmlTests |
2 |
0 (removed) |
|
|
Total |
|
344 test runs with 306 unique tests |
134 test runs |
|