At times, you will find yourself confronted with the situation in which a certain test is failing, and will be failing for some period of time. For example, there may be a bug, which has been logged in Jira and which is well understood, but which won't be fixed today, or tomorrow.
In general, the development community tries not to leave "expected" failures in the head of the tree. Our goal is that if you check out the head of the trunk, and build it and run the tests, they will all pass.
Here are some possible techniques you can use if you find yourself in this situation:
- Many tests are run as part of multiple suites. If a test is failing only in one suite, you can remove the test from that suite and leave it in the others, with comments indicating to restore the test to the suite when the bug is fixed.
- Depending on the test, you may be able to remove a very small part of the test, perhaps commenting out just a single statement of the test code, until the bug is fixed.
- At other times with the canon-based tests we have added comments to the test output itself saying something like "the next few lines are wrong due to DERBY-xxx"
In general, please don't just update the master file so that it contains erroneous results without at least putting comments into both the test and the test output indicating that the output is known to be wrong, and is due to bug DERBY-xxx.
That has the benefit of making it very clear when the bug is fixed and a new diff appears, why that is happening.