best practices - testing strategies

<quote>

Should we add new POM elements? We currently have <testSourceDirectory> for unit tests. Should we add <integrationTestSourceDirectory> and <functionalTestSourceDirectory>?

</quote>
How about :

<build>
       <sources>
          <source>
             <directory/>
             <output/>
          </source>
       </sources>
       <tests>
          <test>
            <type>Unit/IT/Functional/System/Custom</type>
            <directory/>
            <output/>
          </test>
       </tests>
    </build>

I know this would mean changing the POM structure a bit, but IMHO it would be more flexible, and might be better to do it now before we have a huge maven2 following.

2 reasons that I can think off top of my head:

1. The above structure should make it easier to extend Test types.
2. I have come across use cases (a few times now) where Eclipse project tree can have more than one source folder and where it makes little sense to factor out and create two project modules.

Is this a reasonable change  - what do you think?

I consider that Cactus test on my ejb are unit test. Some, because done inside a container will consider it integration test even if a test affect only one ejb at a time. I think the line between unit/functional/integration is sometimes to thin to base the directory structure on it. Why not have a test directory with inside it a separation by type of test such as:
test
 |_junit
 |_cactus
 |_httpUnit
 |_JMeter
 |_...
and so on. The surefire plugin could then call the plugin responsible for cactus, httpUnit....test. It makes the structure clear and simple and the possibility to add test easy.

I really like Rahul suggestion since the division of test can be so different from an one organisation to another. Some might want to separate their tests by frameworks, some might still divide their tests by type but want a way to decompose their junit in two phases of test, ... I think Maven should give some flexibility here and let the user decides. Maven should just has some predefined lifecycle phases for each kind of test and a set of test could be bind to a specific phase by adding a xml element like the one used to bind plugins to a phase.

In my mind, the main information that needs to be captured here is the project knowledge regarding acceptable test suite success rate for a successful build.

Since unit tests verify the semantics of code, much like the compiler verifies the syntax of code, all unit tests must pass at 100%, just as they must all compile.  Integration tests, by definition, test more than one part of the system, and are not required to pass at 100% for the build to be considered successful.  Most likely a release would not be performed until all the integration tests pass at 100%, in fact they might be coded early in the development cycle and then treated as exit criteria for the release.

So, I'd vote for something like the first approach:

project/
  |_ src
    |_ main
    |_ test
      |_ java
      |_ resources
    |_ integ
      |_ java
      |_ resources

Then, I would use regular Java packaging to separate the test code for each different technology, such as JMeter, HttpUnit, and so on, if I felt that distinction was important.  Perhaps I might prefer to split these by purpose instead, such as performance, web, and so on.  The test technology separation doesn't seem to be a decision that Maven needs to be aware of, or something that Maven should try to enforce / standardize.

Two points I'd like to highlight...

From John R Fallows' comment above I agree to the statement that:

The test technology separation doesn't seem to be a decision that Maven needs to be aware of, or something that Maven should try to enforce / standardize.

From Alexandre Poitras' comment above it makes sense that:

Maven should just has some predefined lifecycle phases for each kind of test and a set of test could be bind to a specific phase by adding a xml element like the one used to bind plugins to a phase.

Testing Plugins (JUnit/Surefire/JMeter/HttpUnit) could be bound a test -> type (my comment above) with allowance for overrides if required. I believe this would allow Maven to treat any type of 'test' as a 'test' (no special handling) and delegate handling based on its type to the bound plugin(s).

Yeah but my concern is that a tool can be used for more then one type of test. For instance, junit is often used of course for unit testing but also for intregation testing (this is a common case if you use Spring framework). That's why I was proposing something similar to the plugin model but we could use your method for default values, easily overridable in the pom. This way the testing would be consistent with the pom global approach. The test type element would be like the project packaging element, it inferes default value about your tests but those values can be easily overrided by declaring some plugins and binding them to an execution phase. I think it would be a great solution not changing drasticly the pom format.

Here's a little example of what I mean :
*By the way, forget my comments about having  a tool used for two types of test. I taught you were infering the test type from the tool, but I see now you actually meant test type-> tool. Anyway, what I just proposed still stands.
<build>
<sources>
<source>
<directory/>
<output/>
</source>
</sources>
<tests>
<test>
<type>Unit/IT/Functional/System/Custom</type>
<directory/>
<output/>
</test>
</tests>
<plugins>

1. let's say you want to override the default tool used for integration test
   <plugins>
   <plugin>
   <groupId>org.junit</groupdId>
   <artifactId>junit</artifactId>
   <version/>
   <extensions/>
   <executions>
   <execution>
   <id/>
   <phase>integration-test</phase>
   <goals/>
   <inherited/>
   <configuration/>
   </execution>
   </executions>
   </plugin>
   </plugins>
   </build>

What do you thin?

I just thought I'd add to these comments the approach I'm currently using to perform integration tests with M2 as M2 is now. It may ensure that all angles are covered in any of your recommended approaches.

I have achieved integration testing (using both JunitEE/Cactus and JUnits calling remote interfaces) in M2 using a very simple lightweight POM for each discrete task/execution in the integration phase (i.e. bound to this phase) e.g.

+ application-root  
     pom.xml   (parent pom)
    
   now normal stuff follows e.g. :- 
  
    + normal EAR
    + normal JAR(s)
    + normal WARs
    + test WAR (for JUnitEE or Cactus)

   now follow's the M2 project modules to run the integration tests e.g. :-
  
    + runtests-integration
         pom.xml   (sub-parent pom, invoking child modules in the order below)
        + rebuild-database
               pom.xml  (antrun to rebuild DB schema)
        + start-container
               pom.xml  (antrun or cargo to start container )
        + runtests-serverside  (runs junits needing server started as they invoke remote interfaces)
               pom.xml (surefire:test only picking up tests ending in *ServerTest.java )
        + runtests-incontainer-tests
               pom.xml (antrun for running JUnitEE or Cactus incontainer tests)
        + stop-container
               pom.xml (antrun or cargo)
 
All plugins defined under 'runtests-integration' (and below) are bound to the integration-test phase
 
I found this approach also necessary, because you can only have one instance of the antRun plugin for the integration-test phase, therefore you can't use antrun, then surefire, then antrun again  as the last antrun overrides the first if they are bound to the same phase.

I did want to use Cargo, but unfortunately there was no support for OC4J with an existing configuration.

I have the above 'runtests-integration' suite only execute if the M2 build profile is set to -Pintegrationtests. In this way developers can run a regular M2 build which just runs standalone JUnits, or specify the profile to run a fuller build. We have CruiseControl always running the integrations tests profile.  

With m2.02 I should now convert the Antruns to Ant Plugins.

I'm hoping that when M2 supports integration tests better, I can migrate to the approved strategy, I know this approach is a slight misuse of M2, but this works right now, we have 2 large Applications (both EARs) that are built with M2 via CruiseControl, the above integration projects start the container run all the integration tests (with the 2nd EAR application even invoking the 1st EAR's services)

I have another M2 module in addition to those shown above that actually hold the integration tests :-

    + tests-serverside 
         pom.xml   (JAR pom, building those JUnits needing server started)
        
           The JUnits in this project follow two naming patterns :-
           src/test/java
                 *ServerTest.java  (JUnits needing server started as they invoke remote interfaces )
                 *JEETest.java       (JUnits needing to run inside container via JUnitEE or cactus )

     The above JAR could probably have been split into two separate JARs for each type.
    
     The parent pom.xml overrides the default maven-surefire-plugin by excluding the following :-
    
     <excludes>
      <exclude>**/*ServerTest.java</exclude>
      <exclude>**/*JEETest.java</exclude>
     </excludes>
    
     such that these tests don't get run during the normal test phase. Just regular standalone *Test.java get run.
    

Hope this is useful

Pete

I have summarized testing strategies I found in the web here:

http://pgt.de/2006/04/04/integration-and-unit-test-strategies-with-maven2/

Don't forget to include support for integration testing of plugins.
This gets more complicated as the plugin has yet to be installed locally (and shouldn't be until it passes the tests)

The summary can be found here : http://adminsight.de/articles/programming/integration-and-unit-test-strategies-with-maven2.html

As many people point it out, we want a layout similar to

src
    |_ main
    |_ test
    |_ integ
 

But if  the integration test directory is ismply called "integ" it will appear before the main directory in most IDE. The chosen name must be such as its comes after "test" in alphabetical order.
Thanks!

Any news on the separate directory integration-test strategy?

• I have noticed that lot of test gurus when talking about testing use "unit", "functional", and "scenario". Therefore, another possible alternative directory structure might be:

project/
|_ src
  |_ main
  |_ test
    |_ unit
      |_ java
      |_ resources
    |_ integration
      |_ java
      |_ resources
    |_ functional
      |_ java
      |_ resources
    |_ scenario
      |_ java
      |_ resources

Alternatively, the integration might be removed with the assumption that a scenario test will cover integration tests.

project/
|_ src
  |_ main
  |_ test
    |_ unit
      |_ java
      |_ resources
    |_ functional
      |_ java
      |_ resources
    |_ scenario
      |_ java
      |_ resources