Goal

Have a way for plugins to discover what JDK (or other tools) are to be used, without configuring the plugins.

The current Maven way of achieving this is to run Maven itself with the required JDK.

After toolchains, the JDK that Maven is running within, shall be irrelevant to the project in question.

Motivation

Current way or enforcing project's JDK version (via the enforcer or otherwise) by forcing the user to run Maven itself with the given JDK version breaks embedded use.
Additionally toolchains will allow a type of user interaction that IDE users are used to. Just set the JDK to the project and go.

Types of toolchains

Toolchain idea

Relevant Plugins

jdk

maven-compiler-plugin maven-surefire-plugin maven-javadoc-plugin keytool-maven-plugin exec-maven-plugin webstart-maven-plugin

j2me sdk

j2me-maven-plugin

native tools? c#?

netbeans-platform

nbm-maven-plugin The various goals could make use of it.

see the Guide to Using Toolchains to get a list of known existing toolchains and plugins using them.

Design

Note: I'll be focusing on JDK toolchain. I don't have enough background information for other types of toolchains.
The associated issue is: MNG-468

3 basic points of view:

1. Plugin denotes what toolchain type it requires for it's operation. So compilation, surefire, jnlp, ... need a JDK toolchain instance for example. The actual instance of the toolchain is passed into the plugin by the infrastructure (using MavenSession in current implementation).
Most current plugins that use JDK for processing, use the maven's JDK instance by default. The only change introduced is to use the toolchain in the MavenSession if found. If not, do as we did so far.

Q1: how shall the plugin tell what toolchains it needs? parameter? parameter's or mojo's @toolchain annotation?
The actual retrieval of the toolchain from build context can be done completely behind the scenes, so marking the plugin as toolchain-aware is primarily documentation oriented.

2. User defines the toolchain instances that are available in his current setup. Shall be user based, project independent, stored in $HOME/.m2/toolchains.xml file.
Example toolchains.xml file:

<toolchains>
    <toolchain>
       <type>jdk</type>
       <provides>
           <version>1.5</version>
           <vendor>sun</vendor>
           <id>for_mevenide</id>
       </provides>
       <configuration>
          <jdkHome>/home/mkleint/javatools/jdk</jdkHome>
       </configuration>
    </toolchain>
    <toolchain>
       <type>jdk</type>
       <provides>
           <version>1.6.0</version>
       </provides>
       <configuration>
          <jdkHome>/home/mkleint/javatools/jdk1.6.0</jdkHome>
       </configuration>
    </toolchain>
</toolchains>

3. Project shall be allowed to state which instance of the given toolchain type it requires for the build. Therefore making sure that all plugins use jdk15 for example.
If such toolchain instance is not found in user's local setup, the build shall fail early.
For this purpose a new plugin (maven-toolchain-plugin) is introduced. It is responsible for matching the correct toolchain instances from the user's setup against the requirements of the project and make it available for other plugins to use (in the build context).

<plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-toolchains-plugin</artifactId>
   <version>1.0-SNAPSHOT</version>
   <executions>
      <execution>
         <phase>validate</phase>
         <goals>
            <goal>toolchain</goal>
         </goals>
      </execution>
   </executions>
   <configuration>
       <toolchains>
          <jdk>
              <version>[1.4)</version>
              <!--vendor>sun</vendor-->
          </jdk>
        </toolchains>
  </configuration>
</plugin>

The process of matching required toolchain against the provided ones is following.

  1. Get all toolchains for a given type (eg. jdk)
  2. go through them one by one and try match the requirements in the toolchain-plugin configuration against the provided tokens. Some like 'version' are to be specially handled and allow for range matching etc, the rest is exact match only. The creator of the particular type of toolchain can define the specially handled tokens.
  3. first successful match is pushed into the maven build environment (MavenSession) for use by other plugins down the road
  4. if there are no matches, the build fails.

Backward compatibility

  1. The Toolchain core components need to go to 2.0.9-SNAPSHOT and 2.1-SNAPSHOT codebase.
  2. maven-toolchains-plugin gets the maven prerequisite of 2.0.9-SNAPSHOT. That way anyone actively using toolchains needs to be using 2.0.9-SNAPSHOT+
  3. Any other plugins using the toolchains components don't need to explicitly set the maven prerequite to 2.0.9-SNAPSHOT. The components will work but will fail to deliver a configured toolchain, thus they will keep behaving the same way as they did before. That's critical for further development of plugins.

What changes are needed in plugin code to start using toolchains?

The changes are rather simple. An additional dependency on the toolchains component artifact is necessary and then a short code snippet added to mojo's execute method.

    /**
     * @component
     */
    private ToolchainManager toolchainManager;

    /**
     * The current build session instance. This is used for
     * toolchain manager API calls.
     *
     * @parameter expression="${session}"
     * @required
     * @readonly
     */
    private MavenSession session;

    public void execute() {
        .....

        // get toolchain from context
        Toolchain tc = toolchainManager.getToolchainFromBuildContext( "jdk", //NOI18N
                                session );
        if ( tc != null )
        {
            getLog().info( "Toolchain in javadoc-plugin: " + tc );
            // when the executable to use is explicitly set by user in mojo's parameter, ignore toolchains.
            if ( javadocExecutable  != null )
            {
                getLog().warn( "Toolchains are ignored, 'javadocExecutable' parameter is set to " + javadocExecutable );
            }
            else
            {
                // assign the path to executable from toolchains
                javadocExecutable = tc.findTool( "javadoc" ); //NOI18N
            }
        }

        .....
     }

Implementation

The feature is developed mainly on trunk. toolchains components toolchains plugin
Only the changes to existing plugins are on branch.

after compiling, make sure you put the jar with toolchain components into the 2.1-SNAPSHOT maven binary, into the lib/ subfolder.
Attached is sample project and toolchain definition.

  • No labels

10 Comments

  1. Unknown User (shane.isbell@gmail.com)

    This is an important issue for NMaven. For clarity, I would like to explicitly list the requirements I need for C#. Some of these requirements are met by the proposal.

    1. There are two types of requirements we need: a toolchain for the Java maven plugins and a toolchain for the target platform. For example, I may have a maven plugin requiring JDK 1.5 with a target C# language of Mono compiler under .NET framework version 2.0.

    2. Profiles, such as J2ME (MIDP/CDC, etc) or .NET compact, need to have an associated bootclasspath configurable within the toolchain configurations.

    3. Each module needs its own toolchain. Say module A is building under .NET 1.1 and module B is building under .NET 2.0. I need to support both under the same build. Partially addressed in proposal '3. Project'.

    4. We do need the platform requirements specifiable (but not required) within the pom.xml file. For example, I would like the option to say that this module requires .NET 1.1 framework. You raise this issue in Q2.

    5. There is a notion of compilers as well as executables (such as schema generators, wsdl generators, code analysis tools, etc). There needs to be a way to define toolchains for both of these. This is addressed in the goal statement of the proposal.

    6. There needs to be a fairly robust way of defining default values for the toolchain configuration. For example, I have a state machine that fills in missing requirements based on operating system and what it finds installed on the platform. 

    The biggest deficiency I see within the proposal is the limited notion of 'instance'. For example, say I have two instances of the JDK: jdk14 and jdk15. Now say I want to add support for jikes. I have doubled my instances to jdk14-jikes, jdk14-sun, jdk15-jikes, jdk15-sun. Add in support for 1.3 and 1.6 and it multiples further. The J2ME case with numerous foundations and profiles would be far worse. 

    So the notion of 'instance' or exact toolchain matching breaks down in everything but the simplest case. As an artifact provider, I may want to specify jdk15 but let the plugin choose the compiler vendor for the build developer based on a set of matching rules. To give an example of the problem, say the artifact provider specifies jdk15-sun as the toolchain instance. Now say a build developer - who only has jikes installed on their system - types 'mvn compile', the build will fail when it should not.

    One way of solving this, is to introduce the concept of toolchain requirements (nothing more than a list of name-value pairs, which in this case may just be Java.JDK=1.5). The toolchain plugin then needs to match these requirements based on values of the toolchain (or platform) capabilities, specified within the toolchains.xml file. As mentioned above, you could have some state-machine implementation fill in the missing requirements prior to passing off to the component handling the toolchain matching. If the system finds jikes, it will choose jikes as the compiler; otherwise it will choose Sun's compiler - or vice versa, depending on the matching rules.

  2. Thanks for comments.

    1. "maven plugin requiring JDK 1.5" is out of scope for toolchains proposal. Toolchains are only "hints" for maven plugins to on what tools to use for it's execution. It cannot/shall not influence the runtime environment of the plugin itself.

    2. the configuration section in toolchains.xml shall allow that to happen. On the runtime side of things, I expect to have a basic Toolchain interface that will be extensible by multiple special-purpose subinterfaces. The J2METoolchain can have the bootclasspath then. The J2ME plugin will then use this "type" of toolchain.

    3. shall be working fine, the maven-toolchain-plugin can be configured for each project separately. It even allows for defining different profiles configuration within the same project. Eg. once shall be able setup a profile to build/execute tests with different version of jdk/...

    4. I'm generally leaning towards a toolchains-plugin rather than doing a special section in pom.xml. That plugin will prepare all the toolchain instances for use by other plugins down the stream. The other plugins (compiler/surefire) need to handle the case when the toolchain is not available (either fail or probably just use some default - the running JVM for example or what is defined it the plugin's own configuration.

    6. as mention in 4. I believe the plugin is responsible for correct fallback behaviour. So the decision priorities look like this.
    a. Check the plugin's own configuration (as defined in pom.xml). if present, use it.
    b. check for a toolchain of the type I understand and use. if present, use it.
    c. Some sort of fallback behaviour. for compiler/surefire/.., use the JVM version that is used by the maven build itself. In the case of your C# plugin, try to figure out the right platform based on OS install.
    d. If c. doesn't work, fail.

    The final comment is indeed the hardest nut. The solution you propose sounds reasonable. The require-provide relationship should be flexible. Then we can get rid of the instance identification as it was just the poor man's solution to this problem anyway. It could look like this:

    <toolchains>
        <toolchain>
            <type>jdk</type>
            <provides>
                <jdkVersion>1.5</jdkVersion>
                <!-- anything else that the JDKToolchain understands.. -->
    
                <!-- to allow to virtually anything we can have this: -->
                <id>custom foo.com JDK with tweaked bootclasspath</id>
            </provides>
            <configuration>
                <directory>/home/mkleint/javatools/jdk1.5.0_07</directory>
            </configuration>
        </toolchain>
    <toolchains>
    

    and on the toolchain-plugin's side.

    <plugin>
        <artifactId>maven-toolchain-plugin</artifactId>
        <configuration>
            <toolchains>
               <jdk> 
                  <jdkVersion>1.5</jdkVersion>
               </jdk>
             <toolchains>
        </configuration>    
    </plugin>
    

    Any content under the <jdk> element in toolchain plugin is not mandatory, the plugin will search in the available toolchains for the first successful match. So empty <jdk> can result in any jdk match (1.6, or 1.4), the jdkVersion filled in means only 1.5(plus) gets matched. Ideally we should have the same version resolution in the JDKToolchain (and possibly other toolchains as well) as the enforcer-plugin has now.

  3. Don't forget the maven-antrun-plugin that should also utilize the toolchain e.g. calling rmic.

  4. i'm not sure how the antrun plugin could ever utilize toolchains in a generic manner.

  5. Unknown User (shane.isbell@gmail.com)

    I'd like to see a global configuration section of the toolchains.xml file. I need to add a lot of tools, each of which has a different executable name but the same bin directory.

  6. Shane Isbell: Isn't the findTool() method in the toolchain solving your problem? In the JDK toolchain, I can lookup "java", "javac", "javadoc" tools within one toolchain instance..

  7. Unknown User (shane.isbell@gmail.com)

    The issue that I am running into is that certain executables may be repeated between different toolchains, like the nunitConsole configuration below. If there were a global configuration, I would not need to repeat it.

    <toolchains>
        <toolchain>
            <type>dotnet</type>
            <provides>
                <frameworkVersion>2.0</frameworkVersion>
                <vendor>MICROSOFT</vendor>
                <id>.NET Framework 2.0</id>
            </provides>
            <configuration>
                <csharpCompiler>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\csc</csharpCompiler>
                <nunitConsole>C:\Program Files\NUnit 2.4.6\bin\nunit-console</nunitConsole>
            </configuration>
        </toolchain>
        <toolchain>
            <type>dotnet</type>
            <provides>
                <frameworkVersion>2.0</frameworkVersion>
                <vendor>NOVELL</vendor>
                <id>.NET Framework 2.0 - Novell</id>
            </provides>
            <configuration>
                <csharpCompiler>C:\Program Files\Mono-1.2.6\bin\gmcs</csharpCompiler>
                <nunitConsole>C:\Program Files\NUnit 2.4.6\bin\nunit-console</nunitConsole>
            </configuration>
        </toolchain>
    </toolchains>

  8. Unknown User (gagern)

    I second this require-provide approach. To give you a hard example of what would be nice to have, consider you want a java compiler that checks for 1.4 API compliance and does lint checks as supported since sun jdk 1.5. So I'd want maven to run this command:

     /usr/lib/jvm/sun-jdk-1.5/bin/javac -Xlint:all \
       -Xbootclasspath:/usr/lib/jvm/sun-jdk-1.4/jre/lib/rt.jar
    

    Do you consider finding the appropriate paths to be in the scope of this proposal?
    Would I request a (JavaJDK=1.5, Vendor=sun) compiler and the (JavaAPI=1.4) bootclasspath as two different tools, and combine them on the command line?

    Or would I rather request a (JavaAPI=1.4, JavaJDK=1.5, Vendor=sun) toolchain which has the bootclasspath configured in toolchains.xml? Then I'd definitely want some kind of cross-refernecing within the toolchains.xml in order to reuse values specified only once.

    Or would I specify my requirements more like (JavaAPI=1.5, Lint=all) and some pattern matching solution in the toolchain configuration could find a version that supports lint and pass the correct argument appropriately? This pattern matching engine might even be used to search for /usr/lib/jvm/sun-jdk-${JavaJDK} in order to automatically adapt as new versions get installed.

  9. re: Martin von Gagern.

    I'm not sure who is "I" and what is "maven" in the sentence "So I'd want maven to run this command".

    The primary purpose of the toolchains proposal is to unify maven-plugins usage of resources like JDK used without the need to configure each plugin separately. The configuration is done in one place only "maven-toolchains-plugin", the other plugins reuse it's configuration.

    So the actual code that needs to execute your command line example above is the compiler-plugin or some other custom plugin. That one shall support toolchains, to share the use of jdk picked with surefire-plugin and others. compiler-plugin currently supports your scenario only in if you manually configure the compiler args with "-Xlint:all -Xboothclasspath....". So that means you (as project owner) configure the path to jdk 1.4 rt.jar. I don't think toolchains can help there. Actually it's specifically implemented now that users (project owners) can override the toolchains configuration within the domain plugins. Eg. if you want all plugins to use 1.5, but want to compile with 1.4 for example, you configure the compiler-plugin as you do now ("executable" parameter with path to javac I think) and it doesn't use the toolchain.

    if your example is deemed main stream and supported usecase by compiler-plugin devs, I imagine we could enhance the jdk toolchain to provide boothclasspath and the compiler-plugin could use it. on the user level it would boil down to additional, optional configuration parameter within the toolchain.xml file. for most people it will return the rt. jar of the actual JDK defined, but one could override that to return something else. if that gets implemented, I'm not convinced a support for "cross-referencing for value reuse" is necessary.

  10. Aside from providing access to tools and versioning of them, some scheme for providing source and target version is needed.

    Source compliance level should for example be shared between the compiler plugin and plugins generating source, for example the plugin: http://mojo.codehaus.org/was6-maven-plugin/ejbdeploy-mojo.html#jdkComplianceLevel needs to access the target source level. Today this will have to be pulled from the compiler plugin configuration "manually" - and not through some common shared configuration/API.

    This has also been discussed on IRC: http://dev.rectang.com/logs/codehaus/%23maven/20080723.html&nbsp; (5:17 PM and onwards).