Skip to end of metadata
Go to start of metadata

This is a working document on the topic of converting from gradle to maven.

If you add items / questions / etc, please tag and maybe color it (e.g., <dl>...) to make it easier to track things.  Open questions / issues are in red.

At a high level the motivations are:

  • improve consumability of Edgent by users
    • add publishing of signed Edgent binary release artifacts (JAR, WAR) in maven repos
    • ultimately cease creating signed binary release bundles (tgz)
  • simplify the build tooling
    • the gradle scripting has some non-standard / complicated processing either out of ignorance or to achieve certain things
      • TODO add the "why the complications" note here.  (manifest classpath, binary bundle tgz with transitive deps, ...)
      • Suspect that to simplify maven based tooling we'd need to eliminate some of those things (which would also simplify gradle stuff)

 

Chris Dutz created PR-309 for working on a maven based system.

To run maven from the cmdline

https://maven.apache.org/download.cgi

https://maven.apache.org/install.html

Maven concepts: lifecycles, phases, goals, plugins

High level building and testing

  • mvn clean install       # builds and tests   add -DskipTests to omit testing
  • mvn clean package   # a bit fewer phases than install
  • mvn site:site site:staging  # see target/stagingindex.html
  • mvn dependency:copy-dependencies  # copies all dep jars into various target/dependencies

 

Things I need to educate myself on / all need to understand:

  • how do we generate an official signed source release bundle (tgz)
    • Today
      • gradlew release                # also builds binary tgz bundle
      • gradlew signAll                 # generates .asc, .md5, .sha
      • we have scripts to stage / publish to ASF svn repos
    • Mvn
      • mvn install -Papache-release
      • <dlaboss>...
        • creates target/edgent-parent-1.2.0-SNAPSHOT-source-release.zip
        • see http://maven.apache.org/plugins/maven-gpg-plugin/usage.html for passphrase config info
          • woo hoo generates *-source-release.zip.asc 
          • what about generating md5 and sha sigs?  Sent dev@ mail
        • configuring the zip contents?  name?  zip vz tgz?  # maven-assembly-plugin.  Sent dev@ email
          • "edgent-parent"?   missing "incubating"
          • currently includes
            • gradle 'build' dirs if present, probably Eclipse 'bin' as well
            • src/site (from mvn site)
            • connectors/jdbc/JdbcStreamsTestDb
            • DEPENDENCIES - hmm contains Metrics Core (dropwizard) decl.  What's special about that? vs slf4j.  or should slf4j also be there?
            • api/topology/classes/...   classes.in/...  classes.out/...   # stuff from java7 build w/retrolambda
            • .gradle/...  # gradle cache stuff
            • missing Eclipse .classpath files?
  • how do we generate a rat report for source release artifacts
    • Today
      • gradlew rat                       # we have customizations that exclude various things
    • Mvn
      • Automatically generated as part of the normal maven build (currently disabled because the maven+gradle parallel system would require a lot more exceptions ...)
  • how do we generate junit and jacoco HTML reports
    • Today
      • gradlew reports
        • reports/tests/overview-summary.html    # JUnit
        • reports/coverage/index.html                  # Code coverage
    • Mvn
      • Normal Text and XML reports are genereated in the test-phase
      • HTML reports are created by the site plutin: mvn clean install site:site (The site is generated in target/site directory)
  • how do we just compile (no test build / execution)
    • Today
      • gradlew assemble
      Mvn
      • Compile tests but don't execute them: mvn install -DskipTests
      • Don't compile tests: mvn install -Dmaven.test.skip=true
  • how do we run all/some java8 tests (and build if needed)
    • Today
      • gradlew test --continue                                          # all non-Manual tests
      • gradlew <project>:test --tests *.SomeTest
      • gradlew <project>:test --tests *.SomeTest.someMethod
      • gradlew <project>:cleanTest <project>:test           # force rerun
      • gradlew connectors:kafka:test --tests '*.*Manual'    # after setup&start kafka server
      • gradlew connectors:mqtt:test --tests '*.*Manual'    # after setup&start mqtt server
      • gradlew connectors:jdbc:test                                 # after setup DERBY_HOME
    • Mvn
      • mvn test
      • mvn -pl <project> -Dtest=**/SomeTest test
      • mvn -pl <project> -Dtest=**/SomeTest#someMethod test
      • There is no such concept as continuing a run ...
      • mvn -pl connectors-kafka -Dtest=**/*Manual test (Not 100% sure, what happens in the Gradle build here)
      • mvn -pl connectors-mqtt -Dtest=**/*Manual test (Not 100% sure, what happens in the Gradle build here)
      • mvn -pl connectors-jdbc test (Not 100% sure, what happens in the Gradle build here)

how do Edgent runtime developers develop w/Eclipse

    • Today
      • gradlew setupExternalJars                        # populates ../externalJars/... with ext deps
      • have .classpath files in repo                      # these reference ../externalJars/...
      • then just usual java project auto-build
      • can run test with <project>Test > main > Run as > JUnit test
    • Mvn
      • In IntelliJ all just works
      • have Eclipse mvn plugin (bundled in Eclipse IDE for Java Developers - Neon/4.6.1)
      • pom.xml > Run As > Maven ....
      • The IDE is automatically configured by the settings in pom.xml and the normal java auto build should continue to work.
        • <dlaboss> well minimally it seems like need to change project configs to be a Maven project
        • and then tried updating existing .classpath files to a "maven.pomderived" form... but still no dice yet.
        • Issue with having a .project level .classpath... but sub-project pom.xml (build.gradle, build.xml)?

how do we build and access javadoc

    • Today
      • gradlew assemble   or  gradlew release or  gradlew aggregateJavadoc
      • builds javadoc into build/distributions/docs/javadoc
    • Mvn
      • Automatically built by the maven site plutin: mvn clean install site:site (The site is generated in target/site directory)

how do we build and test for java8 tgt

    • Today
      • gradlew test                                      # build all deps, run all non-Manual tests
    • Mvn
      • mvn test (Would compile and run the tests) 

how do we build and test for java7 tgt

    • Today
      • gradlew assemble                             # generates java8, java7, android tgt class/jar files
      • gradlew test7Compile                       # build appropriate tests
      •  #  muck with EVs to setup to use java7 JRE
      •  # ...
      • gradlew test7Run                              # run w/java7
      • gradlew tests7Reports                      # JUnit and coverage reports
    • Mvn
      • ? build test7Compile tests
      • ? run test7 tests
      • ? generate test7 reports
    • Mvn
      • Generate a binary assembly for Java8 or Java7
      • Create a maven profile for java7 (unfortunately I haven't quite understood the differences of the Java7 and the Android build. There is quite a lot of code that doesn't compile for Java7)
        • <dlaboss> Edgent is written to a restricted Java8 - basically only Java7 + Lambdas.  The gradle/ant machinery for building for java7 is to build with java8 and then generate java7 jars using "retrolambda".  See platform/{java7,andriod}/build.gradle (and leveraged build.xml).

how do we build and test for android tgt

    • Today
      • android tgt is basically the java7 artifacts plus one or two more jars (all included with java8 build)
      • we don't have tests / test runs for the android tgt - ~not an issue since it includes the java7 jars, which we do test
    • Mvn
      • ?  generate the android tgt class/jar files
    • Mvn
      • Generate a binary assembly for Andoid
      • Create a maven profile for Andoid (unfortunately I haven't quite understood the differences of the Java7 and the Android build. There is quite a lot of code that doesn't compile for Java7)
        • <dlaboss> see java7 above

A not quite 2 hour introduction video to Maven:

https://vimeo.com/167857327

Generating and publishing binary java8, java7, android artifacts via mvn-repo

wish gradle system still worked. then could merge maven work w/o requiring everything in place for an immediate switchover

    • reminder - need to overhaul
      • DEVELOPMENT.md
      • source release bundle README (how to build)
      • Website downloads page
      • Website getting started page
      • overview-summary.html for "building Edgent Applications" (at bottom) - more below
      • release mgmt guide and any related pages - e.g., updating website javadoc

need correct NOTICES in jar/war files - some (most but not all) need the one with IBM notice

correct WAR content?  now includes Edgent JARs?

now a 2nd WAR?

The LICENSE link in the generated "site" is the std AL2 version.  Needs to be the correct Edgent source (bundle) license version.

Currently the generated JAR/WAR lack an "edgent" prefix,  e.g., api-topology-1.2.0-SNAPSHOT.jar.  How important is it to add an "apache-edgent" (or "edgent") prefix to the name (artifactId) before we start publishing these in mvn repos?  <dlaboss> discussion on dev@ resulted in Chris adding "edgent-" prefix in the mvn PR.  Reminder that various doc/scripts will require update.

continuous PR-merge integration for java8 and java7

    • ? travis.yml  # gradle version does build and test for both tgts

publishing signed convenience binary artifacts for java8 tgt (with proper LICENSE/NOTICE, etc files in the jar/war)

    • ?mvn   snapshots vs releases

publishing signed convenience binary artifacts for java7 tgt (with proper LICENSE/NOTICE, etc files in the jar/war)

    • ?mvn

publishing signed convenience binary artifacts for android tgt (with proper LICENSE/NOTICE, etc files in the jar/war)

    • ?mvn

Implications for users to use these mvn-repo artifacts?

    • must use mvn or gradle to build their Edgent apps?
    what will they need to do to be able to deploy their app to a "device"
      • how to get their hands on the Edgent jars and their transitive deps so they can package into their own tgz
      • on their own for setting CLASSPATH for their 
    • java8 vs java7 vs android?

Implications for Edgent samples?

    • should we cease generating/distributing binary samples jar?  instead scripts that use mvn to build/run the sample src?
    • or just how to fix sample scripts so they work against binary samples jar? (and no resident Edgent jars and transitive deps)

 

Misc

  1. the PR adds html comments (ALv2 headers) to many markdown file (e.g., README.md).  As it turns out, that further confuses the "wikitext" editor and the preview panel now shows blank for the file.  sheesh, as if some of the braindead behaviors of that editor weren't already enough (smile)  SOLUTION: install the "Markdown text editor" from the Eclipse marketplace and adjust file associations accordingly.  Open the Markdown View to see the preview.  Probably need to put something about this in DEVELOPMENT.md for Eclipse.
  2. Need to put some doc / warning somewhere as we get lots of these warnings and 
    1. mvn clean install

      [INFO] Scanning for projects...

      [WARNING]

      [WARNING] Some problems were encountered while building the effective model for org.apache.edgent.analytics:edgent-analytics-sensors:jar:1.2.0-SNAPSHOT

      [WARNING] 'version' contains an expression but should be a constant. @ org.apache.edgent.analytics:edgent-analytics-sensors:${edgent.version}, /Users/dlaboss/git/incubator-edgen\

      t-4/analytics/sensors/pom.xml, line 31, column 12

    2. Chris: Unfortunately we have to live with that. Using variables for the versions can cause problems. In our case I made sure they don't. But we need that in order to create the java 8 and java 7 packages. But Maven tends to be over careful


 

 

 

 

  • No labels