Skip to end of metadata
Go to start of metadata

This is a working document on the topic of converting from gradle to maven.

If you add items / questions / etc, please tag and maybe color it (e.g., <dl>...) to make it easier to track things.  Open questions / issues are in red.

At a high level the motivations are:

  • improve consumability of Edgent by users
    • add publishing of signed Edgent binary release artifacts (JAR, WAR) in maven repos
    • ultimately cease creating signed binary release bundles (tgz)
  • simplify the build tooling
    • the gradle scripting has some non-standard / complicated processing either out of ignorance or to achieve certain things
      • TODO add the "why the complications" note here.  (manifest classpath, binary bundle tgz with transitive deps, ...)
      • Suspect that to simplify maven based tooling we'd need to eliminate some of those things (which would also simplify gradle stuff)

 

Chris Dutz created PR-309 for working on a maven based system.

PR-309 now includes a maven wrapper (mvnw).  Set JAVA_HOME and then use "mvnw" instead of "mvn" below and you don't have to manually install maven.

Or, to manually install maven

https://maven.apache.org/download.cgi

https://maven.apache.org/install.html

Maven concepts: lifecycles, phases, goals, plugins

High level building and testing

  • mvn clean install       # builds and tests   add -DskipTests to omit testing,  --fail-at-end for "continue"-like
  • mvn clean package   # a bit fewer phases than install
  • mvn clean compile    # even fewer/faster
  • mvn site:site site:staging  # see target/stagingindex.html
  • mvn dependency:copy-dependencies  # copies all dep jars into various target/dependencies
  • see below for building for multiple target platforms (j7,android)

 

Things I need to educate myself on / all need to understand:

  • how do we generate an official signed source release bundle (tgz)
    • Today
      • gradlew release                # also builds binary tgz bundle
      • gradlew signAll                 # generates .asc, .md5, .sha
      • we have scripts to stage / publish to ASF svn repos
    • Mvn
      • mvn install -Papache-release
      • see item #9 below
  • how do we generate a rat report for source release artifacts
    • Today
      • gradlew rat                       # we have customizations that exclude various things
    • Mvn
      • Automatically generated as part of the normal maven build - build fails if anything isn't OK
  • how do we generate junit and jacoco HTML reports
    • Today
      • gradlew reports
        • reports/tests/overview-summary.html    # JUnit
        • reports/coverage/index.html                  # Code coverage
    • Mvn
      • Normal Text and XML reports are genereated in the test-phase
      • HTML reports are created by the site plutin: mvn clean install site:site (The site is generated in target/site directory)
  • how do we just compile (no test build / execution)
    • Today
      • gradlew assemble
      Mvn
      • Compile tests but don't execute them: mvn install -DskipTests
      • Don't compile tests: mvn install -Dmaven.test.skip=true
  • how do we run all/some java8 tests (and build if needed)
    • Today
      • gradlew test --continue                                          # all non-Manual tests
      • gradlew <project>:test --tests *.SomeTest
      • gradlew <project>:test --tests *.SomeTest.someMethod
      • gradlew <project>:cleanTest <project>:test           # force rerun
      • gradlew connectors:kafka:test --tests '*.*Manual'    # after setup&start kafka server
      • gradlew connectors:mqtt:test --tests '*.*Manual'    # after setup&start mqtt server
      • gradlew connectors:jdbc:test                                 # after setup DERBY_HOME
    • Mvn
      • mvn test  --fail-at-end  
      • mvn -pl <project> -Dtest=**/SomeTest test
      • mvn -pl <project> -Dtest=**/SomeTest#someMethod test
      • There is no such concept as continuing a run ...
      • mvn -pl connectors-kafka -Dtest=**/*Manual test (Not 100% sure, what happens in the Gradle build here)
      • mvn -pl connectors-mqtt -Dtest=**/*Manual test (Not 100% sure, what happens in the Gradle build here)
      • mvn -pl connectors-jdbc test (Not 100% sure, what happens in the Gradle build here)

how do Edgent runtime developers develop w/Eclipse

    • Today
      • gradlew setupExternalJars                        # populates ../externalJars/... with ext deps
      • have .classpath files in repo                      # these reference ../externalJars/...
      • then just usual java project auto-build
      • can run test with <project>Test > main > Run as > JUnit test
    • Mvn
      • In IntelliJ all just works
      • have Eclipse mvn plugin (bundled in Eclipse IDE for Java Developers - Neon/4.6.1)
      • pom.xml > Run As > Maven > Existing Maven projects
      • The IDE is automatically configured by the settings in pom.xml and the normal java auto build should continue to work.
        • <dlaboss> seems to work fine now since the Eclipse .project/.classpath files have been removed in the PR

how do we build and access javadoc

    • Today
      • gradlew assemble   or  gradlew release or  gradlew aggregateJavadoc
      • builds javadoc into build/distributions/docs/javadoc
    • Mvn
      • Automatically built by the maven site plutin: mvn clean install site:site (The site is generated in target/site directory)

how do we build and test for java8 tgt

    • Today
      • gradlew test                                      # build all deps, run all non-Manual tests
    • Mvn
      • mvn test (Would compile and run the tests) 

how do we build and test for java7 tgt

    • Today
      • gradlew assemble                             # generates java8, java7, android tgt class/jar files
      • gradlew test7Compile                       # build appropriate tests
      •  #  muck with EVs to setup to use java7 JRE
      •  # ...
      • gradlew test7Run                              # run w/java7
      • gradlew tests7Reports                      # JUnit and coverage reports
    • Mvn
      • Generate a binary assembly for Java8 or Java7
      • Create a maven profile for java7 (unfortunately I haven't quite understood the differences of the Java7 and the Android build. There is quite a lot of code that doesn't compile for Java7)
        • <dlaboss> Edgent is written to a restricted Java8 - basically only Java7 + Lambdas.  The gradle/ant machinery for building for java7 is to build with java8 and then generate java7 jars using "retrolambda".  See platform/{java7,andriod}/build.gradle (and leveraged build.xml).
      • <dlaboss> the current state of the PR (10-July) is to use mvn toolchains and Profiles to include building/testing alternate platforms
        •  create a toolchains-local.xml file by cloning the toolchains-travis.xml file in the repo and setting the appropriate j8-jdk-home and j7-jdk-home locations on your system
          • e.g., on my osx system:
            • j8 jdkHome:  /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home
            • j7 jdkHome:  /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home
        • then build/test via: mvn -t toolchains-local.xml -Djava8.home=$JAVA_HOME clean install -Pplatform-java7,platform-android -DskipTests  # omit -DskipTests to run tests

how do we build and test for android tgt

    • Today
      • android tgt is basically the java7 artifacts plus one or two more jars (all included with java8 build)
      • we don't have tests / test runs for the android tgt - ~not an issue since it includes the java7 jars, which we do test
    • Mvn
      • Generate a binary assembly for Android
      • Create a maven profile for Android (unfortunately I haven't quite understood the differences of the Java7 and the Android build. There is quite a lot of code that doesn't compile for Java7)
      • <dlaboss> see java7 above

A not quite 2 hour introduction video to Maven:

https://vimeo.com/167857327

Issues / TODOs - things without strikethrough still need attention

  1. (post-merge is OK) the PR adds html comments (ALv2 headers) to many markdown file (e.g., README.md).  As it turns out, that further confuses the Eclipse "wikitext" editor and the preview panel now shows blank for the file.  sheesh, as if some of the braindead behaviors of that editor weren't already enough (smile)  SOLUTION: install the "Markdown text editor" from the Eclipse marketplace and adjust file associations accordingly.  Open the Markdown View to see the preview.  Probably need to put something about this in DEVELOPMENT.md for Eclipse.
  2. (done) Need to put some doc / warning somewhere as we get lots of these warnings and 
    1. mvn clean install

      [INFO] Scanning for projects...

      [WARNING]

      [WARNING] Some problems were encountered while building the effective model for org.apache.edgent.analytics:edgent-analytics-sensors:jar:1.2.0-SNAPSHOT

      [WARNING] 'version' contains an expression but should be a constant. @ org.apache.edgent.analytics:edgent-analytics-sensors:${edgent.version}, /Users/dlaboss/git/incubator-edgen\

      t-4/analytics/sensors/pom.xml, line 31, column 12

    2. Chris: Unfortunately we have to live with that. Using variables for the versions can cause problems. In our case I made sure they don't. But we need that in order to create the java 8 and java 7 packages. But Maven tends to be over careful

  3. (done) the connector-kafka pom isn't preserving the non-transitive dep specs that are noted in connectors-kafka/build.gradle (see commentary there)
    1. the effects are that a binary bundle (assembly) or flat dir of ext dependencies will have extra things in it.  Ditto when creating an uber-jar for an Edgent app (or sample)
    2. the pom's specs can be made to match the non-transitive gradle decls/behavior by adding type and exclusion decls like the following.
      1. (17.07.2017) Done  (not done; was only added to kafka_2.10)
      2. (01Aug) still need the exclusions on ALL of the non-Edgent dependencies, not just kafka_2.10
      3. (01Aug) committed changes to add it to ALL
  4. (done) same non-transitive spec issue for the analog to android/hardware/build.gradle in android/android/hardware/pom.xml
    1. dlaboss: actually this is a case where "provided" is the correct dependency config.  The platform where the Edgent app is deployed should have the dependent "com.google.android:android:4.1.1.4" jar/deps.
  5. time observations:
    1. it is much slower (minutes slower) to build with mvn than gradle.  not an issue when developing in an IDE I guess.
    2. 0m35  time mvn compile  # any "install" includes populating the mvn repo
    3. 0m38  time mvn clean compile   # mvn => no build avoidance I think
    4. 3m46   time mvn package -DskipTests -Papache-release  # j8, includes javadoc at least to a certain level, created source bundle, not signed
    5. 2m47  time mvn clean install -DskipTests   # only builds j8
    6. 2m38  time mvn install -DskipTests             # that's right after the prior mvn cmd
    7. 7m9    time mvn -t toolchains-local.xml -Djava8.home=$JAVA_HOME install -Pplatform-java7,platform-android -DskipTests  #   ~similar to "gradlew assemble" I think (but gw includes javadoc)
    8. 10m8  time mvn -t toolchains-local.xml -Djava8.home=$JAVA_HOME install -Pplatform-java7,platform-android,apache_release -DskipTests    #  ~similar to "gradlew release" (gw doesn't run rat, but does create binary tgz and javadoc)
    9. 0m39   time gradlew cleanAll assemble  # includes javadoc gen
    10. 0m7     time gradlew assemble
    11. 0m57   time gradlew release  # j8/j7/android , includes source & binary tgz but not signed, no rat
    12. TESTING...
    13. 10m51   time gradlew test   # j8
    14. 10m56  time gradlew cleanAll test   # j8
    15. 13m16  time mvn clean install  # j8
    16. junit and coverage html reports?
    17. ability to run j7 tests without redoing j8 tests?
  6. (done) Apache Edgent: Samples: Szenarios  - spelling
  7. (done) Apache Edgent: Runtimes - use to be singular "runtime"
    1. (11.07.2017) Well I did notice, that there were multiple runtimes so I thought better name the artifact that way (I would suggest to rename the directory to "runtimes" as well as the other directories containing multiple things all have plural names)
    2. hmm... unlike connectors,providers,samples, "runtime" is the aggregator for various parts of "the" core Edgent runtime (singular) - though some of the pieces of it are optional hence the finer grained jars.  The various runtime sub-projects aren't different implementations of "the" runtime.  I would feel better if we didn't change this level of naming at this time.  Can you live with that?  It can always be revisited after the conversion is complete and the dust settles.  Note, at least one person suggested consolidating various projects - e.g., maybe a single api jar and single runtime jar.
    3. it's been renamed back to runtime
  8. 31July - importing maven projects into Eclipse
    1. (started with new git clone including PR-309; didn't build with mvn before creating new Eclipse workspace and importing existing maven projects)
    2. (post-merge is OK) ~100 pom warnings for imported maven projects in Eclipse
      1. lots of unnecessary version id specs (same as parent - remove them?
      2. some plugin version overrides (can squelch them)
    3. (prefer pre-merge fix) 7 errors for android&distribution poms - but otherwise all seems fine
      1. the import (of all projects/poms) generated a "Setup Maven plugin connectors" dialog. 7 Errors: 1 for maven-toolchain-plugin, 3 for retrolambda plugin process:main, 3 for retrolambda process:test.
  9. source release bundle - names/content
    1. via mvn clean install -Papache-release -DskipTests
      1. see http://maven.apache.org/plugins/maven-gpg-plugin/usage.html for passphrase config info
    2. mvn generated name is currently: edgent-parent-1.2.0-SNAPSHOT-source-release.zip
      1. (17.07.2017) Done should now be "apache-edgent-incubating-1.2.0-SNAPSHOT-source-release.zip"
    3. LICENSE:(need to remove no longer bundled jquery... code), NOTICE:OK
      1. (18.07.2017) Done
    4. formats:only zip now?
      1. (18.07.2017) Now also tar.gz is generated.
    5. checksums:missing md5 and sha?
      1. (18.07.2017) These are usually generated by Nexus when uploading artifacts to Nexus
    6. (post-merge is OK) README: needs updating for how to build (to the point of installing Edgent jars in local mvn repo? j8,j7, and android?) and maybe update tie in to the getting started doc (that will need significant rework)
    7. all tests pass when building from source bundle?
      1. I seem to recall Chris disabled some SSL releated tests?  Others?
        1. (18.07.2017) Think I had some troubles in the beginning, but I think all are on again ... at least all that were on when I started.
        2. <dl> 01Aug all tests are passing (build/run from repo AND from source-bundle) and only skipped tests are correctly:  JDBC and Kafka connectors and one in File connector
      2. (done) do the manual kafka and jdbc tests pass?
        1. <dl> 01Aug - kafka manual tests pass (setup kafka, the "cd connectors-kafka;  mvn test -Dtest=**/*Manual")
        2. <dl> 01Aug - jdbc tests fail - easily fixed by adding dependency to derby and letting mvn download it.  Now JDBC connector tests always run - no longer need manual run.  Added change request to the PR.
        3. <dl> 01Aug - committed the fix

      3. related: samples item #15 (can't merge until we have sample handling all setup & working)
    8. (done) JAVA_SUPPORT.md: needs updating for j7&android
    9. (post-merge is OK) DEPENDENCIES: status of this?  mvn generated. docs transitive deps, but only has codahale metrics-core.
    10. (post-merge is OK) src/site/*: status of this?
      1. (18.07.2017) Just a dummy for now, but this is the place where some detailed documentation with graphics charts and so on could go to and could be automatically generated in the site of the project (automatically updating the website)
    11. (post-merge is OK) Failing Tests.txt: status of this?
      1. (18.07.2017) All automated tests are successful will have to check the manual ones as they seem to have been dropping below my radar.
    12. (post-merge is OK) TODO: file JIRA to track cleaning up other cruft from source bundle (and repo): gradle stuff, appropriate source and binary bundle licensing stuff, PROBLEMS.md
  10. binary release jars,wars - names/content, ...   for j7 and android too
    1. name, MANIFEST/LICENSE (missing the d2 stuff for war), MANIFEST/NOTICE (missing the ibm-contribs one for most jars and the war)
    2. the WAR now contains edgent-jars.  Is that tolerable/OK?
    3. staging at ASF nexus
    4. publishing to maven-central
  11. what needs to be edited to do a non-SNAPSHOT release? (see related info under "build scripts" below)
    1. (11.07.2017) I wrote all of this down for the Flex project: Releasing FlexJS with Maven
  12. All of the other "documentation" noted above (RM guide, DEVELOPMENT.md, source README, getting started, website downloads)
    1. (11.07.2017) I would update this as soon as the technical stuff is handled
    2. (done) reminder to update JAVA_SUPPORT.md.  The mvn mods result in java7 platform now having all of the Edgent components present in java8.  Ditto for Android (minus the usual things).
    3. (done) RELEASE_NOTES (summary of PR309 changes), CONTRIBUTORS
    4. DEVELOPMENT.md, source README, getting started (need to preserve old too), website downloads, RM Guide (probably need to preserve old for now)
  13. updating any of the release mgmt build scripts (noted in the RM-guide) to deal with switch to maven and/or changes in binary artifacts / bundling
    1. hmm... the current scripts learn the version-id of the release from the gradle.properties file.  
    2. related, all of the pom.xml files have the version-id in them.  Maybe create a new properties file with the version-id info that can also be used to drive a new script that edits the poms? 
    3. (11.07.2017) With maven there should be no need for any scripts like this as Maven provides all means to do releases out of the box.
  14. binary release BUNDLE
    1. <dl> ASSERTION: no longer release binary bundle (with or without ext deps)
      1. (post-merge is OK) cleanup binary-release/{LICENSE, NOTICE, README} in the github repo
      2. MUST have a easy way to get/download all edgent binary artifacts and ext deps
        1. get-edgent-jars.sh - (need to fix kafka poms - item #3; how to handle WAR? related item #15)
    2. otherwise, have correct bundle name, content, ...
    3. (11.07.2017) It should be producing a distribution in the distribution module (quite new module)  (<dl> it's ok to retain distribution module and we just not use it at this time)
  15. samples
    1. sent @dev email on 19Jul http://mail-archives.apache.org/mod_mbox/incubator-edgent-dev/201707.mbox/ajax/%3C29D9F1D6-EC63-40CE-890A-341D479D76A6%40apache.org%3E
    2. do not include samples binary jar in a release
    3. add a separate samples source bundle to the release - "apache-edgent-samples-...".  No samples in the main source bundle
      1. TODO "samples" issues/comments in top-level pom.xml and platforms/java7/pom.xml
    4. (done) have non-Edgent-parent-dependent pom(s), supporting uber jar and j8, j7, android building - see samples/README.md
    5. (done) include package-app.sh  (see samples/README.md)
    6. include get-edgent-jars.sh (included but see item #14.a.ii above)
    7. (done) updated doc/READMEs for all samples;  see new samples/README.md for overview, and how to build and run
    8. samples build with mvn and run via uber-jar
      1. <dl> 09Aug all build/run except
        1. any that use the DevelopmentProvider/Edgent console.  e.g., (cd samples/topology; ./run-sample.sh DevelopmentSample)
        2. kafka sample subscribers not working/reporting rcvd tuples?
    9. (done) in lieu of, or in addition to, a Edgent application archtype, add a simple/cloneable app template sample: samples/template
    10. 09Aug <dl> 2nd round of commits to flesh out running of all samples and refactoring of test/svt
    11. 08Aug Chris suggests that migrating the samples to a separate repo may help achieve the goals.
  16. javadoc
    1. presumably it lacks the "groups" organization that the current doc has.  is that OK?  is it otherwise fine?
    2. e.g. seeing (in my modified "samples" handling poms)

  17. mvn archetype for Edgent app
    1. (post-merge is OK)  how will a user get the code supporting this?
  18. build requires Java7 JDK?
    1. Chris mentioned working on eliminating that requirement.  Not sure if Java7 is only required for -Pplatform-java7 and then only for test execution.
  19. Jenkins
    1. (post-merge is OK) setup for auto nightly? builds via Jenkins - Chris, what's needed for this?
    2. (post-merge is OK) related: consider having travis PR validation only do j8 (to speed things up)
  20. (done) websocket connector
    1. component/package/jar names were a mess before the mvn work (my bad)
    2. retain package,class names; filed JIRAs for subsequent cleanup (EDGENT-427, EDGENT-426)
    3. rationalize jar/component names with mvn work - only change the jar names once
      1. simplify
      2. use "websocket" as the high level connector name
      3. it is implicitly a websocket client connector; make "server" the special case (we don't provide/release a server-side connector)
      4. other than external jetty jars, goal is to have a single jar for the connector - edgent-connectors-websocket-<ver>.jar
        1. connector will implicitly be JSR356/javax.websocket based (no javax in the names)
        2. connector will implicitly be Jetty javax.websocket based

 

 

1.1.0 Jar30June-PR-309WantNotes
edgent.connectors.wsclient-javax.websocketedgent-connectors-websocket-wsclient-<ver>

edgent-connectors-websocket-<ver>

Contains o.a.e.c.wsclient.javax.websocket.Jsr356WebSocketClient

JSR356/javax.websocket based impl of the connector

Directly used by Edgent app, needs to add to classpath

Wish class was o.a.e.c.websocket.WebSocketStreams

edgent.connectors.wsclientedgent-connectors-wsclient-<ver>

edgent-connectors-websocket-base<ver>

<TBD remove jar>

 

Contains o.a.e.c.wsclient.WebSocketClient

The generic Edgent WebSocket Connector Client API (independent of JSR356/javax.websocket API)

Motivation was there are other non-javax.websocket websocket client impls out there.

Edgent app needs to add to classpath

Wish class was o.a.e.c.websocket.api.IWebSocketStreams / gone

TBD move into edgent-connectors-websocket component/jar or completely eliminate

edgent.javax.websocketedgent-connectors-websocket-<ver>

edgent-connectors-websocket-misc-<ver>

<TBD remove jar>

Things a JSR356/javax.websocket based impl needs (invisible to Edgent client app)

Defines a JSR356 SSL Hackery interface - o.a.e.javax.websocket.EdgentSslContainerProvider

try to remove this jar in the future (e.g., by combining into the other jar)?

1.1.0 manifest adds external JSR356/javax.websocket API definition jar to classpath

Wish class was o.a.e.c.websocket.runtime.EdgentSslContainerProvider

TBD move into edgent-connectors-websocket component/jar

javax.websocket-clientedgent-connectors-websocket-client-<ver>

edgent-connectors-websocket-jetty-<ver>

<TBD remove jar>

Things for a Jetty-based impl of JSR356/javax.websocket (invisible to Edgent client app)

Has impl for the JSR356 SSL Hackery - o.a.e.javax.websocket.impl.EdgentSslContainerProviderImpl

1.1.0 manifest adds Jetty jars to classpath; service decl for the SSL hackery impl

Wish class were o.a.e.c.websocket.runtime.jetty.EdgentSslContainerProviderImpl

TBD move into edgent-connectors-websocket component/jar

javax.websocket-serveredgent-connectors-websocket-server-<ver>edgent-connectors-websocket-server-<ver>Just for websocket client testing. Not part of binary release.

 

 

 

 

 

 

 


  • No labels