Versions Compared

Old Version 45

changes.mady.by.user Dale LaBossiere

Saved on

compared with

New Version 46

changes.mady.by.user Dale LaBossiere

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. (post-merge is OK) the PR adds html comments (ALv2 headers) to many markdown file (e.g., README.md).  As it turns out, that further confuses the Eclipse "wikitext" editor and the preview panel now shows blank for the file.  sheesh, as if some of the braindead behaviors of that editor weren't already enough (smile)  SOLUTION: install the "Markdown text editor" from the Eclipse marketplace and adjust file associations accordingly.  Open the Markdown View to see the preview.  Probably need to put something about this in DEVELOPMENT.md for Eclipse.
  2. (done) Need to put some doc / warning somewhere as we get lots of these warnings and 

    1. mvn clean install

      [INFO] Scanning for projects...

      [WARNING]

      [WARNING] Some problems were encountered while building the effective model for org.apache.edgent.analytics:edgent-analytics-sensors:jar:1.2.0-SNAPSHOT

      [WARNING] 'version' contains an expression but should be a constant. @ org.apache.edgent.analytics:edgent-analytics-sensors:${edgent.version}, /Users/dlaboss/git/incubator-edgen\

      t-4/analytics/sensors/pom.xml, line 31, column 12

    2. Chris: Unfortunately we have to live with that. Using variables for the versions can cause problems. In our case I made sure they don't. But we need that in order to create the java 8 and java 7 packages. But Maven tends to be over careful

  3. (done) the connector-kafka pom isn't preserving the non-transitive dep specs that are noted in connectors-kafka/build.gradle (see commentary there)
    1. the effects are that a binary bundle (assembly) or flat dir of ext dependencies will have extra things in it.  Ditto when creating an uber-jar for an Edgent app (or sample)
    2. the pom's specs can be made to match the non-transitive gradle decls/behavior by adding type and exclusion decls like the following.
      1. Code Block
            <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.10</artifactId>
      <version>0.8.2.2</version>
      <type>jar</type>
      <exclusions>
        <exclusion>
          <groupId>*</groupId>
          <artifactId>*</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
      2. (17.07.2017) Done  (not done; was only added to kafka_2.10)
      3. (01Aug) still need the exclusions on ALL of the non-Edgent dependencies, not just kafka_2.10
      4. (01Aug) committed changes to add it to ALL
  4. (done) same non-transitive spec issue for the analog to android/hardware/build.gradle in android/android/hardware/pom.xml
    1. dlaboss: actually this is a case where "provided" is the correct dependency config.  The platform where the Edgent app is deployed should have the dependent "com.google.android:android:4.1.1.4" jar/deps.
  5. time observations:
    1. it is much slower (minutes slower) to build with mvn than gradle.  not an issue when developing in an IDE I guess.
    2. 0m35  time mvn compile  # any "install" includes populating the mvn repo
    3. 0m38  time mvn clean compile   # mvn => no build avoidance I think
    4. 3m46   time mvn package -DskipTests -Papache-release  # j8, includes javadoc at least to a certain level, created source bundle, not signed
    5. 2m47  time mvn clean install -DskipTests   # only builds j8
    6. 2m38  time mvn install -DskipTests             # that's right after the prior mvn cmd
    7. 7m9    time mvn -t toolchains-local.xml -Djava8.home=$JAVA_HOME install -Pplatform-java7,platform-android -DskipTests  #   ~similar to "gradlew assemble" I think (but gw includes javadoc)
    8. 10m8  time mvn -t toolchains-local.xml -Djava8.home=$JAVA_HOME install -Pplatform-java7,platform-android,apache_release -DskipTests    #  ~similar to "gradlew release" (gw doesn't run rat, but does create binary tgz and javadoc)
    9. 0m39   time gradlew cleanAll assemble  # includes javadoc gen
    10. 0m7     time gradlew assemble
    11. 0m57   time gradlew release  # j8/j7/android , includes source & binary tgz but not signed, no rat
    12. TESTING...
    13. 10m51   time gradlew test   # j8
    14. 10m56  time gradlew cleanAll test   # j8
    15. 13m16  time mvn clean install  # j8
    16. junit and coverage html reports?
    17. ability to run j7 tests without redoing j8 tests?
  6. (done) Apache Edgent: Samples: Szenarios  - spelling
  7. (done) Apache Edgent: Runtimes - use to be singular "runtime"
    1. (11.07.2017) Well I did notice, that there were multiple runtimes so I thought better name the artifact that way (I would suggest to rename the directory to "runtimes" as well as the other directories containing multiple things all have plural names)
    2. hmm... unlike connectors,providers,samples, "runtime" is the aggregator for various parts of "the" core Edgent runtime (singular) - though some of the pieces of it are optional hence the finer grained jars.  The various runtime sub-projects aren't different implementations of "the" runtime.  I would feel better if we didn't change this level of naming at this time.  Can you live with that?  It can always be revisited after the conversion is complete and the dust settles.  Note, at least one person suggested consolidating various projects - e.g., maybe a single api jar and single runtime jar.
    3. it's been renamed back to runtime
  8. 31July - importing maven projects into Eclipse
    1. (started with new git clone including PR-309; didn't build with mvn before creating new Eclipse workspace and importing existing maven projects)
    2. (post-merge is OK) ~100 pom warnings for imported maven projects in Eclipse
      1. lots of unnecessary version id specs (same as parent - remove them?
      2. some plugin version overrides (can squelch them)
    3. (prefer pre-merge fix) 7 errors for android&distribution poms - but otherwise all seems fine
      1. the import (of all projects/poms) generated a "Setup Maven plugin connectors" dialog. 7 Errors: 1 for maven-toolchain-plugin, 3 for retrolambda plugin process:main, 3 for retrolambda process:test.
  9. source release bundle - names/content
    1. via mvn clean install -Papache-release -DskipTests
      1. see http://maven.apache.org/plugins/maven-gpg-plugin/usage.html for passphrase config info
    2. mvn generated name is currently: edgent-parent-1.2.0-SNAPSHOT-source-release.zip
      1. (17.07.2017) Done should now be "apache-edgent-incubating-1.2.0-SNAPSHOT-source-release.zip"
    3. LICENSE:(need to remove no longer bundled jquery... code), NOTICE:OK
      1. (18.07.2017) Done
    4. formats:only zip now?
      1. (18.07.2017) Now also tar.gz is generated.
    5. checksums:missing md5 and sha?
      1. (18.07.2017) These are usually generated by Nexus when uploading artifacts to Nexus
    6. (post-merge is OK) README: needs updating for how to build (to the point of installing Edgent jars in local mvn repo? j8,j7, and android?) and maybe update tie in to the getting started doc (that will need significant rework)
    7. all tests pass when building from source bundle?
      1. I seem to recall Chris disabled some SSL releated tests?  Others?
        1. (18.07.2017) Think I had some troubles in the beginning, but I think all are on again ... at least all that were on when I started.
        2. <dl> 01Aug all tests are passing (build/run from repo AND from source-bundle) and only skipped tests are correctly:  JDBC and Kafka connectors and one in File connector
      2. (done) do the manual kafka and jdbc tests pass?
        1. <dl> 01Aug - kafka manual tests pass (setup kafka, the "cd connectors-kafka;  mvn test -Dtest=**/*Manual")
        2. <dl> 01Aug - jdbc tests fail - easily fixed by adding dependency to derby and letting mvn download it.  Now JDBC connector tests always run - no longer need manual run.  Added change request to the PR.
        3. <dl> 01Aug - committed the fix
        4. Code Block
          <!-- https://mvnrepository.com/artifact/org.apache.derby/derby -->
<dependency>
    <groupId>org.apache.derby</groupId>
    <artifactId>derby</artifactId>
    <version>10.13.1.1</version>
    <scope>test</scope>
</dependency>
 
well... right now also need to add these org.apache.rat exclusions to the top level pom.xml
  <exclude>**/derby.log</exclude> <!-- test swill.  exclude connectors/jdbc/derby.log didn't work -->
  <exclude>**/JdbcStreamsTestDb/**</exclude> <!-- test swill.  exclude connectors/jdbc/JdbcStreamsTestDb/** didn't work -->


      3. related: samples item #15 (can't merge until we have sample handling all setup & working)
    8. (post-merge is OK) JAVAdone) JAVA_SUPPORT.md: needs updating for j7&android
    9. (post-merge is OK) DEPENDENCIES: status of this?  mvn generated. docs transitive deps, but only has codahale metrics-core.
    10. (post-merge is OK) src/site/*: status of this?
      1. (18.07.2017) Just a dummy for now, but this is the place where some detailed documentation with graphics charts and so on could go to and could be automatically generated in the site of the project (automatically updating the website)
    11. (post-merge is OK) Failing Tests.txt: status of this?
      1. (18.07.2017) All automated tests are successful will have to check the manual ones as they seem to have been dropping below my radar.
    12. (post-merge is OK) TODO: file JIRA to track cleaning up other cruft from source bundle (and repo): gradle stuff, appropriate source and binary bundle licensing stuff, PROBLEMS.md
  10. binary release jars,wars - names/content, ...   for j7 and android too
    1. name, MANIFEST/LICENSE (missing the d2 stuff for war), MANIFEST/NOTICE (missing the ibm-contribs one for most jars and the war)
    2. the WAR now contains edgent-jars.  Is that tolerable/OK?
    3. staging at ASF nexus
    4. publishing to maven-central
  11. what needs to be edited to do a non-SNAPSHOT release? (see related info under "build scripts" below)
    1. (11.07.2017) I wrote all of this down for the Flex project: Releasing FlexJS with Maven
  12. All of the other "documentation" noted above (RM guide, DEVELOPMENT.md, source README, getting started, website downloads)
    1. (11.07.2017) I would update this as soon as the technical stuff is handled
    2. (done) reminder to update JAVA_SUPPORT.md.  The mvn mods result in java7 platform now having all of the Edgent components present in java8.  Ditto for Android (minus the usual things).
    3. (done) RELEASE_NOTES (summary of PR309 changes), CONTRIBUTORS
    4. DEVELOPMENT.md, source README, getting started (need to preserve old too), website downloads, RM Guide (probably need to preserve old for now)
  13. updating any of the release mgmt build scripts (noted in the RM-guide) to deal with switch to maven and/or changes in binary artifacts / bundling
    1. hmm... the current scripts learn the version-id of the release from the gradle.properties file.  
    2. related, all of the pom.xml files have the version-id in them.  Maybe create a new properties file with the version-id info that can also be used to drive a new script that edits the poms? 
    3. (11.07.2017) With maven there should be no need for any scripts like this as Maven provides all means to do releases out of the box.
  14. binary release BUNDLE
    1. <dl> ASSERTION: no longer release binary bundle (with or without ext deps)
      1. (post-merge is OK) cleanup binary-release/{LICENSE, NOTICE, README} in the github repo
      2. MUST have a easy way to get/download all edgent binary artifacts and ext deps
        1. get-edgent-jars.sh - (need to fix kafka poms - item #3; how to handle WAR? related item #15)
    2. otherwise, have correct bundle name, content, ...
    3. (11.07.2017) It should be producing a distribution in the distribution module (quite new module)  (<dl> it's ok to retain distribution module and we just not use it at this time)
  15. samples
    1. sent @dev email on 19Jul http://mail-archives.apache.org/mod_mbox/incubator-edgent-dev/201707.mbox/ajax/%3C29D9F1D6-EC63-40CE-890A-341D479D76A6%40apache.org%3E
    2. do not include samples binary jar in a release
    3. add a separate samples source bundle to the release - "apache-edgent-samples-...".  No samples in the main source bundle
      1. TODO "samples" issues/comments in top-level pom.xml and platforms/java7/pom.xml
    4. (done) have non-Edgent-parent-dependent pom(s), supporting uber jar and j8, j7, android building - see samples/README.md
    5. (done) include package-app.sh  (see samples/README.md)
    6. include get-edgent-jars.sh (included but see item #14.a.ii above)
    7. (done) updated doc/READMEs for all samples;  see new samples/README.md for overview, and how to build and run
    8. samples build with mvn and run via uber-jar
      1. <dl> 09Aug all build/run except
        1. any that use the DevelopmentProvider/Edgent console.  e.g., (cd samples/topology; ./run-sample.sh DevelopmentSample)
        2. kafka sample subscribers not working/reporting rcvd tuples?
    9. (done) in lieu of, or in addition to, a Edgent application archtype, add a simple/cloneable app template sample: samples/template
    10. 09Aug <dl> 2nd round of commits to flesh out running of all samples and refactoring of test/svt
    11. 08Aug Chris suggests that migrating the samples to a separate repo may help achieve the goals.
  16. javadoc
    1. presumably it lacks the "groups" organization that the current doc has.  is that OK?  is it otherwise fine?

    2. e.g. seeing (in my modified "samples" handling poms)

      Code Block
      [INFO] Building Apache Edgent: Providers: IoT 1.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
...
[INFO] --- maven-javadoc-plugin:2.10.4:jar (attach-javadocs) @ edgent-providers-iot ---
[INFO]
Loading source files for package org.apache.edgent.providers.iot...
Constructing Javadoc information...
Standard Doclet version 1.8.0_112
Building tree for all the packages and classes...
Generating /Users/dlaboss/git/dutz-incubator-edgent/providers/iot/target/apidocs/org/apache/edgent/providers/iot/IotProvider.html...
...
Generating /Users/dlaboss/git/dutz-incubator-edgent/providers/iot/target/apidocs/help-doc.html...
1 error
[ERROR] MavenReportException: Error while generating Javadoc:
Exit code: 1 - /Users/dlaboss/git/dutz-incubator-edgent/providers/iot/src/main/java/org/apache/edgent/providers/iot/IotProvider.java:134: error: bad use of '>'
 *     return (t, c) -> buildTopology(IotDevicePubSub.addIotDevice(t), c);
                      ^
Command line was: /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/jre/../bin/javadoc @options @packages
Refer to the generated Javadoc files in '/Users/dlaboss/git/dutz-incubator-edgent/providers/iot/target/apidocs' dir.
  17. mvn archetype for Edgent app
    1. (post-merge is OK)  how will a user get the code supporting this?
  18. build requires Java7 JDK?
    1. Chris mentioned working on eliminating that requirement.  Not sure if Java7 is only required for -Pplatform-java7 and then only for test execution.
  19. Jenkins
    1. (post-merge is OK) setup for auto nightly? builds via Jenkins - Chris, what's needed for this?
    2. (post-merge is OK) related: consider having travis PR validation only do j8 (to speed things up)
  20. (done) websocket connector
    1. component/package/jar names were a mess before the mvn work (my bad)
    2. retain package,class names; filed JIRAs for subsequent cleanup (EDGENT-427, EDGENT-426)
    3. rationalize jar/component names with mvn work - only change the jar names once
      1. simplify
      2. use "websocket" as the high level connector name
      3. it is implicitly a websocket client connector; make "server" the special case (we don't provide/release a server-side connector)
      4. other than external jetty jars, goal is to have a single jar for the connector - edgent-connectors-websocket-<ver>.jar
        1. connector will implicitly be JSR356/javax.websocket based (no javax in the names)
        2. connector will implicitly be Jetty javax.websocket based

...