How to perform a dependency upgrade?
To perform a dependency upgrade we want to ensure that the PR is not introducing any new linkage errors. We do this by combining successful Jenkins test runs with analysis performed using a linkage checker. This allows us to gain confidence that we are minimizing the number of linkage issues that will arise for users. To perform a dependency upgrade:
- Find all Gradle subprojects that are impacted by the dependency change.
- For each Gradle subproject:
- Perform the
afterlinkage checker analysis.
- Provide the results as part of your PR.
- Perform the
- For each Gradle subproject:
- Find and run relevant Jenkins test suites.
How to find all Gradle subprojects that are impacted by the dependency change
Execute the command below will print out a dependency report in a text file for each project:
Grep for a specific maven artifact identifier such as guava in all the dependency reports with:
grep -l "guava" `find ./ -name dependencies.txt`
Linkage checker analysis
This step relies on modifying your local maven repository, typically found in
Use the shell script to do this on your behalf (note that it will run the manual command below on your current workspace and also on HEAD):
/bin/bash sdks/java/build-tools/beam-linkage-check.sh origin/master <your branch name> "artifactId1,artifactId2,..."
If you omit the artifactIds, it uses
beam-sdks-java-core beam-sdks-java-io-google-cloud-platform beam-runners-google-cloud-dataflow-java beam-sdks-java-io-hadoop-format; these artifacts often suffer dependency conflicts.
Copy and paste the output to the PR. If it is large, you may want to use a GitHub gist. For example PRs (1, 2, 3, 4, and 5).
Note that you can manually run the linkage checker on your current workspace by invoking:
./gradlew -Ppublishing -PjavaLinkageArtifactIds=artifactId1,artifactId2,... :checkJavaLinkage
Check the example output is:
Class org.brotli.dec.BrotliInputStream is not found; referenced by 1 class file org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.brotli.BrotliCompressorInputStream (beam-sdks-java-core-2.20.0-SNAPSHOT.jar) Class com.github.luben.zstd.ZstdInputStream is not found; referenced by 1 class file org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.zstandard.ZstdCompressorInputStream (beam-sdks-java-core-2.20.0-SNAPSHOT.jar) Class com.github.luben.zstd.ZstdOutputStream is not found; referenced by 1 class file org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream (beam-sdks-java-core-2.20.0-SNAPSHOT.jar) Class org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.commons.ModuleHashesAttribute is not found; referenced by 1 class file org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.commons.ClassRemapper (beam-vendor-bytebuddy-1_9_3-0.1.jar)
Delete any installed Apache Beam SNAPSHOT artifacts:
rm -rf ~/.m2/repository/org/apache/beam
Run relevant Jenkins test suites
You can find all Jenkins job configurations within https://github.com/apache/beam/tree/master/.test-infra/jenkins and request that the reviewer run the relevant test suites by providing them with a list of all the relevant trigger phrases. You can perform this request directly on your PR or on the dev mailing list, for example.
Google Cloud-related dependency upgrades
To provide the consistent dependencies to Beam users, follow the following steps when upgrading Google Cloud-related dependencies:
- Set the Libraries BOM version.
- Find the latest release in https://github.com/googleapis/java-cloud-bom/releases and set libraries-bom value in BeamModulePlugin.groovy
- Find core Google Java library versions.
- Such as gRPC, Protobuf, Guava, Google Auth Library in the release note of the Libraries BOM and set them in BeamModulePlugin.groovy
- Find appropriate Netty version by checking
io.grpc:grpc-netty's dependency declaration. For example, you can tell gRPC version 1.49.0 was built with Netty "4.1.77.Final" by reading https://search.maven.org/artifact/io.grpc/grpc-netty/1.49.0/jar:
Update netty_version in BeamModulePlugin.groovy
- Find netty-tcnative version via
netty-parentartifact. For example, you can tell Netty 4.1.77.Final was built with netty-tcnative "2.0.52.Final". https://search.maven.org/artifact/io.netty/netty-parent/4.1.77.Final/jar:
Update netty_tcnative_boringssl_static version in BeamModulePlugin.groovy