...
If you don't run this locally, they will be ran during presubmit, by Jenkins. However, if these fail during presubmit, you may not see the output of test failures. So doing this first is recommended to make your development process a bit smoother and iterate on your PR until it passes the presubmit.
How do I perform a dependency upgrade?
To perform a dependency upgrade we want to ensure that the PR is not introducing any new linkage errors. We do this by combining successful Jenkins test runs with analysis performed using a linkage checker. This allows us to gain confidence that we are minimizing the number of linkage issues that will arise for users.
The overall process is:
- Find all gradle subprojects that are impacted by the dependency change
- For each gradle subproject, perform the before and after linkage checker analysis and provide the results as part of your PR
- For each gradle subproject, find and run relevant Jenkins test suites
Find all gradle subprojects that are impacted by the dependency change
The command below will print out a dependency report in a text file for each project:
| Code Block |
|---|
./gradlew dependencyReport |
You may then grep for a specific maven artifact identifier such as guava in all the dependency reports with:
| Code Block |
|---|
grep -l "guava" `find ./ -name dependencies.txt` |
Linkage checker analysis
You can use the shell script to do this on your behalf (note that it will run the manual command below on your current workspace and also on HEAD):
| Code Block |
|---|
/bin/bash sdks/java/build-tools/beam-linkage-check.sh "artifactId1,artifactId2,..." |
You should copy and paste the output to the PR. If it is large, you may want to use a github gist. Some example PRs (1, 2, 3, 4, 5).
Note that you can manually run the linkage checker on your current workspace by invoking:
| Code Block |
|---|
./gradlew -Ppublishing -PjavaLinkageArtifactIds=artifactId1,artifactId2,... :checkJavaLinkage |
Example output is:
| Code Block |
|---|
Class org.brotli.dec.BrotliInputStream is not found;
referenced by 1 class file
org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.brotli.BrotliCompressorInputStream (beam-sdks-java-core-2.20.0-SNAPSHOT.jar)
Class com.github.luben.zstd.ZstdInputStream is not found;
referenced by 1 class file
org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.zstandard.ZstdCompressorInputStream (beam-sdks-java-core-2.20.0-SNAPSHOT.jar)
Class com.github.luben.zstd.ZstdOutputStream is not found;
referenced by 1 class file
org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream (beam-sdks-java-core-2.20.0-SNAPSHOT.jar)
Class org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.commons.ModuleHashesAttribute is not found;
referenced by 1 class file
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.commons.ClassRemapper (beam-vendor-bytebuddy-1_9_3-0.1.jar) |
Run relevant Jenkins test suites
You can find all jenkins job configurations within https://github.com/apache/beam/tree/master/.test-infra/jenkins and request that the reviewer run the relevant test suites by providing them with a list of all the relevant trigger phrases. You can perform this request directly on your PR or on the dev mailing list (example).