...
Set up a few environment variables to simplify Maven commands that follow. This identifies the release candidate being built. Start with RC_NUM equal to 1 and increment it for each candidate.
git checkout ${RELEASE_BRANCH}- Run mvn version to set the proper rc number in all artifacts
mvn versions:set -DnewVersion=${RELEASE_VERSION}-rc${RC_NUM}
- Run Unit tests and ensure they succeed
mvn test -DskipITs=true
- Run Integration Tests and ensure they succeed
mvn verify -DskipUTs=true
- Commit and push this change to RELEASE branch
git commit -am "Bumping release candidate number ${RC_NUM}"There will be some backup files created which needs to be removed. You could do "git clean -fdx" before doing the commit.
git push origin ${RELEASE_BRANCH}
If you already have a remote tag with same name as your branch, you can try below command.
git push origin refs/heads/${RELEASE_BRANCH}
"refs/heads/" refers to a branch.
"refs/tags/" refers to tag.
- Generate Source Release: This will create the tarball under hudi/src_release directory
git checkout ${RELEASE_BRANCH}cd scripts && ./release/create_source_release.sh
If you have multiple gpg keys(gpg --list-keys), then the signing command will take in the first key most likely. You will release this when it asks for a passphrase in a pop up. When asked for passphrase, ensure the intended key is the one asked for.
Command used in script:
gpg --armor --detach-sig ${RELEASE_DIR}/hudi-${RELEASE_VERSION}.src.tgz
To use a specific key: update as follows: // replace FINGERPRINT
gpg --local-user [FINGERPRINT] --armor --detach-sig ${RELEASE_DIR}/hudi-${RELEASE_VERSION}.src.tgz
- Verify Source release is signed and buildable
- cd hudi/src_release
- gpg --verify hudi-${RELEASE_VERSION}-rc${RC_NUM}.src.tgz.asc hudi-${RELEASE_VERSION}-rc${RC_NUM}.src.tgz
- tar -zxvf hudi-${RELEASE_VERSION}-rc${RC_NUM}.src.tgz && cd hudi-${RELEASE_VERSION}-rc${RC_NUM} && mvn clean package -DskipTests
- If they pass, delete the repository we got from the tar-ball
- cd ../ && rm -rf hudi-${RELEASE_VERSION}-rc${RC_NUM}
- Create tag
git tag -s release-${RELEASE_VERSION}-rc${RC_NUM} -m "${RELEASE_VERSION}"
If you run into some issues, and if want to re-run the same candidate again from start, ensure you delete existing tags before retrying again.
// to remove local
git tag -d release-${RELEASE_VERSION}-rc${RC_NUM}
// to remove remote
git push --delete origin release-${RELEASE_VERSION}-rc${RC_NUM}- if apache repo is origin.
git push origin release-${RELEASE_VERSION}-rc${RC_NUM}If a branch with the same name already exists in origin, this command might fail as below.
error: src refspec release-0.5.3 matches more than one
error: failed to push some refs to 'https://github.com/apache/hudi.git'
In such a case, try below command
git push origin refs/tags/release-${RELEASE_VERSION}-rc${RC_NUM}
- Deploy maven artifacts and verify
- This will deploy jar artifacts to the Apache Nexus Repository, which is the staging area for deploying jars to Maven Central.
- Review all staged artifacts (https://repository.apache.org/). They should contain all relevant parts for each module, including pom.xml, jar, test jar, source, test source, javadoc, etc. Carefully review any new artifacts.
git checkout ${RELEASE_BRANCH}- This will deploy scala 2.11 and spark 2 artifacts to repository.apache.org
cd scripts && ./release/deploy_staging_jars.sh --scala_version=2.11
- This will deploy scala 2.12 and spark 2 artifacts to repository.apache.org.
./release/deploy_staging_jars.sh --scala_version=2.12
- This will deploy scala 2.12 and spark 3 artifacts to repository.apache.org.
./release/deploy_staging_jars.sh --scala_version=2.12 --spark_version=3
- Review all staged artifacts by logging into Apache Nexus and clicking on "Staging Repositories" link on left pane. Then find a "open" entry for apachehudi
- Ensure it contains all 3 (2.11, 2.12 with spark2 and 2.12 with spark3) artifacts, mainly hudi-spark-bundle-2.11/2.12, hudi-spark3-bundle-2.12, hudi-spark-2.11/2.12, hudi-spark2-2.11/2.12, hudi-spark3-2.12, hudi-utilities-bundle_2.11/2.12 and hudi-utilities_2.11/2.12.
- With 0.10.1, we had 4 bundles. spark2 with scala11, spark2 with scala12, spark3.0.x bundles and spark3.1.x bundles. Ensure each spark bundle reflects the version correctly. hudi-spark3.1.2-bundle_2.12-0.10.1.jar and hudi-spark3.0.3-bundle_2.12-0.10.1.jar are the respective bundle names for spark3 bundles.
- Once you have ensured everything is good and validation of step 7 succeeds, you can close the staging repo. Until you close, you can re-run deploying to staging multiple times. But once closed, it will create a new staging repo. So ensure you close this, so that the next RC (if need be) is on a new repo. So, once everything is good, close the staging repository on Apache Nexus. When prompted for a description, enter “Apache Hudi, version ${RELEASE_VERSION}, release candidate ${RC_NUM}”.
- Stage source releases on dist.apache.org
- If you have not already, check out the Hudi section of the dev repository on dist.apache.org via Subversion. In a fresh directory
- if you would not checkout, please try svn checkout https://dist.apache.org/repos/dist/dev/hudi again.
svn checkout https://dist.apache.org/repos/dist/dev/hudi --depth=immediates
- Make a directory for the new release:
mkdir hudi/hudi-${RELEASE_VERSION}-rc${RC_NUM}
- Copy Hudi source distributions, hashes, and GPG signature:
mv <hudi-dir>/src_release/* hudi/hudi-${RELEASE_VERSION}-rc${RC_NUM}
- Add and commit all the files.
cd hudisvn add hudi-${RELEASE_VERSION}-rc${RC_NUM}svn commit
- Verify that files are present
- Run Verification Script to ensure the source release is sane
cd scripts && ./release/validate_staged_release.sh --release=${RELEASE_VERSION} --rc_num=${RC_NUM} --verbose
Checklist to proceed to the next step
...