How To Release Tez

GPG Keys Setup

Follow Apache guidelines on setting up your GPG keys and ensure that your fingerprint is updated at id.apache.org.

gpg --armor --fingerprint --list-sigs <keyid>
gpg --armor --export <keyid>

Ensure that you publish your key at http://pgp.mit.edu/ and sks-keyservers.net and keyserver.ubuntu.com. While creating a staging artifact, "Signature Validation" process seems to check your key at sks-keyserver.net and keyserver.ubuntu.com.

The snippet below will take care of the important keyservers for you.

gpg --keyserver pgp.mit.edu --send-key <keyid>
gpg --keyserver keyserver.ubuntu.com --send-key <keyid>
gpg --keyserver pool.sks-keyservers.net --send-key <keyid>


Add PGP Key from above to Tez source tree and distribution repository. KEYS file in Tez distribution repository will be used for release verification in release candidate voting 

Distribution Keys
# Append PGP Key to KEYS file in https://gitbox.apache.org/repos/asf/tez.git
$ svn co https://dist.apache.org/repos/dist/release/tez/ tez-dist/
# copy KEYS file from gitbox tez/KEYS to tez-dist/KEYS and verify difference
$ svn diff
# Commit the changes (no jira required) with a useful message substituting apache user name as appropriate
$ svn ci -m "Add <apache user name> public key to KEYS file"

Create ENV variables for later usage

It's crucial to define the Tez version and release candidate (RC) number for the release process. It's also recommended to add these to your bashrc, since every subsequent step in the release process assumes these variables are set.

export TEZ_RELEASE_VERSION=0.10.5 # define version
export TEZ_RC_VERSION=rc0 # define rc version

Creating a Release Candidate

Create or Checkout the Release Branch
git checkout -b branch-$TEZ_RELEASE_VERSION 
Run Basic Release checks
mvn clean install -DskipTests
mvn clean apache-rat:check
Push branch to remote
git push -u origin branch-$TEZ_RELEASE_VERSION
Generate CHANGES.txt

Currently we don't keep CHANGES.txt in repo and only generate it on demand. To find previous example, search in previous release branch.

To generate CHANGES.txt, please pull information from JIRA and git log. To get all JIRAs of this release, use filter 'fixVersion = <version> AND resolution = resolved'. To get all commits of this release, you can find the last commit in last release in corresponding release branch and do a git diff against current release branch. Open a JIRA to commit the newly generated CHANGES.txt to branch-$TEZ_RELEASE_VERSION. A review of the patch is preferable.

# origin points to the tez repo: https://github.com/apache/tez
# a.b.c: last tez release version
# x.z.y: current tez release version
git log rel/release-a.b.c..origin/branch-$TEZ_RELEASE_VERSION --format="%s" # command to get the commit logs that go in CHANGES.txt

There are cases where information from JIRAs is inconsistent with that from git log (which shouldn't happen but unfortunately happens sometime...). For example, committer may forget to mark JIRA as resolved or set the fix version, or commit message contains wrong JIRA number. So it's better to check both side.

Several things need notice:

  • Incompatible changes should be found by using filter "Hadoop Flags" = "Incompatible change". Don't search by labels 'incompatible' or something similar.
  • Umbrella JIRA should be listed in individually (instead of getting mixed in non-umbrella jira)
  • Changes are grouped by release version. If one patch gets committed into multiple branches, put this change in the oldest one. For example, if a patch is committed in branch 0.9.0 (current release), 0.8.6 (unreleased of 0.8 line), this change should be listed in the section of 0.8.6.
Update Version Number if Required
mvn versions:set -DnewVersion="$TEZ_RELEASE_VERSION"
Modify CHANGES.txt to set the release date to when the vote will likely end ( +3 or 4 days from the vote start ).
If needed, add missing entries in CHANGES.txt for this release's ChangeLog using the output from "git log --pretty=oneline"
Commit and push all the changes.
Create a Release Tag
git tag -a release-$TEZ_RELEASE_VERSION-$TEZ_RC_VERSION -m "Tez $TEZ_RELEASE_VERSION $TEZ_RC_VERSION"
# push the new tag 
git push origin release-$TEZ_RELEASE_VERSION-$TEZ_RC_VERSION 
Deploy the jars to Staging

Your .m2/settings.xml should have something along these lines:

The full apache guide for publishing artifacts can be found at http://www.apache.org/dev/publishing-maven-artifacts.html

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
    <server>
     <id>apache.staging.https</id>
     <username>XXXXXX</username>
     <password>YYYYYY</password>
    </server>
  </servers>
  <profiles>
    <profile>
      <id>gpg</id>
      <properties>
        <gpg.passphrase>XXXXXXX</gpg.passphrase>
        <!-- gpg.executable should be gpg if a lower version is used for creating the key -->
        <gpg.executable>gpg2</gpg.executable>
      </properties>
    </profile>
  </profiles>
</settings>

To deploy jars, run:

mvn clean deploy -Psources,javadoc,sign -DskipTests
# if you get this cryptic error: gpg: signing failed: Inappropriate ioctl for device, then just do this below first:
export GPG_TTY=$(tty)

The above command allows one to see the deployed jars in the Staging area. Log on to https://repository.apache.org and look at the Staging Repositories. "Close" the tez jars recently uploaded to get the temporary staging repo URL.

Create a Release tarball for the Source
mvn clean
git clean -f -x -d

cd .. # assuming that you're standing in a tez git folder, 
cp -R tez-git-$TEZ_RELEASE_VERSION apache-tez-$TEZ_RELEASE_VERSION-src #where tez-git-$TEZ_RELEASE_VERSION is the tez code directory
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf apache-tez-$TEZ_RELEASE_VERSION-src.tar.gz apache-tez-$TEZ_RELEASE_VERSION-src
Sign the Release
gpg2  --armor --output apache-tez-$TEZ_RELEASE_VERSION-src.tar.gz.asc --detach-sig apache-tez-$TEZ_RELEASE_VERSION-src.tar.gz
Generate checksums
shasum -a 512 apache-tez-$TEZ_RELEASE_VERSION-src.tar.gz > apache-tez-$TEZ_RELEASE_VERSION-src.tar.gz.sha512
Create Binary Tarball as a helper for Users

Create a final binary tarball using the minimal and full tarballs: 

  • Create a top-level directory called apache-tez-$TEZ_RELEASE_VERSION-bin
  • Untar the contents of the minimal tarball into this directory.
  • Copy the full tarball into the directory under apache-tez-$TEZ_RELEASE_VERSION-bin/share/ and name the file tez.tar.gz 
  • Copy the minimal tarball into the directory under apache-tez-$TEZ_RELEASE_VERSION-bin/share/ and name the file tez-minimal.tar.gz 
  • Copy the LICENSE* files and NOTICE file from the full tarball and replace the files under apache-tez-$TEZ_RELEASE_VERSION-bin/ which came from the minimal tarball. 
  • If the version is higher than 0.8.x, use the mvn javadoc command to generate the xml-based config documentation and copy the files under apache-tez-$TEZ_RELEASE_VERSION-bin/conf/ with a .template suffix.
  • Tar the top-level directory and do the necessary signing as well as the checksum creation as done for the src tarball. 
# Define the following environment variable
export TEZ_SRC_DIR=/home/user/tez #e.g. tez-src or the absolute path
 
cd ${TEZ_SRC_DIR};
mvn clean install -DskipTests
mvn site
cd ..


mkdir apache-tez-${TEZ_RELEASE_VERSION}-bin
tar -C apache-tez-${TEZ_RELEASE_VERSION}-bin -zxvf ${TEZ_SRC_DIR}/tez-dist/target/tez-${TEZ_RELEASE_VERSION}-minimal.tar.gz
mkdir tmp-tez-full
tar -C tmp-tez-full -zxvf ${TEZ_SRC_DIR}/tez-dist/target/tez-${TEZ_RELEASE_VERSION}.tar.gz
cp tmp-tez-full/LICENSE* apache-tez-${TEZ_RELEASE_VERSION}-bin/
cp tmp-tez-full/NOTICE* apache-tez-${TEZ_RELEASE_VERSION}-bin/
rm -rf tmp-tez-full


mkdir apache-tez-${TEZ_RELEASE_VERSION}-bin/conf
# copy over tez templates into conf dir.
cp ${TEZ_SRC_DIR}/tez-api/target/site/apidocs/configs/tez-default-template.xml apache-tez-${TEZ_RELEASE_VERSION}-bin/conf/
cp ${TEZ_SRC_DIR}/tez-runtime-library/target/site/apidocs/configs/tez-runtime-default-template.xml apache-tez-${TEZ_RELEASE_VERSION}-bin/conf/


mkdir apache-tez-${TEZ_RELEASE_VERSION}-bin/share
cp ${TEZ_SRC_DIR}/tez-dist/target/tez-${TEZ_RELEASE_VERSION}.tar.gz apache-tez-${TEZ_RELEASE_VERSION}-bin/share/tez.tar.gz
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf apache-tez-${TEZ_RELEASE_VERSION}-bin.tar.gz apache-tez-${TEZ_RELEASE_VERSION}-bin
# Sign release and create checksums using commands similar to those called out earlier for the source tarball
gpg2  --armor --output apache-tez-${TEZ_RELEASE_VERSION}-bin.tar.gz.asc --detach-sig apache-tez-${TEZ_RELEASE_VERSION}-bin.tar.gz
shasum -a 512 apache-tez-$TEZ_RELEASE_VERSION-bin.tar.gz > apache-tez-$TEZ_RELEASE_VERSION-bin.tar.gz.sha512


#Verify integrity of checksum files
 


Upload Artifacts for Release Vote

The artifacts ( source tarball, binary tarball, checksums etc) need to be copied over to https://dist.apache.org/repos/dist/dev/tez/. ( This is an SVN repo where the release artifacts need to be committed to ). The artifacts should be created under a directory with the release number and RC number.

svn co https://dist.apache.org/repos/dist/dev/tez/ tez_svn # assuming that you want to store svn files in a folder named "tez_svn" for clarity's sake

# copy contents to "tez_svn" directory, which is the svn working dir
mkdir tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}
cp apache-tez-${TEZ_RELEASE_VERSION}-bin.tar.gz tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}/
cp apache-tez-${TEZ_RELEASE_VERSION}-bin.tar.gz.asc tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}/
cp apache-tez-${TEZ_RELEASE_VERSION}-bin.tar.gz.sha512 tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}/
cp apache-tez-${TEZ_RELEASE_VERSION}-src.tar.gz tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}/
cp apache-tez-${TEZ_RELEASE_VERSION}-src.tar.gz.asc tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}/
cp apache-tez-${TEZ_RELEASE_VERSION}-src.tar.gz.sha512 tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}/

# check contents of "tez_svn" for actual release, you should see something like:
ls tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}/


apache-tez-x.y.z-bin.tar.gz         apache-tez-x.y.z-src.tar.gz       
apache-tez-x.y.z-bin.tar.gz.asc     apache-tez-x.y.z-bin.tar.gz.sha512  apache-tez-x.y.z-src.tar.gz.asc     apache-tez-x.y.z-src.tar.gz.sha512

svn add tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}/
# if this is the first time you commit to svn, you might want to define an editor for commit messages, like: export SVN_EDITOR=vi
svn ci tez_svn/apache-tez-${TEZ_RELEASE_VERSION}-${TEZ_RC_VERSION}/ #specify your ASF username with --username=$USER_NAME if it's not the same as your current user

Release Voting process

Start a Vote

Call for a vote on the dev mailing list with something like this:

Email subject should be: [VOTE] Release Apache Tez-x.y.z RC0

I have created an tez-x.y.z release candidate rc0.
GIT source tag (r***)

https://gitbox.apache.org/repos/asf?p=tez.git;a=commit;h=refs/tags/release-x.y.z-rc0 ( git hash <commit hash> )

Staging site: https://dist.apache.org/repos/dist/dev/tez/tez-x.y.z-rc0/  ( svn revision <revision> )
Nexus Staging URL:

PGP release keys (signed using <GPG KEY>) http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=<GPG KEY>
KEYS file available at https://dist.apache.org/repos/dist/release/tez/KEYS

One can look into the issues fixed in this release at https://issues.apache.org/jira/**

Vote will be open for atleast 72 hours.

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Ending a Successful/Failed Vote

Once the vote passes/fails, send out an email with subject like "[RESULT][VOTE] Apache Tez x.y.z rc0" to dev@tez.apache.org.

List +1s, -1s, 0s with clear distinctions between binding ( PMC and committers ) and nonbinding ( general contributors) votes.

Canceling a Vote

For a cancelled vote due to any reason, change the subject to "[CANCEL][VOTE] Apache Tez x.y.z rc0"

After A Successful Vote

Release the staging repository

Log on to https://repository.apache.org and look at the Staging Repositories. "Release" the tez repository previously sent out in the vote thread, so that it is available for general consumption.

Copy the bits to distribution folder for Apache Tez

Copy the Release Candidate bits from https://dist.apache.org/repos/dist/dev/tez/ to https://dist.apache.org/repos/dist/release/tez/. Create a new directory with the release number and copy the artifacts into that directory (use an svn mv to retain history). If needed, update STABLE to point to the new release. This step need to be done by PMC, or PMC has to grant the permission of dist/release to release manager.

As an example

svn mv https://dist.apache.org/repos/dist/dev/tez/apache-tez-$TEZ_RELEASE_VERSION-$TEZ_RC_VERSION https://dist.apache.org/repos/dist/release/tez/$TEZ_RELEASE_VERSION 
Create the release tag
git tag -a rel/release-$TEZ_RELEASE_VERSION -m "Tez $TEZ_RELEASE_VERSION"
git push origin rel/release-$TEZ_RELEASE_VERSION

Note that it takes 24 hours for the changes to propagate to the mirrors.
Wait 24 hours and verify that the bits are available in the mirrors before sending an announcement. Sometime it takes longer than 24 hours.

Update DOAP file for Tez

Update Tez_DOAP.rdf at the top of the source tree in the master branch to update the releases section for this new release.

Update the SNAPSHOT version on the master

Update the SNAPSHOT version on the master branch to point to the next release.
E.g. if you just released 0.10.1, and Tez goes on the 0.10.x line, the next SNAPSHOT will become 0.10.2-SNAPSHOT.

export TEZ_NEXT_VERSION=4.5.6 # example, fix it
mvn versions:set -DnewVersion="$TEZ_NEXT_VERSION-SNAPSHOT" 

With the DOAP and SNAPSHOT updates, you can create a PR against the master branch, e.g.: https://github.com/apache/tez/pull/335


Update website at tez.apache.org

Items that should be updated on the website. A new link under releases for the new release. The link should go to a landing page that provides release-notes, release artifacts link (See link in the announcement email below) and links to javadocs for the various projects. Ensure that the Releases section as well as the "All Releases" page on tez.apache.org is updated with the contents for this release.

The Javadocs should be generated from the correct release branch to ensure they are properly tagged.
To generate the Javadocs, run the following command from the project’s root directory:

mvn clean install javadoc:aggregate -DskipTests -pl tez-api -pl tez-mapreduce -pl tez-runtime-library
mvn site -f tez-api
mvn site -f tez-mapreduce
mvn site -f tez-runtime-library

The javadocs would be created under target/site/apidocs inside each project directory. Spot-check the javadocs for obvious errors, open files:

tez-api/target/site/apidocs/index.html
tez-mapreduce/target/site/apidocs/index.html
tez-runtime-library/target/site/apidocs/index.html


Clone the Tez site repository from SVN (or, since it’s quite large, reuse an existing clone if you already have one). Here, svn status should return empty results, if your local working copy is clean, which is the recommended state before starting the release update process.

svn co https://svn.apache.org/repos/asf/tez/site
cd site
export TEZ_SITE_SVN_DIR=...  #the folder where you checked out https://svn.apache.org/repos/asf/tez/site

Before starting the release update process, it's recommended to ensure your local working copy is clean. You can verify this by running:

svn status


Create a new folder $TEZ_RELEASE_VERSION under the releases directory in the svn site repo:

mkdir -p ${TEZ_SITE_SVN_DIR}/releases/${TEZ_RELEASE_VERSION}


cp -r tez-api/target/site/apidocs ${TEZ_SITE_SVN_DIR}/releases/${TEZ_RELEASE_VERSION}/tez-api-javadocs
cp -r tez-mapreduce/target/site/apidocs ${TEZ_SITE_SVN_DIR}/releases/${TEZ_RELEASE_VERSION}/tez-mapreduce-javadocs
cp -r tez-runtime-library/target/site/apidocs ${TEZ_SITE_SVN_DIR}/releases/${TEZ_RELEASE_VERSION}/tez-runtime-library-javadocs


Create a release notes file at the path: ${TEZ_SITE_SVN_DIR}/releases/${TEZ_RELEASE_VERSION}/release-notes.txt according to earlier releases' files.

cp ${TEZ_SITE_SVN_DIR}/releases/0.10.4/release-notes.txt ${TEZ_SITE_SVN_DIR}/releases/${TEZ_RELEASE_VERSION}/release-notes.txt


Edit the new release-notes.txt file: ${TEZ_SITE_SVN_DIR}/releases/${TEZ_RELEASE_VERSION}/release-notes.txt


The website is built from the git repo but the javadocs etc are committed only to svn to reduce the size of the git repo.


Update the markdown files for the site—feel free to reuse content from earlier releases as a starting point.

Note: Any changes made here should also be committed back (with an appropriate commit message linking back to an Apache Jira) to the master branch to keep the markdown files up to date.

cp docs/src/site/markdown/releases/apache-tez-0-10-0.md docs/src/site/markdown/releases/apache-tez-${TEZ_RELEASE_VERSION//./-}.md
sed -i '' 's/0\.10\.0/0.10.5/g' docs/src/site/markdown/releases/apache-tez-${TEZ_RELEASE_VERSION//./-}.md


Update docs/src/site/markdown/releases/index.md accordingly, add a new line for the release, like:

-   [Apache TEZ&reg; 0.10.5](./apache-tez-0-10-5.html) (May 30, 2025)


Update docs/src/site/site.xml, add a menu item for the new release, for example:

 <item name="0.10.5" href="./releases/apache-tez-0-10-5.html"/>


Update the site pages using the markup you just modified by running the following from the root of the source code repository:

mvn clean site -pl docs


Verify the site updates are correct by browsing the website locally from the docs/target directory: open docs/target/index.html.


From the docs/target directory, copy the newly generated website into the previously checked out SVN location.

cp -r docs/target/* $TEZ_SITE_SVN_DIR/
rm -rf $TEZ_SITE_SVN_DIR/maven-shared-archive-resources


Check the full web-page locally in the svn folder, which now contains the release-notes.txt as well as the api docs.


Do an svn commit for the tez website with an appropriate commit message linking back to an Apache Jira.

export SVN_EDITOR=vi
svn add --force * --auto-props --parents --depth infinity
svn ci


Announcement

Send out Announcement to dev@tez.apache.org and user@tez.apache.org
Subject: [ANNOUNCE] Apache Tez x.y.z.

The Apache Tez team is proud to announce the release of Apache Tez version x.y.z 

The Apache Tez project aims to create a framework to build efficient and scalable data processing applications that can be modeled as data flow graphs.

<Describe the highlights of the release>

The release bits are at: http://www.apache.org/dyn/closer.lua/tez/x.y.z 

We want to thank all the contributors who made the release possible.

Regards,

The Tez Team

JIRA Updates

The following has to be done by PMC since only PMC has the required permission.

Find all JIRAs that were fixed in this release ( i.e. Fix Version set to the released version ) and do a bulk edit to mark them all as Closed. A useful point to note is that when doing this bulk update, mail notifications should be turned off to avoid triggering a barrage of emails being sent out. Find all JIRAs with fix version x.y.z and not-resolved/not-closed and remove the fix version.

Once this is complete, using the JIRA Administration tab, mark the x.y.z as released and set the appropriate release date.

https://issues.apache.org/jira/plugins/servlet/project-config/TEZ/versions

Create the next version if it is not already done. Find all JIRAs with target version x.y.z and not-resolved/not-closed and change the target version to the next release following the above bulk update process.






  • No labels