Release Process

Prerequisites 

Before creating a release, ensure that:

• You can login to Nexus
• You have a Git workspace for https://git-wip-us.apache.org/repos/asf/incubator-gobblin.git

Apache releases are posted to dist.apache.org, which is a Subversion repository.

We have two directories there, check out these:

• https://dist.apache.org/repos/dist/release/incubator/gobblin/ - this is where PMC approved releases go. Do not upload here until we have a vote passed on dev@gobblin and dev@incubator. Check out this directory and name it apache-dist-release-gobblin
• https://dist.apache.org/repos/dist/dev/incubator/gobblin/ - this is where releases to be voted-on go. Make the release artifact, and commit it here, then post the [VOTE] thread with links here. Check out this directory and name it apache-dist-dev-gobblin

Example:

When working with these directories, make sure you are working with the correct one, otherwise you may be publishing pre-release artifacts to the global release mirror network!

GPG keys

The release manager must have a GPG key to be used to sign the release. You should pre-install gpg tool. The steps here also assume you have the following set (not using whoami if that’s not appropriate):

If you have an existing GPG key, but it does not include your Apache email address, you can add your email address as described in this Superuser.com posting. Otherwise, create a new GPG key giving your Apache email address, using gpg --gen-key then gpg --export-key $GPG_KEY > my-apache.key and gpg --export-secret-key -a $GPG_KEY > my-apache.private.key in the right directory (~/.ssh is a good one). For ease of this tutorial, let us assume you export your private key as: gpg --export-secret-key > ~/.gnupg/secring.gpg 

Upload your GPG public key (complete with your Apache email address on it) to a public keyserver - e.g. run gpg --export --armor $GPG_KEY and paste it into the “submit” box on http://pgp.mit.edu/

Look up your key fingerprint with gpg --fingerprint $GPG_KEY - it’s the long sequence of hex numbers separated by spaces. Log in to https://id.apache.org/ then copy-and-paste the fingerprint into “OpenPGP Public Key Primary Fingerprint”. Submit.

Now add your key to the apache-dist-release-gobblin/KEYS file:

cd apache-dist-release-gobblin

References:

• Post on the general@incubator list
• GPG cheatsheet

Once you have followed these instructions, you should have: 

• Your public key listed in the KEYS file in this repo
• Your public key viewable at https://people.apache.org/keys/committer/your-alias.asc
• Your public key also viewable at https://people.apache.org/keys/group/gobblin.asc

Gradle setup

Configure gradle: edit the file ~/.gradle/gradle.properties and make sure you have the following setup: 

# Maven Signing

signing.keyId=<GPG_KEYID>

signing.password=<GPG_PASSPHRASE>

signing.secretKeyRingFile=~/.gnupg/secring.gpg

 

# Nexus Access

nexusUsername=<NEXUS_USERNAME>

nexusPassword=<NEXUS_PASSWORD>

The GPG key ID can be found by running gpg --list-keys --keyid-format SHORT

Code Validation

Before releasing, you should run a full build to check if all the tests are passing. You can do that by running: 

./gradlew build

If this builds successfully then it means the tests pass and the code is well formatted. But, it doesn't mean that all license headers are in place.

Verify License

We must run checks to ensure that the files have proper license headers. These license checks can be run by executing:

./gradlew rat

You should open the report at build/rat/rat-report.html to validate that all files that are in scope (i.e. not excluded) have the appopriate headers. Running the rat task to generate this report is quick and does not runs other tests. See HEADER for the contents of the license header. These contents should appear at the top of the file as a comment. If a file or set of files needs to be excluded or included from Rat validation, you can update the Rat exclusion list in build.gradle.

Update CHANGELOG

Update the CHANGELOG.md file by adding details about the new version. This is a manual process, and you will have to go through all the JIRAs that are part of this release to summarize highlights, new features, enhancements and bugs. 

The existing CHANGELOG's format is self explanatory, so we will not go in depth about it. 

Update JIRAs

Update JIRAs to specify fixVersion (which is current release version). You can either do it manually for each JIRA, or use the custom tool that we have for this purpose: 

./dev/gobblin-jira-version 

This tool will ask you for the fixVersion that you want to apply on all the JIRAs that do not have a fixVersion and are in resolved or fixed state. Assuming you are making a release for x.y.z, your fixVersion input would be x.y.z

Note:

1. This operation can take time (in order of few minutes). The tool is simple enough, so you can also tweak it for your custom requirements. 
2. JIRAs that are in 'closed' state cannot be edited, so you might hit 'unauthorized exception'. If you encounter such JIRA, please reopen that JIRA and 'resolve' it instead. 

Create a branch for release

Assuming that you are preparing to release version x.y.z from the current commit, then create a branch with:

git checkout -b x.y.z
git push origin x.y.z

Reserve new version for next release

Update the version in gradle.properties in master branch. Eg: 

version=x.z.z

Also, if not done already create a new version in issues.apache.org/jira/browse/GOBBLIN using JIRA administration dashboard. You can specify the start date as of today and leave the end date as blank. 

Create a Release

The following steps will build a tarball suitable for an ASF source release. This also generates accompanying MD5 and ASC files.

First, clean any files unknown to git (WARNING: this removes all untracked files, including those listed in .gitignore, without prompting):

git clean -fdx

Alternatively, you can make a fresh clone of the repository to a separate directory:

git clone https://git-wip-us.apache.org/repos/asf/incubator-gobblin.git incubator-gobblin-release
cd incubator-gobblin-release

To generate the source release, run:

./gradlew clean release -Prelease=true

This generates a source tarball. The -Prelease=true setting prevents -SNAPSHOT from being appended to the version, which is the default behavior. It also prevents any builds from the extracted source tarball from including -SNAPSHOT in the version. It achieves this by modifying gradle.properties within the generated archive.

To generate the convenience binary, run:

./gradlew build -x findbugsMain -x test -x rat -x checkstyleMain

The source and binary tarball needs to be signed. You can do this either manually or automatically. If you have configured your key information in your gradle.properties then your archive should automatically be signed. There should now be a corresponding ASC file alongside the tarball and MD5 file. Otherwise you'll need to sign it manually by running:

gpg --sign --armor --detach-sig build/distribution/source/apache-gobblin-incubating-sources-*.tgz

gpg --sign --armor --detach-sig build/gobblin-distribution/distributions/apache-gobblin-incubating-bin-*.tar.gz

Upload the Release

You should make the release candidate available in https://dist.apache.org/repos/dist/dev/incubator/gobblin/. For example, if you are releasing release candidate RC0 for version x.y.z then you should upload the source distribution files to:

https://dist.apache.org/repos/dist/dev/incubator/gobblin/apache-gobblin-incubating-x.y.z-rc0/

To create a release folder and to check it out (be sure to substitute x.y.z for the actual version), run the following:

svn mkdir https://dist.apache.org/repos/dist/dev/incubator/gobblin/apache-gobblin-incubating-x.y.z-rc0
svn co https://dist.apache.org/repos/dist/dev/incubator/gobblin/apache-gobblin-incubating-x.y.z-rc0 apache-gobblin-incubating-x.y.z-rc0
cd apache-gobblin-incubating-x.y.z-rc0

You could then add the source and binary release as described above and commit.

Tag the release

You should tag the release candidate in git. Assuming you are releasing release candidate RC0 for version x.y.z then you can attach a tag to the current commit with:

git tag -s release-x.y.z-rc0 -m 'Apache Gobblin (incubating) x.y.z RC0'

Then push the tag:

git push origin release-x.y.z-rc0

Note: If on Mac, you might also need to do this: 

brew install pinentry-mac

export GPG_TTY=$(tty)

Staging artifacts in Nexus

General information on publishing to Maven can be found here. To build and upload the archives to the Apache Nexus staging repository, run:

./maven-nexus/maven-nexus.sh -remote 

Note that if you are running the above command from the source archive, then you don't need -Prelease=true. The above command assumes you have configured ~/.gradle/gradle.properties with your GPG key information.

If you now visit the Apache Nexus Repository and click on Staging Repositories, you should see a repository named orgapachegobblin-xxxx, where xxxx is some number. Select the repository and browse the content to make sure the set of files looks right. If it looks correct then Close the repository. The repository is now ready for use. If you look at the summary there is a URL for the repository that may be used to fetch the archives.

Let's suppose you have a Gradle project you'd like to use to test Gobblin. You can add the URL for the Staging Repository to your build.gradle like this:

repositories {
  mavenCentral()
  maven {
    url 'https://repository.apache.org/content/repositories/orgapachegobblin-xxxx'
  }
}

You could also visit the Staging Repository URL in your browser and download the artifacts.

Call for a vote to release

At this point you should have:

1. Published a source release
2. Staged artifacts in Nexus built from source

Now you can call a vote in the Gobblin dev mailing list for release. Look in the archives for previous votes for an example. If the vote passes then you may call a vote in the Incubator general mailing list (this is necessary because Gobblin is still incubating).

Here is a sample call for vote email: https://www.mail-archive.com/dev@gobblin.incubator.apache.org/msg01352.html 

Note: You should allow for 72 hours voting period for both - dev@gobblin mailing list and incubator general mailing list.