Prerequisites
Before creating a release, ensure that:
- You can login to Nexus
- You have a Git workspace for https://git-wip-us.apache.org/repos/asf/incubator-gobblin.git
Apache releases are posted to dist.apache.org, which is a Subversion repository.
We have two directories there, check out these:
- https://dist.apache.org/repos/dist/release/incubator/gobblin/ - this is where PMC approved releases go. Do not upload here until we have a vote passed on dev@gobblin and dev@incubator. Check out this directory and name it apache-dist-release-gobblin
- https://dist.apache.org/repos/dist/dev/incubator/gobblin/ - this is where releases to be voted-on go. Make the release artifact, and commit it here, then post the [VOTE] thread with links here. Check out this directory and name it
apache-dist-dev-gobblin
Example:
svn co https://dist.apache.org/repos/dist/release/incubator/gobblin apache-dist-release-gobblin
svn co https://dist.apache.org/repos/dist/dev/incubator/gobblin apache-dist-dev-gobblin
When working with these directories, make sure you are working with the correct one, otherwise you may be publishing pre-release artifacts to the global release mirror network!
GPG keys
The release manager must have a GPG key to be used to sign the release. You should pre-install gpg
tool. The steps here also assume you have the following set (not using whoami if that’s not appropriate):
ASF_USERNAME=`whoami`
GPG_KEY=$ASF_USERNAME@apache.org
SVN_USERNAME=$ASF_USERNAME
If you have an existing GPG key, but it does not include your Apache email address, you can add your email address as described in this Superuser.com posting. Otherwise, create a new GPG key giving your Apache email address, using gpg --gen-key
then gpg --export-key $GPG_KEY > my-apache.key
and gpg --export-secret-key -a $GPG_KEY > my-apache.private.key
in the right directory (~/.ssh
is a good one). For ease of this tutorial, let us assume you export your private key as: gpg --export-secret-key > ~/.gnupg/secring.gpg
Upload your GPG public key (complete with your Apache email address on it) to a public keyserver - e.g. run gpg --export --armor $GPG_KEY
and paste it into the “submit” box on http://pgp.mit.edu/
Look up your key fingerprint with gpg --fingerprint $GPG_KEY
- it’s the long sequence of hex numbers separated by spaces. Log in to https://id.apache.org/ then copy-and-paste the fingerprint into “OpenPGP Public Key Primary Fingerprint”. Submit.
Now add your key to the apache-dist-release-gobblin/KEYS
file:
cd apache-dist-release-gobblin
(gpg --list-sigs $ASF_USERNAME@apache.org && gpg --armor --export $ASF_USERNAME@apache.org) >> KEYS
svn --username $SVN_USERNAME --no-auth-cache commit -m "Update gobblin/KEYS for $GPG_KEY"
References:
Once you have followed these instructions, you should have:
- Your public key listed in the
KEYS
file in this repo - Your public key viewable at https://people.apache.org/keys/committer/your-alias.asc
- Your public key also viewable at https://people.apache.org/keys/group/gobblin.asc
Gradle setup
Configure gradle: edit the file ~/.gradle/gradle.properties and make sure you have the following setup:
# Maven Signing
signing.keyId=<GPG_KEYID>
signing.password=<GPG_PASSPHRASE>
signing.secretKeyRingFile=~/.gnupg/secring.gpg
# Nexus Access
nexusUsername=<NEXUS_USERNAME>
nexusPassword=<NEXUS_PASSWORD>
The GPG key ID can be found by running gpg --list-keys --keyid-format SHORT
Code Validation
Before releasing, you should run a full build to check if all the tests are passing. You can do that by running:
./gradlew build
If this builds successfully then it means the tests pass and the code is well formatted. But, it doesn't mean that all license headers are in place.
Verify License
We must run checks to ensure that the files have proper license headers. These license checks can be run by executing:
./gradlew rat
You should open the report at build/rat/rat-report.html
to validate that all files that are in scope (i.e. not excluded) have the appopriate headers. Running the rat
task to generate this report is quick and does not runs other tests. See HEADER
for the contents of the license header. These contents should appear at the top of the file as a comment. If a file or set of files needs to be excluded or included from Rat validation, you can update the Rat exclusion list in build.gradle
.
Update CHANGELOG
Update the CHANGELOG.md file by adding details about the new version. This is a manual process, and you will have to go through all the JIRAs that are part of this release to summarize highlights, new features, enhancements and bugs.
The existing CHANGELOG's format is self explanatory, so we will not go in depth about it.
Update JIRAs
Update JIRAs to specify fixVersion (which is current release version). You can either do it manually for each JIRA, or use the custom tool that we have for this purpose:
./dev/gobblin-jira-version
This tool will ask you for the fixVersion that you want to apply on all the JIRAs that do not have a fixVersion and are in resolved or fixed state. Assuming you are making a release for x.y.z, your fixVersion input would be x.y.z
Note:
- This operation can take time (in order of few minutes). The tool is simple enough, so you can also tweak it for your custom requirements.
- JIRAs that are in 'closed' state cannot be edited, so you might hit 'unauthorized exception'. If you encounter such JIRA, please reopen that JIRA and 'resolve' it instead.
Create a branch for release
Assuming that you are preparing to release version x.y.z
from the current commit, then create a branch with:
git checkout -b x.y.z git push origin x.y.z
Reserve new version for next release
Update the version in gradle.properties in master branch. Eg:
version=x.z.z
Also, if not done already create a new version in issues.apache.org/jira/browse/GOBBLIN using JIRA administration dashboard. You can specify the start date as of today and leave the end date as blank.
Create a Source Release
The following steps will build a tarball suitable for an ASF source release. This also generates accompanying MD5 and ASC files.
First, clean any files unknown to git (WARNING: this removes all untracked files, including those listed in .gitignore, without prompting):
git clean -fdx
Alternatively, you can make a fresh clone of the repository to a separate directory:
git clone https://git-wip-us.apache.org/repos/asf/incubator-gobblin.git incubator-gobblin-release cd incubator-gobblin-release
To generate the source release, run:
./gradlew clean release -Prelease=true
This generates a source tarball. The -Prelease=true
setting prevents -SNAPSHOT
from being appended to the version, which is the default behavior. It also prevents any builds from the extracted source tarball from including -SNAPSHOT
in the version. It achieves this by modifying gradle.properties
within the generated archive.
The source tarball needs to be signed. You can do this either manually or automatically. If you have configured your key information in your gradle.properties
then your archive should automatically be signed. There should now be a corresponding ASC file alongside the tarball and MD5 file. Otherwise you'll need to sign it manually by running:
gpg --sign --armor --detach-sig build/distribution/source/apache-gobblin-incubating-sources-*.tgz
Upload the Source Release
You should make the release candidate available in https://dist.apache.org/repos/dist/dev/incubator/gobblin/. For example, if you are releasing release candidate RC0 for version x.y.z
then you should upload the source distribution files to:
https://dist.apache.org/repos/dist/dev/incubator/gobblin/apache-gobblin-incubating-x.y.z-rc0/
To create a release folder and to check it out (be sure to substitute x.y.z for the actual version), run the following:
svn mkdir https://dist.apache.org/repos/dist/dev/incubator/gobblin/apache-gobblin-incubating-x.y.z-rc0 svn co https://dist.apache.org/repos/dist/dev/incubator/gobblin/apache-gobblin-incubating-x.y.z-rc0 apache-gobblin-incubating-x.y.z-rc0 cd apache-gobblin-incubating-x.y.z-rc0
You could then add the source release as described above and commit.
Tag the release
You should tag the release candidate in git. Assuming you are releasing release candidate RC0 for version x.y.z
then you can attach a tag to the current commit with:
git tag -s release-x.y.z-rc0 -m 'Apache Gobblin (incubating) x.y.z RC0'
Then push the tag:
git push origin release-x.y.z-rc0
Note: If on Mac, you might also need to do this:
brew install pinentry-mac
export GPG_TTY=$(tty)
Staging artifacts in Nexus
General information on publishing to Maven can be found here. To build and upload the archives to the Apache Nexus staging repository, run:
./maven-nexus/maven-nexus.sh -remote
Note that if you are running the above command from the source archive, then you don't need -Prelease=true
. The above command assumes you have configured ~/.gradle/gradle.properties
with your GPG key information.
If you now visit the Apache Nexus Repository and click on Staging Repositories, you should see a repository named orgapachegobblin-xxxx, where xxxx is some number. Select the repository and browse the content to make sure the set of files looks right. If it looks correct then Close the repository. The repository is now ready for use. If you look at the summary there is a URL for the repository that may be used to fetch the archives.
Let's suppose you have a Gradle project you'd like to use to test Gobblin. You can add the URL for the Staging Repository to your build.gradle
like this:
repositories {
mavenCentral()
maven {
url 'https://repository.apache.org/content/repositories/orgapachegobblin-xxxx'
}
}
You could also visit the Staging Repository URL in your browser and download the artifacts.
Call for a vote to release
At this point you should have:
- Published a source release
- Staged artifacts in Nexus built from source
Now you can call a vote in the Gobblin dev mailing list for release. Look in the archives for previous votes for an example. If the vote passes then you may call a vote in the Incubator general mailing list (this is necessary because Gobblin is still incubating).
Here is a sample call for vote email: https://www.mail-archive.com/dev@gobblin.incubator.apache.org/msg01352.html
Note: You should allow for 72 hours voting period for both - dev@gobblin mailing list and incubator general mailing list.
Publishing the release
Once the Incubator general vote has passed, you can publish the source release and artifacts.
Source Release
The Gobblin source release are checked into SVN under https://dist.apache.org/repos/dist/release/incubator/gobblin.
To see all the previous releases, run:
svn list https://dist.apache.org/repos/dist/release/incubator/gobblin
Create a directory for the release (replace x.y.z
with the release number) by running:
svn mkdir https://dist.apache.org/repos/dist/release/incubator/gobblin/apache-gobblin-incubating-x.y.z svn co https://dist.apache.org/repos/dist/release/incubator/gobblin/apache-gobblin-incubating-x.y.z apache-gobblin-incubating-x.y.z-release cd apache-gobblin-incubating-x.y.z-release
Now copy the source release files into this directory and commit them. Within 24 hours they will be distributed to the mirrors. Then it should be available for download at http://www.apache.org/dyn/closer.cgi/incubator/gobblin/apache-gobblin-incubating-x.y.z/
Artifacts
To distribute the artifacts, simply select the staged repository for Gobblin that you prepared in Nexus and chooose Release. They should then be available within the next day or so in the central repository.