This document describes how to make a release of Sqoop 2 from git branch "sqoop2". It is a work in progress and should be refined by the Release Manager (RM) as they come across aspects of the release process not yet documented here.
NOTE: For the purpose of illustration, this document assumes that the version being released is 1.99.1, and the following development version will become 1.99.2
Prerequisites
Policy documents
Frequently asked questions for making Apache releases are available on Releases FAQ page.
The Release Manager (RM) must go through the policy document to understand all the tasks and responsibilities of running a release.
Give a heads up
The RM should first create an umbrella issue and then setup a timeline for release branch point. The time for the day the umbrella issue is created to the release branch point must be at least two weeks in order to give the community a chance to prioritize and commit any last minute features and issues they would like to see in the upcoming release.
The RM should then send the pointer to the umbrella issue along with the tentative timeline for branch point to the user and developer lists. Any work identified as release related that needs to be completed should be added as a subtask of the umbrella issue to allow users to see the overall release progress in one place.
To: dev@sqoop.apache.org Subject: Work on $release release has started We are starting the process to prepare for Sqoop $release release. I have opened JIRA $jira to cover the tasks under this release. If you have any JIRA in progress and would like to include it in this release, please follow the process to do so. Feel free to comment on the JIRA if you have any comments/suggestions. Thanks, $RM
Sanitize JIRA
Before a release is done, make sure that any issues that are fixed have their fix version setup correctly. Run the following JIRA query to see which resolved issues do not have their fix version set up correctly:
project = sqoop and resolution = fixed and fixVersion is empty
The result of the above query should be empty. If some issues do show up in this query that have been fixed since the last release, please bulk-edit them to set the fix version to '1.99.1'.
You can also run the following query to make sure that the issues fixed for the to-be-released version look accurate:
project = sqoop and resolution = fixed and fixVersion = '1.99.1'
Monitor active issues
It is important that between the time that the umbrella issue is filed to the time when the release branch is created, no experimental or potentially destabilizing work is checked into the trunk. While it is acceptable to introduce major changes, they must be thoroughly reviewed and have good test coverage to ensure that the release branch does not start of being unstable.
If necessary the RM can discuss if certain issues should be fixed on the trunk in this time, and if so what is the gating criteria for accepting them.
Creating Release Artifacts
Create a release branch
Creating branches in git is easy, simple, cheap, fast and race condition less with comparison to SVN. There is no need explicitly announce that you're going to create release branch and ask everyone to stop committing.
1. Clone fresh repository copy
git clone https://git-wip-us.apache.org/repos/asf/sqoop.git cd sqoop
2. Checkout Sqoop 2 branch
git checkout sqoop2
3. Check that current HEAD points to commit on which you want to base new release branch. Checkout particular commit if not.
git log # Check current branch history. HEAD should point to commit that you want to be base for your release branch git checkout abcdef123 # Check out particular commit that should be base for release branch if -^
4. Create new release branch with name "branch-$version"
git branch branch-1.99.1
5. Push new branch to Apache repository
git push origin branch-1.99.1
6. Check that branch was correctly propagated to Apache repository. You should be able to see your branch on following git webui.
7. Update version to your release version in all pom.xml files (.pom.xml, */pom.xml and *//pom.xml). You should be changing the version only in <parent> tag (with exception of root pom.xml file that obviously do not have a parent).
simple trick to get all the pom.xmls changed
grep 2.0.0-SNAPSHOT * -RF | grep pom.xml
8. Check your changes by running package build
mvn package
9. Commit your changes and push them to Apache repository
git commit -a -m "Changing version to 1.99.1" git push origin branch-1.99.1
10. Send an email announcing new branch
To: dev@sqoop.apache.org
Subject: New release branch 1.99.1I've just created new release branch for upcoming 1.99.1 release. Please continue committing to main Sqoop 2 branch "sqoop2" as usual, I'll cherry-pick commits to branch-1.99.1 on per needed basis.
https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=shortlog;h=refs/heads/branch-1.99.1
Thanks,
$RM
Update change log file
1. It's important to update changelog file with all changes that went it to the release. JIRA is generating list of all changes in particular release in "Release notes" section:
- On main page click "versions" from left menu
- Click on your version
- There is link "Release notes" on top left corner of the context area.
2. You can use ruby script jira2changelog.rb attached to this page to transform this page into desired output.
jira2changelog.rb 'https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311320&version=12323641' # Change with your version Release notes
3. Update CHANGELOG.txt file in branches sqoop2 and branch-<your-version>.
4. It's important to keep this file up-to-date with each committed change after initial branch cut.
Create release tarballs
1. Check out release branch
git clone https://git-wip-us.apache.org/repos/asf/sqoop.git git checkout branch-1.99.1
2. It may be worth checking to make sure we don't have any files that are out of sync with git before we start packaging.
git status --ignored
3. Make sure that LICENSE.txt contains license information for all third-party libraries. In order to do this check properly, you will need to quickly create all packages. You need to check jars available in client/lib, server/lib and insider sqoop server war file (server/webapps/sqoop.war).
//mvn package -DskipTests
//mvn package -DskipTests -Pbinary -Dhadoop.profile=100
// Since release 1.99.5 we have one hadoop profile we support
mvn package -DskipTests -Pbinary -Dhadoop.profile=200
// This script will help determine which dependencies are used but not in the existing license file
ls -1 dist/target/sqoop-2.0.0-SNAPSHOT-bin-hadoop200/*/lib/ | grep ".jar" | grep -v "SNAPSHOT.jar" | sort -u | xargs -I {} ruby -e "file = '{}'; cutoff = file.index(/\d+\./); if cutoff then puts file.slice(0..cutoff-2) else puts file.slice(0..file.index('.jar')-1) end" | xargs -I {} sh -c 'if ! grep --quiet "For {}[-<version>]*.jar" LICENSE.txt ; then echo {} ; fi'
// This script will help determine which dependencies are in the license file but are not currently being used
grep "For .*jar" LICENSE.txt | xargs -I {} ruby -e "jar = '{}'; jar = jar.slice(4..jar.length-1); if jar.include?('<version>.jar') then puts jar.slice(0..jar.index('<version>.jar')-2) else puts jar.slice(0..jar.index('.jar')-1) end" | xargs -I {} sh -c 'if ! ls -1 dist/target/sqoop-2.0.0-SNAPSHOT-bin-hadoop200/*/lib/ | grep --quiet '{}' ; then echo '{}' ; fi' | sort
3. Create tag on this commit to identify precise point where the RC was generated and push this tag to main repository
git tag -a release-1.99.1-rc0 -m "Sqoop 1.99.1-rc0 release" git push origin release-1.99.1-rc0
4. Create temporary directory where you'll be preparing all required artifacts
mkdir -p /tmp/sqoop-release-preparations
5. Create source artifact and move it to your temporary directory
mvn clean verify package -Psource mv dist/target/sqoop-1.99.1.tar.gz /tmp/sqoop-release-preparations
6. Create binary artifact for each supported hadoop version and move generated artifacts to your temporary directory
// Since release 1.99.5 we have one hadoop profile we support
for version in 200; do mvn clean verify package -Pbinary -Dhadoop.profile=$version; mv dist/target/sqoop-1.99.1-bin-hadoop${version}.tar.gz /tmp/sqoop-release-preparations ; done
Update KEYS file
If your PGP key is not yet in the project's KEYS file, you need to first add that in. To do this, checkout the KEYS file and update it using the following commands:
$ svn co https://dist.apache.org/repos/dist/release/sqoop sqoop-release ... $ cd sqoop-release $ (gpg --list-sigs <KEY-ID> && gpg --armor --export <KEY-ID> ) >> KEYS $ svn commit -m "Adding PGP public key to KEYS file" KEYS ...
Once this file has been updated, it will automatically be pushed to http://www.apache.org/dist/sqoop/KEYS.
This will take some time to propagate in which you can continue with the other steps of the release process.
Create signatures and check sums
1. Change your working directory to the temporal one
cd /tmp/sqoop-release-preparations
2. Sing each file with your key
for file in *.tar.gz; do gpg --armor --output $file.asc --detach-sig $file; done
3. You can immediately verify your signature
for file in *.tar.gz; do gpg --verify $file.asc $file; done
4. Create md5 check sum
for file in *.tar.gz; do md5sum $file > $file.md5; done
5. Create sha1 check sum
for file in *.tar.gz; do sha1sum $file > $file.sha; done
The structure of the file names is given by Apache Software foundation and can't be changed. The name must be in form <tarball_name>.tar.gz.<analysis> (for example sqoop-1.99.3.tar.gz.asc). Having name sqoop-1.99.3.asc (without .tar.gz) is not acceptable!
6. Upload artifacts and all created check sums with signatures to to repos/dist/dev/sqoop
svn checkout https://dist.apache.org/repos/dist/dev/sqoop sqoop-dev mkdir 1.99.1_rc0 cd 1.99.7_rc0 cp /tmp/sqoop-release-preparations/* . svn add . svn commit -m "Stage Sqoop 1.99.1 RC0 artifacts"
Running the vote
Call for sqoop dev list votes
Send an email to dev@ list. For example,
To: dev@sqoop.apache.org Subject: [VOTE] Release Sqoop version 1.99.1 This is the first cut of Sqoop 2 branch, version 1.99.1. Purpose of this release is to provide early bits to our users so that they can test the product and provide early feedback.This is the first cut of Sqoop 2 branch, version 1.99.1. Purpose of this release is to provide early bits to our users so that they can test the product and provide early feedback. *** Please cast your vote by Friday 2012-12-21 *** The list of fixed issues: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=blob;f=CHANGELOG.txt;h=c22af4131a3061dc78922a82a68eedc7e9ac599e;hb=85a8e1a8b3445360d66c8f812947cb0a7e8230c9 The tarball (*.tar.gz), signature (*.asc), checksum (*.md5, *.sha): http://people.apache.org/~jarcec/sqoop-1.99.1-rc0/ The tag to be voted upon: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=tag;h=refs/tags/release-1.99.1-rc0 The KEYS file: http://www.apache.org/dist/sqoop/KEYS
Need 3 +1 votes from PMC members.
You can close the vote after voting period expires when you accumulate sufficient votes. Example close email:
To: dev@sqoop.apache.org Subject: [CLOSED] [VOTE] Release Sqoop version 1.99.1 Thanks everyone for voting for this release, I'm closing the vote now. I'll send the results shortly.Thanks everyone for voting for this release, I'm closing the vote now. I'll send the results shortly. $RM
After closing the vote, you need to send results. Example email:
To: dev@sqoop.apache.org Subject: [RESULTS] [VOTE] Release Sqoop version 1.99.1 This vote passes with 8 +1 votes (4 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Bilung Lee * Olivier Lamy * Arvind Prabhakar * Jarek Jarcec Cecho Committers: * Cheolsoo Park * Abhijeet Gaikwad Community: * Hari Shreedharan * Venkatesan Ranganathan 0 votes * No votes -1 votes * No votes Vote thread: http://markmail.org/message/faioizetvcils2zo I'll continue in the release process. Jarcec
Rolling out the Release
Close JIRA version
You need to close the release in JIRA so that everyone knows that your version should not be used as "fixVersion" for new bugs. Go to JIRA "Administer project" page and follow "Versions" in left menu. Table with list of all releases should appear, click on additional menu on the right of your release and choose "Release" option. Submit release date and you're done.
Upload the artifacts
Sqoop is using SVN based release procedure. In order to release you have to checkout release repository located on https://dist.apache.org/repos/dist/release/sqoop and add release artifacts there. Note: this step can only be performed by a PMC member.
svn co https://dist.apache.org/repos/dist/release/sqoop sqoop-release cd sqoop-release mkdir 1.99.1 cp $source_to_your_artifacts 1.99.1/ svn add 1.99.1 # Optionally change KEYS file in case that you've added your key for the first time svn commit -m "Release 1.99.1"
Wait for 24 Hours
It may take up to 24 hours for all mirrors to sync up.
Remove old releases from dist area
Don't worry, they will be archived on archive.apache.org. We need to release the pressure on all Apache mirrors so only most recent releases are allowed to be in the distribution area. Note: this step can only be performed by a PMC member.
Publish binary artifacts in Maven Repository
1. Configure your Apache credentials in your maven settings.xml file (usually in ~/.m2/settings.xml). More information is available in official guide.
<settings>
<servers>
<server>
<id>apache.releases.https</id>
<username>APACHE-ID</username>
<password>APACHE-PASSWORD</password>
</server>
</servers>
</settings>
2. Go to release branch (tag that was released)
3. Create, sign and deploy artifacts into staging repository
for version in 200; do mvn clean deploy -Psign -Dhadoop.profile=$version ; done
4. Login to https://repository.apache.org and select "Staging Repositories" under "Build Promotion" menu on the left.
5. Select org.apache.sqoop from the list of repositories and click "Close" using "Apache Sqoop 1.99.1" as description.
6. Select org.apache.sqoop from the list of repositories and click "Release" using "Apache Sqoop 1.99.1 artifacts" as description
Update webpages
You need to update Sqoop webpages to reflect new release. Instructions are on separate wiki page.
Announce the release
Send an email to announce@apache.org (the from: address must be @apache.org). For example,
To: announce@apache.org, user@sqoop.apache.org, dev@sqoop.apache.org Subject: [ANNOUNCE] Apache Sqoop 1.99.1 released The Apache Sqoop team is pleased to announce the release of Sqoop 1.99.1. This is the first cut of Sqoop2 branch. Major step forward of the project. Apache Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores, such as relational databases. The release is available here: http://www.apache.org/dyn/closer.cgi/sqoop/1.99.1 The full change log is available here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311320&version=12323641 Your help and feedback is more than welcome. For more information on how to report problems and to get involved, visit the project website at http://sqoop.apache.org/. The Apache Sqoop Team