Jena Release Process

This document outlines the procedure for producing a Jena release including some gotchas that may catch people out.

Pre-Release Preparation

Email the dev@ list to say as release is imminent, to let people finish up anything they want to get into the next release.

Run RAT to check for missing licenses, you can download RAT binaries from the Apache Creadur project. Jena is not RAT-clean - there are many very small testing files that don't have a license statement in them. It is very important to look for any java files without a license - there should be none.

Check version numbers and dependencies between modules. These should be the current SNAPSHOTs.

Creating a Release Candidate

You must use maven3, maven2 is unreliable, sometimes missing out the source-release

Step 1 - Make a fresh clone of the code base

> git clone https://git-wip-us.apache.org/repos/asf/jena.git JENA
> cd JENA

Read/Write Connection Required

Don't forget the https - the release candidate creation process will need to do some svn commits so must use a read/write connection to SVN

Cloning from Existing Local Clone

To speed up the cloning process it may be more efficient to do a local clone from your existing clone and then update the remote e.g.

> git clone --no-hardlinks /path/to/repo JENA
> git remote set-url origin https://git-wip-us.apache.org/repos/asf/jena.git

This avoids having to create a new clone over HTTPs while still allowing the later stages of the process to do the necessary commits and pushes.

You should make sure that your new clone has appropriate user name information configured:

> git config user.name "Your Name"
> git config user.email you@apache.org

Next you should ensure that you are on the appropriate branch, we release from master which you can ensure like so:

> git checkout master

Finally make sure that you don't have any spurious changes locally (and that nobody has pushed any changes while you were doing your clone):

> git status
# On branch master

nothing to commit, working directory clean

Step 2 - Make a clean maven repo

It is strongly suggested that you create a brand new maven repository and create a settings file to go with it, this settings file needs to contain the following:

<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <localRepository>${user.home}/.m2/jena-release</localRepository>
  <servers>
    <server>
        <id>apache.releases.https</id>
        <username>username</username>
        <password>password</password>
    </server>
  </servers>
</settings>

Subsequent steps will refer to this settings file as settings.xml

This will create a special jena-release repo under your standard maven install, it also configures the credentials you will need in later steps. You may need to ensure that the jena-release directory exists before carrying on, if you've previously done a release candidate for a prior release you should ensure that the contents of this repository are emptied before continuing.

Step 3 - Check/Set Versions

If still separate version numbers for each module:

The parent POM should be the SNAPSHOT.
The dependencies should be the SNAPSHOT.

Normally, this is a no-work stage.

Step 4 - Maven Release Dry run

Possible Git Configuration Issue

If you are using a more recent version of Git (1.8.5 onwards I think) then there is a behavioural change in git which can cause the release plugin to fail to actually prepare releases properly (it won't commit anything) unless you have appropriately configured your repository to have backwards compatible behaviour.

See SCM-740 for some discussion on this, you may need to set the following:

> git config status.displayCommentPrefix 1

You can check if your repository is appropriately configured by doing the following:

> git status
# On branch master
# Your branch is up-to-date with 'origin/master'.

If the above lines are not prefixed with a # then your repository is not appropriately configured

Run the following maven command:

> mvn -s settings.xml release:clean release:prepare -DdryRun=true -Papache-release

The defaults for "mvn release:prepare" should mostly be fine, it is recommended that you use a tag of the form jena-x.y.z-rc1 in case you later need to cancel the vote and create a new release candidate from a later commit.

Check when running this step but if the project is setup correctly no other changes should be needed.

Step 5 - Check the release setup

Check the produced release.properties file, in particular check the versions numbers and the SCM tag all look OK.

Then check the artifacts produced in the various target directories:

JENA/target/apache-jena-VER-source-release.zip
JENA/apache-jena/target/apache-jena-VER.tar.gz and .zip
JENA/jena-fuseki/target/jena-fuseki-VER-distribution.tar.gz and .zip

Step 6 - Run Maven Release

Run the following maven commands:

> mvn -s settings.xml release:clean release:prepare  -Papache-release
> mvn -s settings.xml release:perform  -Papache-release

Common/Known Issues with Maven Release

Problem tagging SCM

On "mvn release:clean release:prepare -Papache-release"

If you see something like:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-release-plugin:2.1:prepare (default-cli) on project jena-tdb: Unable to tag SCM
[ERROR] Provider message:
[ERROR] The svn tag command failed.
[ERROR] Command output:
[ERROR] svn: No such revision 1245420

It's because the Apache SVN mirrors aren't quite sync'ed. Resume with:

mvn -s settings.xml release:prepare -Papache-release

and it will continue where it left off. Note : no "release:clean" here.

Problem transferring artifacts

If you receive errors about failing to transfer artifacts and the error mentions response 401 this means you are not properly authenticating to the Apache Maven repository. Ensure that you are using a correctly configured settings.xml file per Step 2

Recovering from Release Failures

If all else fails:

> mvn release:rollback

Or in the worse case you can use git revert to revert the POMs to a state for cutting a fresh release candidate. You will need to revert the two maven-release-plugin commits whose hashes you first need to find by viewing the git log. Note that my git log output is set to default to a quite succinct output format so your git may produce a much more verbose log:

> git log

[942ae23] - rvesse 26 hours ago:  [maven-release-plugin] prepare for next development iteration
[db72eb8] - (tag: jena-2.12.1-rc1) rvesse 26 hours ago:  [maven-release-plugin] prepare release jena-2.12.1
> git revert 942ae23
> git revert db72eb8

Each of the two commits should be reverted individually or git will complain about conflicts and you must make sure to revert them in the appropriate order. Each revert will require you to make a new commit message though one will be auto-suggested for you.

You should also make sure you do a hard reset of your clone:

> git reset --hard

Then manually check the POMs to check that the expected versions are present.

Finally clean the maven local repository for stuff from the build

rm -rf .m2/repository/org/apache/jena/jena-MOD/VER/ etc
mvn clean

Step 7 - Close the Staging Repository

Finding the Repository

The staging repository can be found by going to http://repository.apache.org - you should be able to log in with your normal Apache login. Hit the Staging Repositories link in the sidebar to see the list of repositories, if the release went OK you should see a org.apache.jena-VER (u: username a: your IP) where VER is the version whose release candidate you are preparing.

Tick the box next to it in the list and it will display the artifacts in that repository in a separate pane below, browse through this to check that all the expected artifacts are present

Closing the Repository

Close it by selecting it in the Staging repositories list and then clicking the Close link. Note that if you try to click on the repository URL prior to closing it you will receive a Not Found error from the server, you cannot browse the repository by URL until you close it.

Once closed you can click on it, copy the URL from the address bar as you will need to reference this later.

Step 8 - Get the distribution tools

Before you can prepare the distribution you need to get the dist tools, check these out into a clean location:

svn co https://svn.apache.org/repos/asf/jena/dist/

KEYS File

If you have not made a release or added your key to the KEYS file previously you should do this now:

gpg --list-sigs "Your Name" >> KEYS
gpg --armor --export "Your Name" >> KEYS

Remember to commit the KEYS file after you have done this. See Signing Releases for more notes on the KEYS file.

Step 9 - Build the distribution hierarchy

Firstly you need to fetch the staging repository in order to have the release candidate artifacts available:

mkdir -p repository.apache.org/content/repositories/orgapachejena-NNN/org/apache/jena
wget -e robots=off --wait 1 --mirror -np https://repository.apache.org/content/repositories/orgapachejena-NNN/org/apache/jena

Note: Remember to replace NNN appropriately with the actual ID number of the staging repository you closed back in Step 7

You need a version of wget with HTTPS support for the above to work, if you built and installed wget yourself you must have added the --with-ssl or --with-ssl=openssl option when you ran the configure script. You cannot retrieve the content from the repository using wget without HTTPS installed

After this is done move the mirrored artifacts like so:

mv repository.apache.org/content/repositories/orgapachejena-NNN/ REPO

You are now ready to run the actual script which requires two arguments, the first is the version of the main Jena distribution and the second is the version of the Fuseki distribution. If you want to dry run you can add echo as a third argument:

./dist.sh 2.7.3 0.2.4

The script will create a new directory dist-jena/ under which you should find all the artifacts appropriately laid out.

Step 10 - Upload the Release Candidate

You now need to copy the proposed release candidate to your public web area on people.apache.org e.g. http://people.apache.org/~rvesse

Note that if accessing your URL returns a 404 rather than a directory listing you first need to ssh into people.apache.org and run the following:

mkdir /home/username/public_html/

If this is done you can use your favourite SCP/SFTP tool to upload the release candidate, for example:

scp -r dist-jena people.apache.org:public_html/Jena-VER

You should replace the normal directory browser header included in that upload for the RC.html release header like so:

scp RC.html people.apache.org:public_html/Jena-VER/HEADER.html

This ensures that people browsing to the release candidate directory receive a clear message that it is a release candidate and not an official Apache release.

Step 11 - Local Clean Up

Go and clean up your local maven repositories and be sure to do a svn update on your working copy of trunk to grab the updated POMs

Step 12 - Call the VOTE

Look at the email archives for sample vote emails e.g.

Make sure to include links to the Staging Repository, SCM Tag, Proposed Release Candidate URL and KEYS file URL

Since Jena has now moved to commit your MUST provide a commit hash as well as a tag for the release candidate since Git Tags are mutable

Step 13 - Rebuild the jena development build on Jenkins

This should happen automatically, check http://builds.apache.org

Making a Release

If a Release Candidate vote passes successfully then you need to follow these steps to make it an official release, see Apache Releases FAQ if you need more detail on how to do any of these steps.

Step 1 - Release the staging repo

Go to http://repository.apache.org and login with your Apache account, use the Staging Repositories link from the sidebar to access the list of staged repositories. Tick the box next to the Apache Jena repository that was created when you prepared the release candidate. Hit the Release link at the top of the page, enter "Release Jena X.Y.Z" in the description box and hit OK. You should get a Repositories successfully released message after a few seconds.

Step 2 - Install the release files (source and binaries).

Like everything else Apache distributions are made via a svnpubsub process, firstly check out the Jena distribution area to a clean directory:

svn co https://user@dist.apache.org/repos/dist/release/jena/ SOMEDIR

Once this is checked out you firstly need to remove the old version from the checked out copy, in the dist directory the following commands will do this for you:

cd source
svn delete jena*
cd ../binaries
svn delete apache*
svn delete jena*
cd ../

Then you can copy the proposed distribution area you made in Step 9 of the release candidate into the checked out copy:

cp -R ../PROPOSED-DIST .

Then do the following to ensure all the new artifacts are added to svn:

cd source
svn add jena*
cd ../binaries
svn add apache*
svn add fuseki*
cd ../
svn commit --message "Committing the Jena VER artifacts to the Jena release area"

Step 3 - Reset versions in JIRA

Go into JIRA and mark the versions as released, ensure the new snapshot versions are entered if they aren't already.

To do this you should log into JIRA and then go to the Administration link in the top right corner. Click on Jena under the Projects list on the Administration page, then on the Project config page click on Versions in the left hand sidebar. If the releases don't already have a release date listed for them you need to click on the little cog on the right hand side of the row containing the version being released and select "Release" from the drop down.

If versions have not been added already for the next releases you should add those now.

Step 4 - Update the website

Update the Download Pages

Check the following pages to ensure they are up to date.

http://jena.apache.org/download/index.html – This has references to the current version in the file links so a global replace of the old version numbers with the new version numbers is necessary

Update the Javadoc

For javadoc it is better to use the publish script as a lot of change can overwhelm web CMS. It is easier to delete the old java doc and add the new.

You can checkout a copy of the website by doing the following:

svn co https://username@svn.apache.org/repos/asf/jena/site/trunk/ WEBSITE

Then in the website directory do the following to remove the old documentation:

cd content/documentation/javadoc/
cd arq
svn delete *
 
# Repeat for each module - consult subsequent modules table for complete list
svn commit --message "Deleting old javadoc"

You can then go ahead and copy the updated documentation from your release candidate area like so:

cp -R RC/jena-arq/target/apidocs arq
cd arq
svn add *
cd ../

 
# Repeat for each module - consult subsequent modules table for complete list
 
cd ../
svn commit --message "Adding updated javadoc"

The svn commit may take an extremely long time, it may be best to do commits on the individual module directories particularly if you have limited bandwidth or are concerned about dropping your connection partway through the commit

Javadoc Modules

The following table shows the mappings between the module paths in the website hierarchy and the actual javadoc in the build:

Javadoc module on website	Path to Javadocs within build
arq	jena-arq/target/apidocs
fuseki	jena-fuseki/target/apidocs
jdbc	jena-jdbc/jena-jdbc-driver-bundle/target/apidocs
jena	jena-core/target/apidocs
sdb	jena-sdb/target/apidocs
security	jena-security/target/apidocs
spatial	jena-spatial/target/apidocs
tdb	jena-tdb/target/apidocs
text	jena-text/target/apidocs

Publishing the Javadoc

You then need to go the CMS to actually publish the updated docs, firstly check the status of the site build by going to http://ci.apache.org/builders/jena-site-staging. If you see anything listed under Current Builds then the build is still running, wait a few minutes and then refresh (the build can take a long time for big changes).

If you delete the old documentation and add the new documentation in the same commit the CMS build may fail with the following error:

Unmergeable scheduling requested on an entry: Can't add '/usr/local/websites/jena/trunk/content/documentation/javadoc/arq/META-INF' to a parent directory scheduled for deletion at /usr/local/cms/build/lib/ASF/SVNUtil.pm line 69

To resolve this you will need to do the following:

> svn mv javadoc temp
> svn commit -m "Temporarily move javadoc"

Then wait for the CMS to build which should now succeed, then do the following:

> svn mv temp javadoc
> svn commit -m "Move javadoc back to correct location"

And again the build should now complete successfully .

Once there are no Current Builds running click on the most recent build listed to ensure it matches your commit message and has succeeded. You can now go to https://cms.apache.org/jena/publish and enter a commit message stating that you are publishing the updated Javadoc. Note that you can view the diff if you want but it may take a long time for the CMS to generate it especially if a lot of documentation has changed.

Publish the updated Jena site.

You can use the web CMS to make other changes and should go ahead and publish immediately.

Using the publish.pl script is easier - the very large changes the javadoc causes makes CMS pulishing slow/unreliable.

Any addition checks and changes can be done via the CMS web interface.

Step 5 - Apply final Git Tag

You can now apply the final git tag for the release:

> git tag jena-x.y.z jena-x.y.z-rc1
> git push origin jena-x.y.z

Note that you must push the tag as otherwise it is only in your clone and not visible in the official repositories.

Step 6 - Release

Wait 24 hours to allow all the Apache mirrors to sync up.

Announce via email to:

users@jena.apache.org
announce@apache.org (include a one paragraph general description of what Jena is)
semantic-web@w3.org (if significant changes)

Voting on a Release

When voting on a release candidate the following points need to be checked:

have there been any library changes? if so, check LICENSE and NOTICE.
do the signatures verify?
is there a source archive?
can the source archive really be built?
is there a correct LICENSE and NOTICE file in each artifact (both source and binary artifacts)?
does the NOTICE file contain all necessary attributions?
check the dependencies.
do all the tests work?
if there is a tag in the SCM, does it contain reproduceable sources?

Child pages

Release Process