Child pages
  • Release Process
Skip to end of metadata
Go to start of metadata

Jena Release Process

This document outlines the procedure for producing a Jena release including some gotchas that may catch people out.

Pre-Release Preparation

Email the dev@ list to say as release is imminent, to let people finish up anything they want to get into the next release.

Check version numbers and dependencies between modules.  These should be the current SNAPSHOTs.

Note new dependencies (these will need to be checked for any LICENSE/NOTICE additions needed).

Notes

The Jena build includes a RAT check for missing licenses. The build should have the necessary exclusions and these are checked even on SNAPSHOT development builds. Jena is not RAT-clean if the RAT tool is run unconfigured from the command line - there are many very small testing files that don't have a license statement in them, often because they are copies of W3C files.

It is very important to look for any java files without a license - there should be none.

A useful way to check 3rd party licenses is to run:

mvn license:aggregate-add-third-party
egrep -iv "BSD|ASF|MIT|CDDL|EPL|Apache|Eclipse|Public Domain" target/generated-sources/license/THIRD-PARTY.txt

This creates a file in {{target/generated-sources/license/THIRD-PARTY.txt}} for all dependencies, recursively.  This check is not a perfect replacement from manual checking but it is useful to assist in that process when dependencies change.


 

You must use maven 3.5.0 or later.

Creating a Release Candidate

Step 1 - Make a fresh clone of the code base

> git clone https://git-wip-us.apache.org/repos/asf/jena.git JENA
> cd JENA

Read/Write Connection Required

Don't forget the https - the release candidate creation process will need to do some git commits so must use a read/write connection to GIT

Cloning from Existing Local Clone

To speed up the cloning process it may be more efficient to do a local clone from your existing clone and then update the remote e.g.

> git clone --no-hardlinks /path/to/repo JENA
> git remote set-url origin https://git-wip-us.apache.org/repos/asf/jena.git

This avoids having to create a new clone over HTTPs while still allowing the later stages of the process to do the necessary commits and pushes.

You should make sure that your new clone has appropriate user name information configured:

> git config user.name "Your Name"
> git config user.email you@apache.org

Next you should ensure that you are on the appropriate branch, we release from master which you can ensure like so:

> git checkout master

Check git configuration:

Possible Git Configuration Issue

If you are using a more recent version of Git (1.8.5 onwards I think) then there is a behavioural change in git which can cause the release plugin to fail to actually prepare releases properly (it won't commit anything) unless you have appropriately configured your repository to have backwards compatible behaviour.

See SCM-740 for some discussion on this, you may need to set the following:

> git config status.displayCommentPrefix 1

You can check if your repository is appropriately configured by doing the following:

> git status
# On branch master
# Your branch is up-to-date with 'origin/master'.
#
nothing to commit, working directory clean

If the above lines are not prefixed with a # then your repository is not appropriately configured

Finally, make sure that you don't have any spurious changes locally (and that nobody has pushed any changes while you were doing your clone).

Step 2 - Make a clean maven repo

It is strongly suggested that you create a brand new maven repository. If you've previously done a release candidate for a prior release you should ensure that the contents of this repository are emptied before continuing.

For runnign maven, create a settings file containing the following:

<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <localRepository>${user.home}/.m2/jena-release</localRepository>
  <servers>
    <server>
        <id>apache.releases.https</id>
        <username>username</username>
        <password>password</password>
    </server>
  </servers>
</settings>

This will create a special jena-release repo under your standard maven install, and configures the credentials for the Apache releases you will need in later steps. You may need to ensure that the jena-release directory exists before carrying on,

Subsequent steps will refer to this settings file as $SETTINGS

If your default maven setup includes your Apache username and password, you can use this instead:

mvn -Dmaven.repo.local=${user.home}/.m2/jena-release ...

Step 3 - Check/Set Versions

All the modules should be at the SNAPSHOT for the release version.

Normally, this is a no-work stage.

Then set an environment variable to pass these to maven, this removes the need to type them during the build process.

# Set versions
export VER="-DreleaseVersion=x.y.z -DdevelopmentVersion=x.next.z-SNAPSHOT -Dtag=jena-x.y.z-rcN"

Step 4 - Maven Release Dry run

Run the following maven command:

> mvn -s $SETTINGS release:clean release:prepare $VER -DdryRun=true -Papache-release

The defaults for "mvn release:prepare" should mostly be fine.

Check when running this step but if the project is setup correctly no other changes should be needed.

Step 5 - Check the release setup

Check the produced release.properties file, in particular check the versions numbers and the SCM tag all look OK.

Then check the artifacts produced in the various target directories: at the moment they will have the SNAPSHOT version because it was a dry run. The ".asc" files should exist.

  • JENA/target/jena-VER-source-release.zip
  • JENA/apache-jena/target/apache-jena-VER.tar.gz and .zip
  • JENA/apache-jena-fuseki2/target/apache-jena-fuseki-VER.tar.gz and .zip
  • JENA/apache-jena-osgi/target/jena-osgi-VER.jar

Step 6 - Run Maven Release

Run the following maven commands:

> mvn -s $SETTINGS release:clean release:prepare $VER -Papache-release
> mvn -s $SETTINGS release:perform $VER -Papache-release

It is recommended that you use a tag of the form jena-x.y.z-rc1 in $VER in case you later need to cancel the vote and create a new release candidate from a later commit

Common/Known Issues with Maven Release

SNAPSHOT dependencies

The release dry run checks for use of SNAPSHOTs. If a module depends on a SNAPSHOT for testing (e.g. jena-jdbc-driver-bundle, jena-jdbc-driver-remote for a version of Fuseki), the dry run offers to clean this up. Accept the offer and set the version to the intended release version. The actual real run does not, and correctly updates the dependency. This may depend on the version of the release plugin.

Problem transferring artifacts

If you receive errors about failing to transfer artifacts and the error mentions response 401 this means you are not properly authenticating to the Apache Maven repository. Ensure that you are using a correctly configured settings.xml file per Step 2

Recovering from Release Failures

If all else fails:

> mvn -s $SETTINGS release:rollback

Or in the worse case you can use git revert to revert the POMs to a state for cutting a fresh release candidate.  You will need to revert the two maven-release-plugin commits whose hashes you first need to find by viewing the git log.  Note that my git log output is set to default to a quite succinct output format so your git may produce a much more verbose log:

> git log

[942ae23] - rvesse 26 hours ago:  [maven-release-plugin] prepare for next development iteration
[db72eb8] - (tag: jena-2.12.1-rc1) rvesse 26 hours ago:  [maven-release-plugin] prepare release jena-2.12.1
> git revert 942ae23
> git revert db72eb8

Each of the two commits should be reverted individually or git will complain about conflicts and you must make sure to revert them in the appropriate order.  Each revert will require you to make a new commit message though one will be auto-suggested for you.

You should also make sure you do a hard reset of your clone:

> git reset --hard

Then manually check the POMs to check that the expected versions are present.

Finally clean the maven local repository for stuff from the build

rm -rf .m2/repository/org/apache/jena/jena-MOD/VER/ etc
mvn -s $SETTINGS clean

Step 7 - Close the Staging Repository

Finding the Repository

The staging repository can be found by going to http://repository.apache.org - you should be able to log in with your normal Apache login. Hit the Staging Repositories link in the sidebar to see the list of repositories, if the release went OK you should see a org.apache.jena-NNN (u: username a: your IP) where NNN is the repository number, chosen by the staging repository for the release candidate you are preparing.

Tick the box next to it in the list and it will display the artifacts in that repository in a separate pane below, browse through this to check that all the expected artifacts are present

Closing the Repository

Close it by selecting it in the Staging repositories list and then clicking the Close link. This can take a few minutes; the interface may need refreshing with F5 to see the "closed" state. When closed, get the URL for the repository by selecting (top panel) and looking at the "URL" in the bottom panel. You cannot browse the repository by URL until you close it.

If closing fails then you can click the Activity tab to see the verification checks Apache Nexus ran on the staging repository and clicking on any failed rules will tell you why those failed.

Apache Nexus will attempt to verify the GPG signature on artifacts during closing the repository. If your key is not uploaded to one of the Key servers used, or one of those key servers is down you won't be able to close the repository. Ensure your key is appropriately uploaded to public key servers and then try closing the repository again.

Once closed you can click on it, copy the URL from the address bar as you will need to reference this later.

Step 8 - Get the distribution tools

Before you can prepare the distribution you need to get the dist tools, check these out into a clean location:

svn co https://svn.apache.org/repos/asf/jena/dist/

KEYS File

If you have not made a release or added your key to the KEYS file previously you should do this now:

gpg --list-sigs "Your Name" >> KEYS
gpg --armor --export "Your Name" >> KEYS

Remember to commit the KEYS file after you have done this. See Signing Releases for more notes on the KEYS file.

Step 9 - Build the distribution hierarchy

Get the script dist.sh and edit it to set the repo number NNN then run it to create a new directory dist-jena/.

# Create the distribution area in dist/jena/
./dist.sh x.y.z

This script download the necessary files from the nexus server.

The script will create a new directory dist-jena/ under which you should find all the artifacts appropriately laid out.

Fixup checksums

MD5 is not total secure and SHA1 is now considered too weak. The Apache Software Foundation is discouraging the use of MD5 and SHA1 checksums for the source release files in apache.org/dist/jena/source.

Binary artifacts going to the maven repositories are not affected. Use of ".md5" and ".sha1" is currently hardwired into the maven toolset.

Checksums for files that will reside in apache.org/dist/jena/source on the Apache Software Foundation servers need fixing up.

The build process creates a SHA512 checksum in "jena/target"; it is not part of the maven repository upload. The "dist.sh" script looks for this in "../jena/target" so if you run the script in a directory that is a sibling of "jena" (the cloned github repo), it will find the file, otherwise set environment variable "SRC" or edit the script.

An alternative way is to script chksum (from the dist-tools svn repo) which will add .sha512 checksums, and delete the .md5 and .sha1 checksums. 

cd dist-jena/binaries/
../../chksum
cd ../source
../../chksum

Step 10 - Upload the Release Candidate

You now need to copy the proposed release candidate to svn repository for release candidates, https://dist.apache.org/repos/dist/dev/jena/.

svn co  https://dist.apache.org/repos/dist/dev/jena jena-dist-dev
# Remove any old stuff
( cd jena-dist-dev ; svn rm binaries/* sources/* )

Now copy in distribution and commit it.

cp -R dist-jena/* jena-dist-dev
# Add the new files.
svn add jena-dist-dev/binaries/* jena-dist-dev/source/*
# Commit (this is slow)
svn commit -m "Jena X.Y.Z RC N" jena-dist-dev

Step 11 - Local Clean Up

Go and clean up your local maven repositories and be sure to do a "git pull" on your working copy of trunk to grab the updated POMs.

Step 12 - Call the VOTE

Look at the email archives for sample vote emails e.g.

Make sure to include links to the Staging Repository, SCM Tag, Proposed Release Candidate URL and KEYS file URL

Jena uses git: you MUST provide a commit hash as well as a tag for the release candidate since git tags are mutable.

Step 13 - Rebuild the jena development build on Jenkins

This should happen automatically but you can do it now, rather than wait for the daily build.  See http://builds.apache.org.

Making a Release

If a Release Candidate vote passes successfully then you need to follow these steps to make it an official release, see Apache Releases FAQ if you need more detail on how to do any of these steps.

Step 1 - Release the staging repo

Go to http://repository.apache.org and login with your Apache account, use the Staging Repositories link from the sidebar to access the list of staged repositories. Tick the box next to the Apache Jena repository that was created when you prepared the release candidate. Hit the Release link at the top of the page, enter "Release Jena X.Y.Z" in the description box and hit OK. You should get a Repositories successfully released message after a few seconds.

Step 2 - Install the release files: source and binaries

Like everything else Apache distributions are made via a svnpubsub process.

The process is to clear up the previous release, then move the source and binaries from the proposed area to the released area.

MSG="Jena X.Y.Z Release"

svn rm -m "Remove previous release" https://dist.apache.org/repos/dist/release/jena/binaries
svn cp -m "$MSG" https://dist.apache.org/repos/dist/dev/jena/binaries      https://dist.apache.org/repos/dist/release/jena

svn rm -m "Remove previous release" https://dist.apache.org/repos/dist/release/jena/source
svn cp -m "$MSG" https://dist.apache.org/repos/dist/dev/jena/source        https://dist.apache.org/repos/dist/release/jena

Make sure KEYS and HEADER are up to date.

svn rm -m "Remove old version"  https://dist.apache.org/repos/dist/release/jena/KEYS
svn cp -m "$MSG" https://dist.apache.org/repos/dist/dev/jena/KEYS          https://dist.apache.org/repos/dist/release/jena

svn rm -m "Remove old version"  https://dist.apache.org/repos/dist/release/jena/HEADER.html
svn cp -m "$MSG" https://dist.apache.org/repos/dist/dev/jena/HEADER.html   https://dist.apache.org/repos/dist/release/jena

(Optional) You can now clear up dist/dev/jena for next time. It is easier to do this using your local copy of dist/dev/jena and commit the changes but if you want to do it remotely you can svn rm the directories dist/dev/jena/binaries and dist/dev/jena/sources. Make sure it is the dist/dev/jena versions, which is easier to use your local copy of dist/dev/jena for this.

Congratulations! You have just released Apache Jena.

Withing  a few minutes http://apache.org/dist/jena/ will be updated - it is asynchronous and you probably need a hard refersh (cntrl-F5) to see the changes appear.

Remember: everything is in svn so if anything goes wrong, you can get back the original files.

Step 3 - Reset versions in JIRA

Go into JIRA and mark the versions as released, ensure the new snapshot versions are entered if they aren't already.

To do this you should log into JIRA and then go to the Administration link in the top right corner. Click on Jena under the Projects list on the Administration page, then on the Project config page click on Versions in the left hand sidebar. If the releases don't already have a release date listed for them you need to click on the little cog on the right hand side of the row containing the version being released and select "Release" from the drop down.

If versions have not been added already for the next releases you should add those now.

Close all "resolved" tickets for this version.

Step 4 - Update the website

Update the Download Pages

Check the following pages to ensure they are up to date.

Create Javadoc

The dist-tools/ has a helper script for preparing new javadoc.  Check the settings (it needs the version and either the staging repo or where the local copies are for the javadoc.jar files).

In a clean directory

cp dist-tools/Javadoc new-javadoc
cd new-javadoc
# Check prepare-javadoc
./prepare-javadoc

This creates a directory "J/".

It does not create a new index.mdtext for the top level directory but does create them for all sub-areas.

See "dist-tools/Javadoc/README" for additional notes.

Update the Javadoc

For Javadoc, you have to use the publish.pl script as a lot of change overwhelms web CMS. It is easier to delete the old Javadoc and add the new.

You can check out a copy of the website by doing the following:

svn co https://username@svn.apache.org/repos/asf/jena/site/trunk/ WEBSITE

Then in the website directory do the following to remove the old documentation:

cd content/documentation/javadoc/

svn rm *
svn commit -m "Remove previous javadoc" .

cp -R ???/J/* .
svn add *
# This is a slow step.
svn commit -m "Javadoc for Jena ?.?.?" .

Splitting the changes so that all the deletions are committed then the new javadoc added makes the process more reliable.

The svn commit may take an extremely long time, it may be best to do commits on the individual module directories particularly if you have limited bandwidth or are concerned about dropping your connection partway through the commit

Publishing the Javadoc

You then need to go the CMS to actually publish the updated docs, firstly check the status of the site build by going to http://ci.apache.org/builders/jena-site-staging. If you see anything listed under Current Builds then the build is still running, wait a few minutes and then refresh (the build can take a long time for big changes).

If you delete the old documentation and add the new documentation in the same commit the CMS build may fail with the following error:

Unmergeable scheduling requested on an entry: Can't add '/usr/local/websites/jena/trunk/content/documentation/javadoc/arq/META-INF' to a parent directory scheduled for deletion at /usr/local/cms/build/lib/ASF/SVNUtil.pm line 69

To resolve this you will need to do the following:

> svn mv javadoc temp
> svn commit -m "Temporarily move javadoc"

Then wait for the CMS to build which should now succeed, then do the following:

> svn mv temp javadoc
> svn commit -m "Move javadoc back to correct location"

And again the build should now complete successfully .

Once there are no Current Builds running click on the most recent build listed to ensure it matches your commit message and has succeeded. You can now go to https://cms.apache.org/jena/publish and enter a commit message stating that you are publishing the updated Javadoc. Note that you can view the diff if you want but it may take a long time for the CMS to generate it especially if a lot of documentation has changed.

Publish the updated Jena site.

You can use the web CMS to make other changes and should go ahead and publish immediately.

Using the publish.pl script is easier - the very large changes the Javadoc causes makes CMS publishing slow/unreliable.

Any addition checks and changes can be done via the CMS web interface.

Step 5 - Apply final Git Tag

You can now apply the final git tag for the release:

> git tag jena-x.y.z jena-x.y.z-rc1
> git push origin jena-x.y.z

Note that you must push the tag as otherwise it is only in your clone and not visible in the official repositories.

Step 6 - Release

Wait 24 hours to allow all the Apache mirrors to sync up.

Announce via email to:

  • users@jena.apache.org
  • announce@apache.org (include a one paragraph general description of what Jena is)
  • semantic-web@w3.org (if significant changes)

Voting on a Release

When voting on a release candidate the following points need to be checked:

  • have there been any library changes? if so, check LICENSE and NOTICE.
  • do the signatures verify?
  • is there a source archive?
  • can the source archive really be built?
  • is there a correct LICENSE and NOTICE file in each artifact (both source and binary artifacts)?
  • does the NOTICE file contain all necessary attributions?
  • check the dependencies.
  • do all the tests work?
  • if there is a tag in the SCM, does it contain reproduceable sources?
  • No labels