Versions Compared

Old Version 25

changes.mady.by.user Jeff Hammerbacher

Saved on

compared with

New Version Current

changes.mady.by.user Michael A. Smith

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Use `build` to build python sdist and wheels

This page is prepared for Apache Avro committers. You need committer rights to create a new Apache Avro release.

Table of Contents

Branching

Skip this section if this is NOT the first release in a series (i.e. release X.Y.0).

  1. Update CHANGES.txt to include the release version and date (use Unreleased for the date if it is unknown) and remove Trunk (unreleased changes).
  2. Commit these changes to trunk.
    No Format
    
svn commit -m "Preparing for release X.Y.Z"
  3. Create a branch for the release series:
    No Format
    
svn copy https://svn.apache.org/repos/asf/avro/trunk \
https://svn.apache.org/repos/asf/avro/branches/branch-X.Y -m "Branching for X.Y releases"
  4. Update CHANGES.txt to add back in Trunk (unreleased changes).
  5. Update the default version in share/VERSION.txt on trunk to X.Y+1.0-SNAPSHOT. Be sure not to leave a trailing newline.
  6. Commit these changes to trunk.
    No Format
    
svn commit -m "Preparing for X.Y+1.0 development"

Updating Release Branch

These operations take place in the release branch.

Build environment

I often use Docker to build a clean and consistent build environment:

No Format
# Build a clean docker image from scratch.
DOCKER_RUN_ENTRYPOINT=true DOCKER_BUILD_XTRA_ARGS=--no-cache DOCKER_IMAGE_NAME=avro-build:X.Y-rcN ./build.sh docker

# Run the docker container, automatically mapping ~/.gnupg, ~/.m2 and the current directory to the container.
DOCKER_IMAGE_NAME=avro-build:X.Y-rcN ./build.sh docker

# Omitting DOCKER_IMAGE_NAME will use avro-build-$USER:latest

Branching

Skip this section if this is NOT the first release in a series (i.e. release X.Y.0).

  1. Create a release branch for the release series (from the master):

    No Format
    git checkout -b branch-X.Y


  2. Update the version:

    No Format
    mvn versions:set -DnewVersion=X.Y.0 -DgenerateBackupPoms=false


    1. Update the version number
  3. Check out the branch with:
    No Format
    
svn co https://svn.apache.org/repos/asf/avro/branches/branch-X.Y
  4. Update CHANGES.txt to include the release version and date (this change must be committed to trunk and any intermediate branches between trunk and the branch being released).
  5. Update the version number
    1. in share/VERSION.txt to be ''
    avro-
    1. X.Y.
    Z-SNAPSHOT
    1. 0'
    '
    1. . Be sure
    not
    1. NOT to leave a trailing newline.
    2. Update the version number in lang/js/package.json to X.Y.0
    3. Update the version number in pom-XXXX.xml used for unit tests in the lang/java/maven-plugin/src/test/resources directory too.

  6. Update the release branch.

    No Format
    git
    Commit these changes.
    No Format
    
svn commit -mam "Preparing for release X.Y.Z0"
    Add the fix version 
    git push apache branch-X.Y
    .Z to the Avro JIRA
  7. If not already done, merge desired patches from trunk into the branch and commit these changes. You can find the revision numbers using svn log CHANGES.txt in the branch and in trunk.
    No Format
    
cd branch-X.Y
svn merge -rR1:R2 ../trunk .
svn commit -m "Merge -r R1:R2 from trunk to X.Y branch. Fixes: AVRO-A, AVRO-B."
  8. For each patch merged, change the fix version for the JIRA issue to be X.Y.Z

  10. Return to the master branch.
     No Format
    git checkout master

  11. Update the master version to match the next major version:
     No Format
    mvn versions:set -DnewVersion=X.(Y+1).0-SNAPSHOT -DgenerateBackupPoms=false

    1. Update the version number in share/VERSION.txt to be ''X.(Y+1).0-SNAPSHOT'. Be sure NOT to leave a trailing newline
    Go through CHANGES.txt, JIRA, and svn log to be sure that the issues included in the branch match in each location
    1. .
    2. Update the version number in 
    share
    1. lang/js/
    VERSION
    1. package.
    txt
    1. json to 
     be ''avro-
    1.  X.(Y
    .Z''. Be sure not to leave a trailing newline.
  12. Tag the release candidate (R is the release candidate number): 
     No Format
    
svn copy https://svn.apache.org/repos/asf/avro/branches/branch-X.Y \
https://svn.apache.org/repos/asf/avro/tags/release-X.Y.Z-rcR -m "Avro X.Y.Z-rcR release."

Building
  1. Build the release & run unit tests. 
     No Format
    
./build.sh clean dist test

  2. Check that release files look ok - e.g. unpack the sources and run tests.
  3. Sign the release (see Step-By-Step Guide to Mirroring Releases for more information). 
     No Format
    
./build.sh sign

  4. Copy release files to a public place. 
     No Format
    
ssh people.apache.org mkdir public_html/avro-X.Y.Z-RC0
scp -pr dist/* people.apache.org:public_html/avro-X.Y.Z-RC0

  5. Call a release vote on avro-dev at hadoop.apache.org.
Publishing
Once three PMC members have voted for a release, it may be published.
  1. Tag the release: 
     No Format
    
svn move https://svn.apache.org/repos/asf/avro/tags/release-X.Y.Z-rcR \
https://svn.apache.org/repos/asf/avro/tags/release-X.Y.Z -m "Avro X.Y.Z release."

  2. Copy release files to the distribution directory and make them writable by the hadoop group. 
     No Format
    
ssh people.apache.org
cp -pr public_html/avro-X.Y.Z-candidate-0 /www/www.apache.org/dist/avro/avro-X.Y.Z
cd /www/www.apache.org/dist/avro
chgrp -R hadoop avro-X.Y.Z
chmod -R g+w avro-X.Y.Z

  3. The release directory usually contains just two releases, the most recent from two branches, with a link named 'stable' to the most recent recommended version. 
     No Format
    
ssh people.apache.org
cd /www/www.apache.org/dist/avro
rm -rf avro-A.B.C; rm stable
ln -s hadoop-A.B.D stable

  4. Publish Java artifacts to the Maven repository: 
     No Format
    
ssh people.apache.org
mkdir /www/people.apache.org/repo/m2-ibiblio-rsync-repository/org/apache/avro/X.Y.Z
cd /www/www.apache.org/dist/avro/avro-X.Y.Z/java
cp -p avro-X.Y.Z{,-javadoc,-sources}.* /www/people.apache.org/repo/m2-ibiblio-rsync-repository/org/apache/avro/X.Y.Z

  5. Wait 24 hours for release to propagate to mirrors.
  6. Prepare to edit the website. 
     No Format
    
svn co https://svn.apache.org/repos/asf/avro/site

  7. Update the documentation links in author/content/xdocs/site.xml.
  8. Update the release news in author/content/xdocs/releases.xml.
  9. Regenerate the site, review it, then commit it. 
     No Format
    
ant -Dforrest.home=/usr/local/forrest -Djava5.home=/usr/local/jdk1.5
firefox publish/index.html
svn commit -m "Updated site for release X.Y.Z."

  10. Deploy your site changes. 
     No Format
    
ssh people.apache.org
cd /www/hadoop.apache.org/avro
svn up

  11. Copy the new release docs to people.apache.org:/www/hadoop.apache.org/avro/docs/rX.Y.Z and update the docs/current link, by doing the following in your home directory on people.apache.org: 
     No Format
    
ssh people.apache.org
tar xzf /www/www.apache.org/dist/avro/avro-X.Y.Z/avro-doc-X.Y.Z.tar.gz
chmod -R 775 avro-doc-X.Y.Z
chgrp -R hadoop avro-doc-X.Y.Z
cp -rp avro-doc-X.Y.Z /www/hadoop.apache.org/avro/docs/X.Y.Z
rm -r avro-doc-X.Y.Z
cd /www/hadoop.apache.org/avro/docs/
ln -s X.Y.Z current

  12. Send announcements to the user and developer lists once the site changes are visible.
  13. Update the version number in share/VERSION.txt to be ''avro-X.Y.N-SNAPSHOT'', where ''N'' is one greater than the release just made.
  14. In Jira, ensure that only issues in the "Fixed" state have a "Fix Version" set to release X.Y.Z.
  15. In Jira, "release" the version. Visit the "Administer Project" page, then the "Manage versions" page. You need to have the "Admin" role in Avro's Jira for this step and the next.
  16. In Jira, close issues resolved in the release. Disable mail notifications for this bulk change.
Using AWS for Avro Build and Release
I've chosen the instance ID of ami-2d4aa444, which is the 32-bit Lucid Lynx with instance storage in us-east-1 (see http://uec-images.ubuntu.com/releases/lucid/releaseImage Removed for other options). Be sure to open port 80 in the security group under which you start the instance, as Javadoc needs to fetch remote documentation. Once you've started and ssh'ed into the instance, execute the below bash commands. Note also that you'll need to navigate Sun/Oracle's sweet user interface for installation. I've found it easiest to use tab to move to the "OK" box and space to select "OK".
To build the documentation for the distribution, you need to have Java 5 for use with Apache Forrest (really the worst thing ever). Instructions for the installation are at http://java.sun.com/j2se/1.5.0/install-linux.htmlImage Removed, and the binary installation file is available for download at http://java.sun.com/javase/downloads/5u22/jdkImage Removed. You'll also need to download Forrest, currently available as a tarball at http://apache.siamwebhosting.com/forrest/apache-forrest-0.8.tar.gzImage Removed, and set the FORREST_HOME environment variable to point to where you unpack the tarball. Lastly, you'll need to pass -Djava5.home=... with the path to where you installed Java 5 and -Dforrest.home=... to ant.
 Code Block

# If you want to check out code and edit it with emacs
$ sudo apt-get install emacs23-lucid subversion

# C dependencies
$ sudo apt-get install gcc automake libtool asciidoc

# C++ dependencies
$ sudo apt-get install g++ libboost-all-dev source-highlight flex bison doxygen

# Ruby dependencies
$ sudo apt-get install ruby rake rubygems libopenssl-ruby
$ sudo gem install echoe

# Python dependencies
$ sudo apt-get install python-pip
$ sudo pip install setuptools

# Java dependencies
$ sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
$ sudo apt-get update
$ sudo apt-get install ant sun-java6-jdk

See Also
    1. +1).0-SNAPSHOT
    2. Update the version number in pom-XXXX.xml used for unit tests in the lang/java/maven-plugin/src/test/resources directory too.
  2. Update the master branch
     No Format
    git commit -am "Preparing for X.(Y+1).0 development"
git push apache master


Updating Release Branch
These operations take place in the release branch.
  1. Check out the branch with:
     No Format
    git checkout branch-X.Y

  2. Update the version:
     No Format
    mvn versions:set -DnewVersion=X.Y.Z -DgenerateBackupPoms=false

    1. Update the version number in share/VERSION.txt to be X.Y.Z. Be sure NOT to leave a trailing newline.
    2. Update the version number in lang/js/package.json to X.Y.Z.  Update or regenerate lang/js/package-lock.json.
    3. Update the version number in pom-XXXX.xml used for unit tests in the lang/java/maven-plugin/src/test/resources directory too.
    4. Update the versions in doc/examples/java-example and doc/examples/mr-example.
  3. Commit the update to the release branch.
     No Format
    git commit -am "Preparing for release X.Y.Z"

  4. Add the fix version X.Y.Z to the Avro JIRA
  5. If not already done, cherry-pick desired patches from master into the branch and commit these changes.
     No Format
    git checkout branch-X.Y
git cherry-pick <commit>

  6. For each patch merged, change the fix version for the JIRA issue to be X.Y.Z
  7. Go through JIRA, and git log to be sure that the issues included in the branch match in each location.
  8. Update the version number in lang/c/version.sh (the variables libavro_micro_version, libavro_interface_age and libavro_binary_age) according to the libtool versioning rules as described in that file. Note the libtool version number is completely unrelated to the Avro release version number.
  9. Commit these changes.
     No Format
    git commit -am "Preparing to build X.Y.Z"

  10. Tag the release candidate (R is the release candidate number):
     No Format
    git tag -s release-X.Y.Z-rcR -m "Avro X.Y.Z rcR release."

  11. Push the release tag and branch changes:
     No Format
    git push apache release-X.Y.Z-rcR
git push apache branch-X.Y

Building
Unless you have set up the required dependencies to build Avro for all languages, the following should be run inside a Docker container (see BUILD.md for instructions).
  1. Build the release & run unit tests. It is strongly recommended to do this inside the docker container.

     No Format
    # To launch a bash shell inside the docker container.
./build.sh docker

# Once inside the container.
./build.sh clean test dist

  2. Check that release files look ok - e.g. unpack the sources and run tests.
  3. Sign the release (see Step-By-Step Guide to Mirroring Releases for more information).
     No Format
    ./build.sh sign
# Or if using a non-default key (AVRO-3858):
GPG_LOCAL_USER=B0CCBDD6 ./build.sh sign
    To sign a release, your key must be present in the dist/KEYS file. See the Apache guide to Signing Releases for more details. Also, if you've updated the dist/KEYS file, be sure to update the public Apache KEYS file as well:
     Code Block
    svn co --depth=files https://dist.apache.org/repos/dist/release/avro/ archive-avro/
cd archive-avro/

head -n 14 KEYS
This file contains the PGP keys of various developers.

Users: pgp < KEYS
or
       gpg --import KEYS


Developers: 
    pgp -kxa <your name> and append it to this file.
or
    (pgpk -ll <your name> && pgpk -xa <your name>) >> this file.
or
    (gpg --list-sigs <your name>
    && gpg --armor --export <your name>) >> this file.
    As you can see, the header of the KEYS file has instructions to add your key. Then commit using: svn commit -m "Add key for ..."
  4. Copy release files to the public staging area https://dist.apache.org/repos/dist/dev/avro/
     No Format
    svn co https://dist.apache.org/repos/dist/dev/avro/ avro-dev-dist
mkdir avro-dev-dist/avro-X.Y.Z-rcR
cp -pr avro/dist/* avro-dev-dist/avro-X.Y.Z-rcR
cd avro-dev-dist
svn add avro-X.Y.Z-rcR
svn commit -m "Artifacts for Avro X.Y.Z RCR"

  5. Stage the default Hadoop 2 version of Java artifacts to the Maven repository:
     No Format
    mvn clean -P dist,sign deploy -DskipTests=true -Davro.version=X.Y.Z
    Make sure that you've followed to guide to configure Maven: http://www.apache.org/dev/publishing-maven-artifacts.html
  6. Find the Staging Repository, and close it.
  7. Call a release vote on dev at avro.apache.org. Include the URL of the staging repository.
Publishing
Once three PMC members have voted for a release, it may be published. Please prepare in advance your account and request correct permissions to be able to publish in the different services: PyPi, RubyGems, NuGet, npm and CPAN
  1. Tag the release:
     No Format
    git checkout release-X.Y.Z-rcR
git tag -s release-X.Y.Z -m "Avro X.Y.Z release."
git push apache release-X.Y.Z


  2. Copy release files to the release repository (PMC permissions required):
     No Format
    svn copy https://dist.apache.org/repos/dist/dev/avro/avro-X.Y.Z-rcR \
https://dist.apache.org/repos/dist/release/avro/avro-X.Y.Z -m "Avro X.Y.Z release."


  3. The release directory usually contains just two releases, the most recent from two branches, with a link named 'stable' to the most recent recommended version (PMC permissions required):
     No Format
    svn co https://dist.apache.org/repos/dist/release/avro/ avro-release-dist
cd avro-release-dist
svn rm avro-A.B.C; rm stable
ln -s avro-X.Y.Z stable
svn commit -m  "Avro X.Y.Z release."


  4. Publish Java artifacts to the Maven repository:
    Find the Staging Repository and release it.
  5. Publish Python artifacts to PyPI. To do this you'll need an account on PyPi, and write access to the Avro package - ask the existing owners for permission if you don't have it. Also see the build and twine documentation.
     No Format
    # Install build and twine in the docker image
export PATH=~/.local/bin/:$PATH
python3 -m pip install --user --upgrade build pip twine

mkdir -p tmp/py
cd tmp/py
tar xvfz ../../dist/py/avro-X.Y.X.tar.gz
cd avro-X.Y.Z
python3 -m build  # build a sdist, and then the wheel from the sdist.
twine upload dist/*


  6. Publish Ruby artifacts to RubyGems. Again, you'll need an account and you need to be an owner.
     No Format
    gem push dist/ruby/avro-X.Y.Z.gem


  7. Publish JavaScript artifacts to npm. Again, you'll need an account and you need to be an owner.
     No Format
    npm login
npm publish dist/js/avro-js-X.Y.Z.tgz

  8. Publish C# artifacts to NuGet. Again, you'll need an account and you need to be an owner. You also need to generate an additional key to publish.
     No Format
    mkdir -p tmp/csharp
cd tmp/csharp
tar xvfz ../../dist/csharp/avro-csharp-X.Y.Z.tar.gz
dotnet nuget push main/Apache.Avro.X.Y.Z.nupkg -k YOUR_KEY -s https://api.nuget.org/v3/index.json
dotnet nuget push codegen/Apache.Avro.Tools.X.Y.Z.nupkg -k YOUR_KEY -s https://api.nuget.org/v3/index.json
dotnet nuget push codec/Avro.File.Snappy/Apache.Avro.File.Snappy.X.Y.Z.nupkg -k YOUR_KEY -s https://api.nuget.org/v3/index.json
dotnet nuget push codec/Avro.File.BZip2/Apache.Avro.File.BZip2.X.Y.Z.nupkg -k YOUR_KEY -s https://api.nuget.org/v3/index.json
dotnet nuget push codec/Avro.File.XZ/Apache.Avro.File.XZ.X.Y.Z.nupkg -k YOUR_KEY -s https://api.nuget.org/v3/index.json
dotnet nuget push codec/Avro.File.Zstandard/Apache.Avro.File.Zstandard.X.Y.Z.nupkg -k YOUR_KEY -s https://api.nuget.org/v3/index.json

  9. Publish Perl artifacts to CPAN. Again, you'll need an account and you need to be an owner.
     No Format
    cpan-upload -u YOUR_ID Avro-X.Y.Z.tar.gz

  10. Wait 24 hours for release to propagate to mirrors.
  11. Prepare to edit the website. THIS SECTION IS OUT OF DATE.  TODO!
     No Format
    svn co https://svn.apache.org/repos/asf/avro/site


  12. Update the documentation links in author/content/xdocs/site.xml.
  13. Update the release news in author/content/xdocs/releases.xml.
  14. Run Apache Forrest using Apache Ant. Example is with forrest installed using homebrew: http://macappstore.org/apache-forrest/
     No Format
    cd site
ant -Dforrest.home=/usr/local/Cellar/apache-forrest/0.9/

# Alternatively, using forrest inside the avro-build docker.
ant -Dforrest.home=/usr/local/apache-forrest

  15. Copy the new release docs, which are generated at the dist step, to website and update the docs/current link:
     No Format
    tar xzf dist/avro-doc-X.Y.Z.tar.gz
mv avro-doc-X.Y.Z ../site/publish/docs/X.Y.Z
cd ../site/publish/docs
svn add X.Y.Z
rm current
ln -s X.Y.Z current
svn commit -m "Adding documentation for release X.Y.Z."


  16. Send an announcement email to the the mailing lists: announce@apache.org, user@avro.apache.org. and dev@avro.apache.org once the site changes are visible. Use your personal `@apache.org` email address or it will be refused.
  17. In Jira, "release" the version. Visit the Manage versions page. You need to have the "Admin" role in Avro's Jira for this step and the next.
  18. In Jira, close issues resolved in the release. Disable mail notifications for this bulk change.
  19. Update the release information in Apache Committee Report Helper (you should receive a reminder by email).
  20. Create a new release on Github:
    1. Draft a new release, using the release-X.Y.Z tag created in step 1.
    2. Release title: Apache Avro X.Y.Z
Post-release steps:
Prepare next release cycle
  1. Update the version where ''N'' is one greater than the release just made:
     No Format
    mvn versions:set -DnewVersion=X.Y.N-SNAPSHOT -DgenerateBackupPoms=false

    1. Update the version number in share/VERSION.txt to be ''X.Y.N-SNAPSHOT'. Be sure NOT to leave a trailing newline.
    2. Update the version number in lang/js/package.json to X.Y.N-SNAPSHOT
    3. Update the version number in pom-XXXX.xml used for unit tests in the lang/java/maven-plugin/src/test/resources directory too.
Update to the release branch.
 No Format
git commit -am "Preparing for release X.Y.N"
git push apache branch-X.Y
Update Documentation
  1. Update the Apache Avro wikipedia page
  2. Update this guide with the changes or issues you found while releasing the version. Extra points if you can reduce or automate some steps!
Validate published artifacts
You can check the published version of the artifacts is correctly available in the URL or if you want to double check that the artifacts can be downloaded/used you can use the suggested CLI
  1. Java - https://search.maven.org/artifact/org.apache.avro/avro
     No Format
    docker run -it maven:3 mvn dependency:get -Dartifact=org.apache.avro:avro:X.Y.Z

  2. Python 2 / 3 - https://pypi.org/project/avro / https://pypi.org/project/avro-python3
     No Format
    docker run -it python:2 pip install 'avro==X.Y.Z'
docker run -it python:3 pip install 'avro==X.Y.Z'

  3. Ruby - https://rubygems.org/gems/avro
     No Format
    docker run -it ruby:2.3 gem install avro:X.Y.Z

  4. Javascript - https://www.npmjs.com/package/avro-js
     No Format
    docker run -it node:10 npm install avro-js@X.Y.Z

  5. C# - https://www.nuget.org/packages/Apache.Avro / https://www.nuget.org/packages/Apache.Avro.Tools
     No Format
    docker run -it mcr.microsoft.com/dotnet/core/sdk:3.1
$ dotnet new console -o test-project && cd test-project && dotnet add package Apache.Avro --version X.Y.Z
$ dotnet tool install --global Apache.Avro.Tools --version X.Y.Z


  6. Perl - https://metacpan.org/release/Avro
     No Format
    docker run -it perl:5 cpan Avro


See Also