Page tree
Skip to end of metadata
Go to start of metadata



The release lead role in Apache MXNet means you are responsible for a few different things:

  1. Preparing for release candidates:
    1. informing the community of timing
    2. prepare release notes (bug fixes, features added)
    3. making code changes with necessary version updates
    4. cutting a release candidate
  2. Running the voting process for a release:
    1. calling votes and triaging issues, revote if necessary
  3. Finalizing and posting a release:
    1. marking & publishing the official release
    2. updating the MXNet website
    3. announcing the release


Pre-requisites to become a Release Lead: (based on feedback from Apache MXNet Mentors)

  1. You are a current project committer
  2. You’ve gained access to configure build jobs at as a part of the jobsadmin group (ask a PMC mentor to add you)


1. Preparing the Release Candidate:

Step 1.1. Prepare Release Notes

Draft release notes which should include the list of new features and bug fixes, along with limitations, known issues, etc. Seek help from committers on @dev to achieve good coverage of all important fixes/API changes in the release. 

  • All problems/fixes in release notes MUST have the PR link and a possible workaround if any
  • New features must be added to the readme/tutorial
Use this script to get commits since a particular date
# to get a list of commits since <date>
git log --pretty=format:"%h - %cn, %cd : %s" --no-merges --since="2017-09-01" > commits.txt
# To get a list of committers since <date>
git shortlog -sne --since="10 Aug 2017"

Note: Check if this release has to go out on a blog on Medium (Sukwon) beforehand. If so, keep time to work on the blog post as this may take a while longer. This blog post needs to go through multiple reviews. 

Step 1.2. Inform the Community of timing

Recommendation: While we have added email content in this wiki, it would be best to refer to emails that were sent for the most recent release. You can find them in the apache archive (or your inbox!):

After every release, the community should decide on a list of features to deliver for the following release. When the clear target date for a release has been estimated, the release lead should inform the community of the release schedule. All features which the community wishes to include in the release should be merged and tested successfully before the date of the first release candidate. The first release candidate should be on a commit which passed builds from at least 3 days before cutting the release candidate. This will make the release lead's job easier. Below is a template for the information:

Community announcement email template
Subject: Release plan - MXNET $release
I am starting the process to prepare for MXNET $release release. I have drafted release notes ($link) to cover the tasks under this release.
A release candidate will be cut on $utc_date and voting will commence $utc_date_vote0 - $utc_date_vote1. If you have any additional features in progress and would like to include it in this release, please assure they have been merged by $utc_date-72 comment so I may update the release notes.
Feel free to add any other comments/suggestions.

Step 1.3. Make sure all files have correct License headers and the top level LICENSE file has been updated

  • Any new files added in this release must have a valid license header. 
  • Make sure any new submodules or dependencies that have been added in this release have been mentioned in the top level LICENSE file - There is no test to cover this (and is a frequent cause of a -1)
  • Make sure the Apache RAT test is passing (contact mbaijal@ in case of issues). Run manually on the final commit as mentioned below (will be part of a nightly test soon)
    Link -
  • DISCLAIMER is correct, filenames include “incubating”
  • More Details: The top level LICENSE and NOTICE files are correct and dependency licenses are acceptable
    • LICENSE and NOTICE files at the root of the artifact directory must only reflect the contents of the artifact in which they are contained.
    • Any new dependency with a non-ASF license must be added to the top level LICENSE file as directed in the links below.
    • All source files have license headers where appropriate.
    • The provenance of all source files is clear (ASF or software grants)
  • Some More details: Refer to the following Links 


Step 1.4. Make code changes with necessary version updates

At least 3 days prior to cutting the first release candidate, bump up the version number and address other changes that the release depends on.

    1. Bump up the version number. An example can be followed here:, update
      More recent example -
      make sure no broken/non-existing links are added to the master branch.
    2. Check with the website lead and docs lead that all necessary updates have made it into the release branch
      1. Most updates happen post-release.
      2. Broken link updates in tutorials and faqs would be an exception.
    3. Create a RSA GPG key of length 4096, upload it to the public server, and add it to the KEYS file (do this process once for each release lead): & See more detailed instructions on creating the key here: Instructions for updating the KEYS file can be found here: Note that you must use apache email instead of personal email for the key. 

      Adding Keys
      # checkout the apache mxnet repo
      svn co apache-mxnet
      cd apache-mxnet
      # update the KEYS file ...
      # commit the update
      svn commit -m “update keys file for xxx ” --username your_username --password your_passwd

Step 1.5. Cut the Release Branch

Note: From version 1.3 all branches are created with a name v1.3.x to make patch releases easier. For patch releases cutting a new branch is not necessary. Fixes for patch releases should be merged to v#.#.x branch and latest commit should be tagged with an exact version number, ie v1.3.1.

    1. The release candidate should contain all planned features, bug fixes, and code changes above. The release candidate commit should have passed the merge build, pip builds

    2. Docs should also be manually built locally & checked for correctness. TODO: how to retrieve the commit hash.

    3. Checkout the commit into a new branch "v#.#.x" and push it (It might be a good idea to do this in advance and freeze the branch)

    4. Update NEWS and README. An example can be followed here: Note that this assumes the existence of the tag of next release, which is not created yet. Therefore, the update only happens on the release branch (not the master branch) so broken links are not visible to users.

    5. Manual Docs Build - 

        1. The must also be updated with the correct tag
        2. This script must be built locally to test docs (confirm procedure with Santhosh, Yao) and automate after 0.12.0.

Step 1.6 Test the Release Branch

Once all the PRs have been merged the final commit hash on the release branch must be tested. (You should begin testing as soon as the release branch is created, but make sure it is run atleast once on the final commit)

  1. Nightly Tests
    Following jobs are run as a part of the nightly tests:
    1. NightlyTests
    2. NightlyTestsForBinaries
      These jobs should already contain needed branches if they are cut. If not, check the jobs configuration. These are triggered by DailyTrigger. Either wait until the jobs have been triggered or trigger them manually.
    3. Backward Compatibility checker (currently restricted).
    4. Some jobs might take up to 4-5 hours, so plan accordingly.
    5. In case of a failure of any job, check the console logs, the newly added PRs and try and resolve issues or contact community for a fix (on @dev list, for example). 
  2. Jenkins CI Tests
    1. The release branch must pass the tests on the CI. For example, here -
  3. RAT Check - make sure the RAT check passes (it's already part of the verification pipeline as a stage, no special actions needed)

Step 1.7. Tag release candidate (To be done by a committer)

    1. Go to the GitHub repo’s “releases” tab
    2. Click “Draft a new release”
        • Provide the release tag in the form of “1.0.0.rc0” where 0 means it’s the first release candidate
        • Select the commit by clicking Target: branch > Recent commits > $commit_hash 
        • Copy and paste change into the description box from here
        • Select “This is a pre-release"

Step 1.8. This step has not been done in the past (question)

Tag all the dependent submodules for every MXNet release. If a code-change (e.g. bug-fix) is required to a dependent sub-module then we take a branch from the tag and apply the code-change only to the branch for the sub-module so that the change is minimal. This should allow the MXNet release process to have better convergence towards a stable release.

Step 1.9. Create artifacts for the release and push to the dist folder

a) The src tar - apache-mxnet-src-#.#.#.rc0-incubating.tar.gx:

      • Clone the Repo and checkout the release branch 
      • remove R-package until the licensing issue is resolved
      • Remove all the .git files
      • Remove all other files of the form .* (eg .travis, .DS_Store etc)
      • on MacOS use gnu-tar
      • Try to untar it in a different type of machine and try a classic build with `make`. Follow build instructions in the "Build from Source" section
Creating the src tar
git clone apache-mxnet-src-0.12.0.rc0-incubating
git checkout v0.12.0
git submodule update --init --recursive
git checkout 0.12.0.rc0
rm -rf R-package/*
rm -rf R-package
rm -rf .DS_Store
find . -name ".git*" -print0 | xargs -0 rm -rf
brew install gnu-tar 
gtar -czvf apache-mxnet-src-0.12.0.rc0-incubating.tar.gz apache-mxnet-src-0.12.0.rc0-incubating


b) apache-mxnet-src-#.#.#.rc0-incubating.tar.gz.asc: 

details here:

d) Create SHA checksum: apache-mxnet-src-#.#.#.rc0-incubating.tar.gz.sha512

gpg --armor --output apache-mxnet-src-0.11.0-incubating.tar.gz.asc --detach-sig apache-mxnet-src-0.11.0-incubating.tar.gz
shasum -a 512 apache-mxnet-src-0.11.0-incubating.tar.gz > apache-mxnet-src-0.11.0-incubating.tar.gz.sha512


e) Create a folder for this RC in the dist folder: 0.12.0-incubating.RC0 folder 

Uploading to dist
# Checkout the Apache directory in Apache distribution SVN "dev" repo
$ svn co
# select p
# Make directory for this RC in the above directory
$ cd mxnet
$ mkdir #.#.#.rc0
# Move artifacts into this folder
$ cp ../apache-mxnet-src-#.#.#.rc0-incubating.tar.gz* #.#.#.rc0
$ svn add #.#.#.rc0
$ svn commit -m "Add mxnet-#.#.#.rc0" --username "lxn2"

Step 1.10. Validate release package (Committer or Contributor)

As per the Apache documentation, verify that the release candidate artifacts satisfy the following:

    • PGP signatures and SHA256 checksum verification - 
svn co && cd mxnet/#.#.#.rc0
brew install gpg coreutils
gpg --import ../KEYS
validate the signatures
gpg --verify ./apache-mxnet-src-0.11.0.rc3-incubating.tar.gz.asc
gsha512sum --check  ./apache-mxnet-src-0.11.0.rc1-incubating.tar.gz.sha512 

Step 1.11. Test the final release package (Committer or Contributor)

    • Download the package that was just uploaded to svn as above. 
    • Test that a build is successful - spinning up a deep learning Ubuntu AMI will give you an environment with all dependencies to build. For example:

      ci/ -p ubuntu_gpu
    • if Possible, run a simple (mnist) test to verify the src.

Step 1.12. Upload the release artifacts to Github release tag

    • Edit the release tag from Step 1.7, and upload the release artifacts from Step 1.9 (which were verified in 1.10-1.11).

After creating RC, Give 2 days to build scala/ R packages if needed before starting vote on dev@.

Step 1.13. Build Scala Packages 

Follow the release process to build Scala packages and include the stage repo links in the voting email.

MXNet-Scala Release Process

2. Running the voting process for a release:

The full voting process consists of two phases: (1) vote on (3 days) (2) vote on (3 days)

Step 2.1. Vote on dev@ 

For the dev@ vote, there must be at least 3 binding +1 votes and more +1 votes than -1 votes. Once that quorum has been reached, the vote then moves onto the general@ vote.

To find out if the person is a committer/IPMC member, you can check apache phonebook:

Use the email template below to start the vote:


Subject: [VOTE] Release MXNet version 0.11.0.RC#
This is the vote to release Apache MXNet (incubating) version 0.11.0. Voting will start now and close $utc_now+72.
Link to release notes: <Link>
Link to release candidate 0.11.0.rc0: <Link> 
Link to source and signatures on apache dist server: <Link>
Link to scala packages on the staging repo: <Link>


Please remember to TEST first before voting accordingly:
+1 = approve
+0 = no opinion
-1 = disapprove (provide reason)


Step 2.2. Send out the Vote Results on dev@


Subject: [RESULTS] [VOTE] Release MXNet version 0.11.0.RC#
This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes.
+1 votes
PMC Members:
* $Name
* $Name 
* $Name
* $Name
* $Name
* $Name
* $Name
0 votes
* No votes
-1 votes
* No votes
Vote thread:
I'll continue with the release process and the release announcement will follow in the next few days.

Step2.3. Triage issues and revote if necessary 

Should any vote fail to reach quorum or release lead determines that another RC needs to go out, the release lead should triage the issues, create GitHub issues, and move to fix the issues. Once fixed, make changes to NEWS & in case it specifically lists the old RC# number. Start the process for another release candidate. In the new voting email, detail what has changed since the last release candidate. Repeat until the vote passes.



Subject: [CANCELLED] [VOTE] Release MXNet version 0.11.0.RC#
    We are CANCELLING this vote because...
    We will address this issue and send out RC#+1 for another vote.


Step 2.4. Start a vote on general@

For the vote on general@, anyone can contribute a vote, but only "Incubator PMC" (IPMC) votes are binding.
To pass, there must be 3 binding +1 votes and more +1 votes than -1 votes.

In ASF, votes are open "at least" 72hrs (3 days). If you don't get enough number of binding votes within that time, you cannot close the voting deadline. You need to extend it.

Send out a notice to vote with the template below.
To get the fingerprint of your keys, run `gpg2 --list-keys --fingerprint`


This is a call for a releasing Apache MXNet (incubating) #.#.#, release
candidate #.

Apache MXNet (incubating) community has voted and approved the release.

Vote thread:

Result thread:

The source tarball, including signatures, digests, etc. can be found at:

The tag to be voted upon is v0.2.0-incubating-rc3:

The release hash is #########:

Release artifacts are signed with the following key:

KEYS file available:

For information about the contents of this release, see:
<link to release notes>

The vote will be open for 72 hours.

[ ] +1 Release this package as 0.1.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Step 2.5: Send out the Vote Results on general@

Send out an email on general@ in the same voting thread which you sent in previous step to summarize the result.

3. Finalizing and posting a release:

3.1. Marking & Publishing the Official release

Once the release candidate passes the vote, we draft the official release on GitHub and rename the release candidate as the official release.

Step 3.1.1: Draft official release on GitHub:

    • Go to the GitHub repo’s “releases” tab
    • Click “Draft a new release”
    • Provide the release tag in the form of “0.12.0”
    • Select the commit by clicking Target: master > the passing release candidate tag
    • Copy and paste NEWS change into the description box

Step 3.1.2: Rename and publish the source package:

First upload to dev and then move to releases as follows 

#create a directory under releases -

This process needs to change ->

Rename and Resign the RC to create GA
# Step 1: Download and Untar
tar -zxvf apache-mxnet-src-0.11.0.rc3-incubating.tar.gz

# Step 2: rename (remove the rc) and tar again
mv apache-mxnet-src-0.11.0.rc3-incubating apache-mxnet-src-0.11.0-incubating
gtar -czvf apache-mxnet-src-0.11.0-incubating.tar.gz apache-mxnet-src-0.11.0-incubating

# Step 3: Resign now
# Step 4: First upload here - 
svn mv
# Step 5: Move from dev to release
svn mv --message "0.11.0 release moving from dev to release repo"

# Wait 24hours for the src mirroring. The final download page for the announce email is somewhere here (example 0.11.0) -

Step 3.1.3: Download the nightly pip build, rename, and publish:(by Sheng)

Step 3.1.4: Build, test, and publish the Docker images: (by Meghna)

This was done as follows in 0.11.0. In the future use the automated pipeline - 

        • Update the tag in the file or select the correct tag in the Jenkins Job
        • Select the correct Repo name  = mxnet
        • Trigger the Job with IS_PUBLISH=false
        • Login to the slave and test each image for correct version by checking logs and/or Readme. 
        • Manually upload the 20 images that are built. 
        • Make sure to update the latest and previous docker images

Step 3.1.5: Verify the resources are uploaded

    • Check that the resources are present in It may take a while for them to be visible. This will be mirrored throughout the Apache network. There are a few remaining steps.  

Step 3.1.6: Test

Test multinode and multi gpu (kvstore=1)

3.2. Update the MXNet website

Now that a new version has been drafted on GitHub with source and pip packages published, we need to update the MXNet website.

    1. Add the new installation version on the install page: 
      1. Set the new version to be default. See this example: docs/_static/js/options.js.
      2. Add the option and all of the new selector classes. See this example: docs/install/
    2. Add news about the new release. See this example: docs/_static/mxnet-theme/index.html.
    3. Add download links to the install archives. See this example: docs/install/ Make sure to use the Apache distro servers. 
      1. Current:
      2. Legacy:
    4. Update the pip package table:
    5. Go to find job “restricted-website-build”. Login if you're not already.
    6. Click Configure and look for the section with the String Parameters.
    7. Tags to build
      1. If this is a minor release, the branch should already be in the first parameter "tags_to_build".
      2. If this is a major release, you will need to add the branch to the list in the Default Value field. Note these are semicolon separated values.
    8. Tag list
      1. If this is a minor release, you will want to bump the version up for the branch you're updating. Example: change 1.2.0 to 1.2.1.
      2. If this is a major release, you will need to add the new version number to the list. Example: add 1.3.0. Note these are semicolon separated values.
    9. Click Save.
    10. Click “Build with Parameters”, verify that the settings you just updated look correct, then click Build. 
    11. The docs will be published to There can be a delay due to some edge caching with Apache infra. The restricted-website-build job also runs four times a day.

Not Sure (question) Manually update the website’s download links:

    • TODO: link to Apache mirror source, pgp, sha 

3.3. Update & on the MXNet master branch ( can happen after the announcement). 

3.4. Review the announcement email content on before sending the announcement.

3.5. Check if this release has to go out on a blog on  (Sukwon)

    • In order to get write access to, one need to first register on and ask Henri ( to grant the permission.

3.6. Announce the release

Once everything is working (website docs, website changes, 24 hours after upload) create an announcement on the website and then send an e-mail to the mailing list.

Follow instructions from here:

Note: There must be a link in the announcement to the downloads page, not to the dyn/closer page. Downloads must be mirrored from the official Apache distribution site, not from github or other site. e.g. "A Link to the"

Send an email to, and with the subject: [ANNOUNCE] Apache MXNet <release number> Release and a body along the lines of:




subject: [ANNOUNCE] Apache MXNet 0.11.0 Release


The Apache MXNet community is happy to announce Apache MXNet version 0.11.0! We hit some major milestones with this release!

This is our first official release as an incubating Apache project. The project has now fully migrated its codebase and website to Apache. 

This release includes code contributions from developers from Apple, Samsung, Microsoft and many other.

We have also crossed over 400 contributors on the project so far. The 0.11.0 release features Apple Core ML model converter, Support for Keras v1.2.2.


ADD A DESCRIPTION! - A Flexible and Efficient Library for Deep Learning


A blog that explains an end to end use case of building an ios app using MXNet and Core ML:


The AWS blog that highlights the key features of the release:


A full list of the changes in this release can be found in the release notes:


Link to Download:


To build this project, view this page and select “Build from Source”:


The Docker Images can be found here:


The Pip package can be found here:


The Release Tag is here:


MXNet Resources

   - Issues:

   - Wiki:

   - Mailing list(s): 


For more information on Apache MXNet, please see: 



Apache MXNet(incubating) Team




Apache MXNet (incubating) is an effort undergoing incubation at The

Apache Software Foundation (ASF), sponsored by the name of Apache

Incubator PMC. Incubation is required of all newly accepted

projects until a further review indicates that the

infrastructure, communications, and decision making process have

stabilized in a manner consistent with other successful ASF

projects. While incubation status is not necessarily a reflection

of the completeness or stability of the code, it does indicate

that the project has yet to be fully endorsed by the ASF.



3.7. Bump up the versions on the master  

       Once the release is complete, make sure that all PRs that were created during the release process only for the release branch (version updates, README NEWS etc) are now merged into the master branch too. 

  1. Merge the PR you created for the release branch into the master
  2. Add a new PR to update the version to one up/or next release for pip-releases

3.8. Clean up older releases on ASF mirrors  

  1. clean up older releases stored in and 

    Delete old releases from ASF mirrors (assuming new release version is 1.1.0 and the old one is 1.0.0
    # checkout apache release repo
    svn co apache-mxnet-release
    cd apache-mxnet-release
    # remove the previous release folder
    svn rm 1.0.0
    svn commit -m 'commit message'
    # checkout apache dev repo
    svn co apache-mxnet-dev
    cd apache-mxnet-dev
    # remove the previous release candidate folders
    svn rm 1.1.0.rc*
    svn commit -m 'commit message'

Enjoy an adult beverage (or bubble tea) of your choice, and congratulations on making a MXNet release.

The Release Checklist Template

For Every Release you must use the following checklist to ensure that all tasks happen in a timely manner - 

In case you need to update this list, edit the original spreadsheet

Task #TaskOwnerRelative DateAbsolute Date
 PreReqs for Release Start:PM  
1Finalize the Release DatePMT-15 
2High Level List of Features and artifact specsPMT-15 
3Prepare Release Notes - draft 1PMT-15 
4Inform the communityReleaseLeadT-15 
5PR to update the Website into master/RBWebsiteLeadT-14 
6PR to update the Docs into master/RBDocsLeadT-14 
7PR to update the versions into master/RBReleaseLeadT-14 
8Validate License Headers and top Level LICENSE fileReleaseLeadT-14 
9Validate/update submodulesReleaseLeadT-14 
10Stabilize the code for CI to passReleaseLeadT-14 
11Merge all necessary PRs into masterCommunityT-13 
12Code Freeze and Release Start: Cut the Release BranchReleaseLeadT-10 
13Finalize the Release Notes based on PRs that got in - final draftPMT-10 
14PR to update the NEWS and README into the RBReleaseLeadT-9 
15Validate the docs build locally by checking release-specific docs like pip install, etc.DocsLeadT-9 
16Test part 1: Run the unit Tests on the Release BranchReleaseLeadT-8 
17Test part 2: Run the Nightly TestsReleaseLeadT-8 
18Create the Github Tag for the rc0ReleaseLeadT-7 
19Create the src tar and sign, Upload the src tarReleaseLeadT-7 
20Validate the signaturesRelease LeadT-7 
21Clone svn repo and do a manual testRelease LeadT-7 
 Begin Apache VotingReleaseLeadT-7 
22Start the vote on dev@ReleaseLead  
23Send out the results of the vote on dev@ReleaseLeadT-4 
24Revote if necessary (Steps 15-22)ReleaseLeadNot Accounted For 
25Start the vote on general@ReleaseLeadT-4 
26Send out the results of the vote on general@ReleaseLeadT-1 
 End Apache Voting   
27Create the final release tag on githubReleaseLeadT-1 
28Rename, resign and upload the src tar to final dirReleaseLeadT-1 
29Update the website using tagWebsite LeadT-1 
30Release the official pip packagepip LeadT-1 
31Release the official docker imagesDocker LeadT-1 
32After 24 hrs, validate the packages are uploadedRelease LeadT 
33Draft the offical announce email and reviewRelease LeadT-1 
34Send out the email on announce@Release LeadT 
35Update the apache blogRelease LeadT 
36update the aws blogPMT 
37send internal announcementPMT 
38Update the version on masterRelease LeadT 

Notes for reference


   - 3 +1 votes from IPMC members (these are the votes that count but we
   should open up to the whole podling community)
   - For podlings, 2 additional constraints:
   - Release artifacts must include “incubating” in final file name (ex:
   - Release artifacts must include disclaimer in the release artifacts

   - The Incubator PMC expects the source releases to be staged on so that
   they can easily be moved to the release location via svn mv   (
   - After graduating, RC’s go into
   and official releases go into

   - Apache Press Team []
   must review and coordinate releases for branding
   - On website and in release DISCLAIMER file:
   - Apache Podling-Name is an effort undergoing incubation at The Apache
      Software Foundation (ASF), sponsored by the name of Apache TLP sponsor.
      Incubation is required of all newly accepted projects until a further
      review indicates that the infrastructure, communications, and decision
      making process have stabilized in a manner consistent with other
      ASF projects. While incubation status is not necessarily a reflection of
      the completeness or stability of the code, it does indicate that the
      project has yet to be fully endorsed by the ASF.
      - Website should include Apache Incubator logo:

   - Release should include:
      - LICENSE
      - NOTICE - attribution notices

   - A release must contain source package which is cryptographically
   signed by Release lead with detached signature. It must be tested prior
   to voting for release.
   - Release must only contain appropriately licensed code
   - Please ensure you wait >=24 hours after uploading a release before
   making announcements so mirrors catch up
   - Releases of more than 1GB of artifacts require a heads-up to
   Infrastructure in advance.
   - Which directory for what build?

   - Artifacts MUST be accompanied by:
      - apache-mxnet-src-0.10.1-incubating.asc - contains OpenPGP
      compatible ASCII armored detached signature
      - apache-mxnet-src-0.10.1-incubating.sha - SHA checksum (SHOULD)
   - Publish KEYS file in distribution directory root
      - Signing keys MUST be published in KEYS file, SHOULD be available in
      global public keyserver, SHOULD be linked
      into web of trust
      - Keys MUST be RSA & 4096 bits

   - Apache RAT can assist in checking license compliance
   - Eventually we should set up a build system to sign our releases with
   cryptographic signatures. For now we’ll do it manually.

   - Create a signature and sign releases as mentioned above



  • No labels