This page describes process and provides instructions to release manager on how to release Apache Ignite
Following picture demonstrates release stages sequence.
Table of contents
P0. Initializing
Release phases dates are discussed, as release manager during the first discussion. Release Manager should be PMC Member.
Moving scope freeze and code freeze dates maybe not the best option since a lot of contributors synchronize their efforts to make feature completed by particular moment.
Once release manager found, he/she can setup environment.
Prerequisites for release manager
0.1. Write access to Apache Ignite GIT & SVN repository (commiter role) - https://gitbox.apache.org/repos/asf?p=ignite.git
0.2. Access to Team City Release tasks - https://ci.ignite.apache.org/project.html?projectId=ApacheIgniteReleaseJava8 If you don't have necessary rights ask for assigning on dev. list.
0.3.Your GPG key available in https://apache.org/dist/ignite/KEYS - Steps to add:
P1. Scope development and implementation
P1.1 Implementation and Scope Discussion
Difference between this and the following phase: it is not possible to move in-progress tickets to a next release because branches not diverged.
New branch creation moment is not formal, and it can happen just before or simultaneously with scope freeze. The important thing is announcing this moment and reminding contributors to cherry-pick commits to release branch.
P1.2 Implementation and Scope Finalization
Removing issues from the scope based on estimated dates of completion, importance to release and on community discussion. Private discussion between contributors is possible, but it is recommended to discuss features in public to allow all community members to share their arguments and concerns.
The first step to be done to enter scope finalization phase it to create a branch in Ignite code base (origin). This moment be precisely the same as the start of P1.1
1.2.1. Create release branch, e.g ignite-2.8, push it to the ASF repository. (optional) consider creating PR for release, having PR is convenient because GitHub will notify you about commits to the branch.
1.2.2. Run TC tests (on the appropriate branch), use https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_RunAllNightly
1.2.3. Estimate state of TC, check failures history (if any).
1.2.4. Create scheduled build triggers for daily Run All (Nightly) run. Usually, triggers are set up using Apache Ignite TeamCity Bot (triggering of builds there depends on agent availability). Use an intellectual trigger to trigger release branch adaptively, or ask on the dev. list to add.
1.2.5. Create/schedule the nightly release build triggering – Releases Apache Ignite Nightly (may require TC permissions).
End of this phase is scope freeze.
P2. Rampdown
This phase is some time to complete the implementation of features from the initial scope. In parallel with implementation, some less essential fixes and features are moved to the next release.
End of ramp-down is code freeze. Code freeze is based on dates, but the actual time of freeze is determined by announcing.
P3. Stabilization
During the stabilization phase, only blockers are accepted. Including anything to the release should be approved by the release manager for each commit.
Note blockers defined here not as any critical bug, but
- an issue in the product which makes product unstable or non-functioning
- performance drop has proven by a benchmark
- a security issue, or
- a regression of existing features.
Run load tests. Run ducktests as well. This will help you find issues not covered by CI tests, such as runtime dependency issues.
P4. Vote preparation (building release candidate)
Once all changes are applied, it is possible to prepare a release candidate. It is possilbe to build several release candidates without starting a vote. First RCs can have part of changes scheduled for release.
4.0. Prepare Release Notes
4.0.1. Collect Release Notes (this action is manual now). Consider preparing HTML file in addition to text for future step 6.3.6. Check all completed tickets from the release page. Ticket may or may not have release notes, and this depends on if it is reasonable to mention change for users.
4.0.2. Optionally you may ask for a review of resulting release notes on the list.
4.0.3. Update RELEASE_NOTES.txt and commit Release Notes changes into master and a release branch.
4.1 Ensure Documentation Readiness and Accouncement Blog Post Activity
- Check that Ignite technical documentation and APIs are fully ready for the release. Documentation has to be released together and on the same day with the rest of the release artifacts (binaries, Docker images, etc.) once the vote passes.
- Confirm that there is a community member who will be working on an announcement blog post for blogs.apache.org to cover major changes coming in the version. Ideally, the blog post should be published and added into an announcement email the same day we'll be releasing release artifacts (binaries, documentation, etc.). That's an example of an Ignite announcement article.
4.2 Update versions and year for release
4.2.1. Update release branch version in maven poms and assembly
Update version in the release branch (execute the script from Ignite project root directory, you can use WSL or GitBash on Windows):
./scripts/update-versions.sh 2.7.6
and commit changes. This step is essential for release because later the source code will be published into SVN. Version update is required to be committed before RC building.
4.2.2. Update documentation version
In the release branch, open the docs/_config.yml file and update the version property with the release number.
4.2.3. Update year in copyright messages
Double-check that year in copyright messages is correct. If the release is minor and based on an older branch, this branch can contain an outdated value of the year. Fix if needed. Example commit of updating year.
Following step is not actual since 2.8, there is the feature for an auto-generating year: Update copyright year manually in IgniteVersionUtils
4.2.4. Update DEB & RPM package version
Version of packages update is not automated since it requires to add verison to history to update it. Update is more or less similar to this commit
4.2.4.1. Update packaging/deb/changelog, add similar lines to the top of the files (version, comment, mainteiner, date):
apache-ignite (2.7.6-1) unstable; urgency=low * Updated Apache Ignite to version 2.7.6 -- Petr Ivanov <mr.weider@gmail.com> Wed, 20 Aug 2019 20:58:44 +0300
Check dates are correct, and version is postfixed with '-1' - it is package version number, it is required by format
4.2.4.2. Update packaging/rpm/apache-ignite.spec, change current version; insert new line to changes log:
Name: apache-ignite Version: 2.7.6
and add lines similar to following after change log
* Fri 20 Aug 2019 Peter Ivanov <mr.weider@gmail.com> - 2.7.6-1 - Updated Apache Ignite to version 2.7.6
Make sure dates are correct, add package build number, '-1' at the end of package version.
4.2.3.3. (optional) ask for review on the list.
4.2.3.4. Commit package version changes into master and a release branch.
4.3. Prepare RC on Teamcity
4.3.1. Run TC Main Release Build
Run [RELEASES] Apache Ignite / Main [1] Release Build for ignite-x.y or for ignite-x.y.z branch. You will need to enter version number (value if x.y.0 or x.y.z), and specify release candidate number. For testing purposes, it is recommended to use rc0. You may several times re-build same rc- number.
4.3.2. Download and check release build
Download and unzip release archive. It can be found at "Artifacts" tab on build page. Example:
~/download:[]$ unzip release-2.7.0-rc0.zip
Do some checks related to release (full list of check to be done can be found in section 5.1):
- Validate version of Ignite, RC number, and packages versions (from 'packages' folder) are correct.
- Unpack build provided with RC: svn/vote/apache-ignite-{$version}-bin.zip . Run Ignite node from resulting apache-ignite-{$version}-bin/bin folder.
- Validate node starts, correct version, and copyright messages produced.
4.4. Sign and deploy RC using vote preparation scripts
Run vote scripts to prepare RC before voting. You may skip steps 4.4.1 & step 4.4.2 in case you want to run some testing of release. In case release build was made for testing-only purposes, you can go to step 4.4.3 in this section.
4.4.1. Run script to create VCS tag in git
Run script vote_1[git]create_rc_tag.sh Example of script output:
Please, check corresponding git tag is created and is available in ASF repository: https://gitbox.apache.org/repos/asf?p=ignite.git;a=tags
4.4.2. Create & close (publish) staging repository
Note: If you've already uploaded staging, you should remove it from nexus - https://repository.apache.org/#stagingRepositories
Run script ./vote_2\[mvn\]\[pgp\]deploy_to_staging.sh Example of script output:
Go to Nexus UI https://repository.apache.org/#stagingRepositories, log in using Apache credentials and close repository.
Provide some comment for closing repository, e.g. `Repository for Apache Ignite 2.7.6 - RC0`.
Close is a heavy background process, which makes repository visible for others. After some time, check if the repository closed successfully and visible.
Once repository is closed, write down a link to content, e.g. https://repository.apache.org/content/repositories/orgapacheignite-1464/ This link is useful for vote thread and sharing with others, who is going to test RC.
4.4.3. Prepare signed distribution for SVN
4.4.3.1. Sign DEB/RPM packages
Pre-build packages can be found at 'release-2.7.6-rc0\packages' folder of release candidate archive.
Building packages locally supported only on Linux, WSL not supported. Steps to build from scratch (optionally):
Build script will sign artifacts automatically (in case of build locally), but TC pre-build packages need to be signed by release manager key.
Find your public key ID by running
gpg --list-keys
Key Id will look like as 16 hexadecimal, e.g., 612654F7; it is the same value from step 0.3.2.
Sing RPM pre-build package by command
rpm --define "_gpg_name 612654F7" --addsign packages/pkg/*.rpm Enter pass phrase: Pass phrase is good. packages/pkg/apache-ignite-2.7.5-1.noarch.rpm:
Sign DEB pre-build package by command
dpkg-sig -k 612654F7 --sign builder packages/pkg/*.deb Processing packages/pkg/apache-ignite_2.7.5-1_all.deb... You need a passphrase to unlock the secret key for user: "Apache User (CODE SIGNING KEY) <login@apache.org>" 4096-bit RSA key, ID 612654F7, created 2019-03-19 Signed deb packages/pkg/apache-ignite_2.7.5-1_all.deb
You can validate that file modification time was changed after singing.
4.4.3.2. Sign artfacts
Run script vote_3_step_2[pgp]sign_artifacts.sh . Example of script output
Please check results at ./svn/vote. Optionally you may also check hash/sign using https://www.apache.org/info/verification.html
4.4.3.3. Deploy artfacts to SVN
Run script vote_3_step_3[svn]deploy_artifacts.sh. This script will upload your local artifacts to a a development area under e.g. https://dist.apache.org/repos/dist/dev/2.7.0-rc0 which is used for staging releases candidates.
Each ASF committer has a write access to this area – WHERE CAN I STAGE A RELEASE CANDIDATE?
Example of output:
Check binaries and sources are available in the SVN: https://dist.apache.org/repos/dist/dev/ignite/
4.5 Run additional TC steps
4.5.1. Verify file changes using TeamCity
There is TC task for generating report containing difference comparing current release with previous.
https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency
You should run it and do sanity check for a changed files.
To start the build you need to specify current (staging) version and released version of Apache Ignite. See "Artifacts" tab to get task results. Example:
For 2.7 release "apache-ignite-hadoop" removed and some new dependencies introduced for a benchmarks.
4.5.2. Run automated release check step on TC
Run '[3] Apache Ignite Release Vote | Check RC' https://ci.ignite.apache.org/buildConfiguration/ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum
This step will validate hashes, Javadoc, build from sources, licenses. So it it required to run it to simplify/skip some local checks from section 5.1.
4.6 Sending Release For Vote
After the community agrees that the codebase is ready for a release, release manager should send the release for a vote.
The email subject should be "[VOTE] Release Apache Ignite <version> <rc>", e.g. "[VOTE] Release Apache Ignite 2.7.5 RC4".
Vote email should include reference that vote is formal and votes meaning for +1,0,-1.
Here is the sample email:
Examples of Voting threads Vote on 2.7.5; Vote on 2.7.0
Optionally you can create a separate discussion for questions related to RC: Discussion on 2.7.5
P5. Voting on Release and Release Verification
All community members are encouraged to verify release. PMC members have a binding vote, but each vote matters.
5.1 Release Candidate Verification
The following list are steps, which can be done by each member, see also https://www.apache.org/info/verification.html
It is not a complete list, community members can test any module or feature of the product.
Check that all JIRA issues for the specific version are Closed (example JQL: `project = IGNITE AND fixVersion <= 1.7 AND status != closed`).
curl "https://issues.apache.org/jira/rest/api/2/search?jql=project=ignite+AND+status+!=+closed+AND+fixVersion<=1.7&fields=summary" | grep '"total":0,"issues":\[\]'
Check that sha512 checksum is correct.
sha512sum -c *.sha512
Check that signature is correct.
gpg --verify-files *.asc
- Check that version is correct.
Check licenses from the source code.
mvn clean validate -DskipTests=true -P check-licenses
Build the binary releases from the source code.
mvn clean package -DskipTests
Build Ignite.NET binaries and NuGet packages
cd modules\platforms\dotnet build -skipJava
- Do some trivial checks, e.g. start local topology, run several examples from binaries gained at previous step. Unpacked binaries can be found at sources/target/release-package
See also Incubator Release Checklist for more checks.
5.2 Voting process
In case there is -1 vote present (it is not a veto, even if it was casted by PMC), release manager has ultimate rigth to cancel vote immediately to fix related issues.
Usually +1 means that community member agrees that current release is better with previous.
Usually with +1 member shares which checks were done.
Vote should be open for at least 72hours. In case insufficient votes are available vote can be kept open as long as it needed without additional announce. But in this case release manager has option to close the vote as unsuccessfull in case time is out.
Whenever consensus cannot be reached, standard Apache Voting Process will be used to reach a solution.
P6. Release
6.1. Closing Vote
Send an email with subject "[RESULT] <vote subject>" ("[RESULT] [VOTE] Release Apache Ignite <version> (<rc>)") and provide vote results: if it passes or not, provide voting details.
For example, "[RESULT] [VOTE] Release Apache Ignite 2.7.5 (RC1)" and body similar to 2.7.5 vote result, 2.7.0 vote result . See also other example:
In case vote is unsuccessful or closed by release manager, prepare new release candidate with fixes starting from step 4.2
6.2. Run Release scripts
In case vote was sucessfull, run release* scripts from release candidate local home from step 4.2.1.
6.2.1. Move binaries
Run release_1[svn]move_binaries.sh (Note: Only PMC members have write access to the release repository). Example of output
Result should be available at https://dist.apache.org/repos/dist/release/ignite/
Please check results at:
* binaries: https://apache.org/dist/ignite/{version}
There can be some sync lag with moving files and availability at https://apache.org/dist/ignite/
6.2.2. Upload packages to bintray
During running script you will be asked for auth API key to upload. Login to the https://apache.jfrog.io/ui/ with Apache ID credentials and get the API key from the authentication settings.
Run script release_2[bintray]upload_packages.sh, example of output
6.2.3. Deploy (java,c++,.net,scala) API documentation to the website
Run Script release_3[git]deploy_docs_to_site.sh
This script will checkout ignite-website from GIT, copy API documentation, and push changes back.
Check results availability at https://gitbox.apache.org/repos/asf?p=ignite-website.git;a=tree;f=releases;hb=refs/heads/master
6.2.4. Create Release tag from RC tag
Run script release_4[git]create_release_tag.sh Example of output
Please check results at https://gitbox.apache.org/repos/asf?p=ignite.git;a=tags
6.3. Manual release steps
6.3.1. Release staging repository
Release maven staging (https://maven.apache.org/developers/release/maven-project-release-procedure.html).
Go to Nexus UI https://repository.apache.org/#stagingRepositories, login using Apache credentials and release repository for accepted release candidate. You can release it by pressing the Release button (provided that you have successfully closed the staging repository).
This will move the components into the release repository of OSSRH where it will be synced to the Central Repository (see also https://central.sonatype.org/pages/releasing-the-deployment.html#close-and-drop-or-release-your-staging-repository)
The sync into Maven Central central staging from repository.apache.org occurs every 4 hours. There is a separate hourly schedule that runs which pushes from staging to the other central machines, and then updates the indexes.
- If release step for staging was successful you can access artifacts using service repository: https://repository.apache.org/service/local/repositories/releases/content/org/apache/ignite/ignite-core/2.7.5/
- Once repository is synced you can see result in https://repo.maven.apache.org/maven2/org/apache/ignite/ignite-core/
- With some delay artifact will be available on the site https://mvnrepository.com/artifact/org.apache.ignite/ignite-core
6.3.2. Update sources snapshot version in the master branch
Execute the script from Ignite project root directory, this will update version in the master branch :
./scripts/update-versions.sh 2.9.0-SNAPSHOT
and commit changes. This step is not required if minor release was issued,e.g. 2.7.5
6.3.2.1 Update sources snapshot version in the ignite-extensions repository
The dependency of ignite-extensions must be updated according to the new version in the master branch.
See the example below:
IGNITE-13993
-
Getting issue details...
STATUS
6.3.4. Publish technical documentation
Publish the technical documentation on the website: How to Document - Publishing to the website
6.3.5. Prepare HTML release notes from text release notes
Checkout GIT repository for Apache Ignite site: https://gitbox.apache.org/repos/asf/ignite-website.git
Just copy-paste latest release notes from previous release, replace text formatting with headers, lists items.
Place these notes to corresponding release folder, for example releases/2.7.5. If folder is absent, check results of step 6.2.3.
Commit and push changes. File should be available on the site automatically.
6.3.6. Update website with new download links
Once the new release is published to dist.apache.org/release wait a day or so for mirrors to catch up.
After the release was propagated across all the mirrors, update the "Source Releases" and "Binary Releases" sections of the website downloads page:
- Add a reference to the new release pointing out to https://downloads.apache.org/. Make sure to use [preferred]/[distdir] as part of the URL.
- Mark the new release as the latest one.
- Change the previous latest release record to link to https://archive.apache.org/ server.
- Add references to the release notes, API and technical documentations.
6.3.7. Update version for generating users advices to update
Find `latest` text file in GIT repository for Apache Ignite site.
Update Ignite version in this file. Commit changes and check result at https://ignite.apache.org/latest
6.3.8. Mark JIRA version as released
Go to JIRA project administration, select Apache Ignite, select Versions, select appropriate version and select Actions->Release.
6.3.9. Prepare docker image
This section Intended for preparing a docker image for https://apacheignite.readme.io/docs/docker-deployment
Prerequisite: Docker installed. For example, for Windows you can download https://docs.docker.com/docker-for-windows/ You may need to register at docker hub.
Credentials can be found in SVN:
https://svn.apache.org/repos/private/pmc/ignite/credentials/hub.docker.com.txt (PMC only)
You can get a binary distribution from the 'vote' folder of the release candidate.
Follow instructions from the README file and Dockerfile are available at:
https://github.com/apache/ignite/tree/master/deliveries/docker/apache-ignite
To build images run the following commands (insert correct release number):
docker build . -f ./x86_64/Dockerfile -t apacheignite/ignite:2.15.0 -t apacheignite/ignite:latest
This build takes a distribution from some folder named like apache-ignite*
You can validate images created by calling
docker images
Run image to check it works well:
docker run -it --rm --name apache-ignite <IMAGE>
To publish image run
docker login
Run
docker push apacheignite/ignite:2.15.0 docker push apacheignite/ignite:2.15.0-s390x docker push apacheignite/ignite:latest
Build and deploy s390x, arm64 images:
docker buildx build . -f ./s390x/Dockerfile -t apacheignite/ignite:2.15.0-s390x --push --platform linux/s390x --build-arg IGNITE_VER=2.15.0 docker buildx build . -f ./arm64/Dockerfile -t apacheignite/ignite:2.15.0-arm64 --push --platform linux/arm64
Check results at https://hub.docker.com/r/apacheignite/ignite
6.3.10. GCE & AWS virtual machines
6.3.10.2. AWS deployment
Todo: add description of steps ?
In addition, update the links to the images on the following documentation pages whenever is needed:
6.3.10.3. GCE deployment
Todo: add description of steps ?
https://apacheignite.readme.io/docs/google-compute-deployment
Prepare the cloud images and update links to them here https://ignite.apache.org/download.cgi#docker.
6.3.11. Delete previous release from dist and dev SVN directories
Old releases should be archived. Delete previous releases from https://dist.apache.org/repos/dist/release/ignite/... (PMC only):
svn rm -m "Archiving release X.Y.Z" https://dist.apache.org/repos/dist/release/ignite/X.Y.Z/ || true
Replace their download URLs by https://archive.apache.org/dist/ignite/...
Delete unsucessfull release candidates and packages (if any) from dev section of Apache SVN repository https://dist.apache.org/repos/dist/dev/ignite/
6.3.12. Upload NuGet packages to nuget.org
Retrieve all .nupkg files from build artifacts of step 4.5
Initial setup
You may need to install client tools for .NET https://docs.microsoft.com/en-us/nuget/install-nuget-client-tools (download file & add it to PATH). You can use nuget.exe 4.x version (because NuGet.exe 5.0 and later require .NET Framework 4.7.2 ).
It may be required to configure nuget.exe using NuGet.Config(XML) https://docs.microsoft.com/en-us/nuget/tools/cli-ref-push for using nuget.org as default push source.You can find configuration file under user home directory, for example, on Windows
cd %appdata%\Nuget
Configuration on Linux can be skipped. The push source is set up by -s param in the script.
Example of configuration:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
</packageSources>
<config>
<add key="defaultPushSource" value="https://nuget.org/" />
</config>
</configuration>
Credentials (API key to push packages) can be found in https://svn.apache.org/repos/private/pmc/ignite/credentials/nuget.org (PMC only, use Apache ID to login).
Windows
Windows, .NET Classic, PowerShell:
ls *.nupkg | % { nuget push $_.FullName API_KEY_HERE }
Power shell example ouptput:
Linux
Linux, .NET Core, sh:
for i in *.nupkg; do dotnet nuget push $i -k API_KEY_HERE -s "https://nuget.org/"; done
Check result at https://www.nuget.org/packages/Apache.Ignite/
Packages become visible from UI with some delay (~5-10 minutes)>
6.3.14. Check artifacts are available in Maven Central
Make sure that the artifacts were synced up to Maven Central. Refer to the recently happened issue ( INFRA-13073 - Getting issue details... STATUS )
- Once the repository is synced you can see the result in https://repo1.maven.org/maven2/org/apache/ignite/ and http://repo2.maven.org/maven2/org/apache/ignite/
- With some delay, artifact will be available on the site https://mvnrepository.com/artifact/org.apache.ignite/ignite-core
Usually, this sync should be completed in 4 hours after releasing repository at step 6.3.1. If it is not released, contact infra.
6.3.14. Update compatibility check versions
Create a new issue to enforce compatibility checks and prepare the PR. Add the released version to the IgniteReleasedVersion enum. Ask for a review on dev-list if necessary.
6.4. Announce the release
6.4.0 Prepare the blog post with the release improvements
Prepare a new blog post with the most major and meaningful release improvements.
Post this on - https://blogs.apache.org/ignite/ (ask Maxim Muzafarov or Denis A. Magda if there is any help required with the publication).
6.4.1. Send the release to announce to Apache Lists
Announce the release by sending a single message to announce@apache.org and Ignite's user/dev lists.
Please note that the 'announce 'list is ASF-level list and not all subscribers are aware of Apache Ignite. So it is recommended to include a short description of the project and put the link to the post from the 6.4.0.
Only PMC and Committers are able to send to the 'announce list.
Refer to the examples below:
6.4.2. Announce security vulnerabilities
If release includes any fixes for the security issue, then announce security vulnerabilities that were fixed in the release.
Follow the ASF process https://www.apache.org/security/committers.html.
Check with security@ignite.apache.org if there are any vulnerabilities.
6.4.3. Update Export Matrix
In case there are changes in set of used cryptographic libraries (see Crypto Notice), it is required to update Export Matrix http://www.apache.org/licenses/exports/#matrix
Matrix is stored in ASF-level SVN: https://svn.apache.org/repos/asf/infrastructure/site/trunk/content/licenses/exports/index.page/eccnmatrix.xml
Only VP, product can commit to matrix source. Ask VP, Ignite on a list to update.
Notice
This page was merged from several sources contributed by Nikolay Izhikov, Anton Vinogradov ,Denis A. Magda , Dmitry Pavlov , Pavel Tupitsyn , Andrey N. Gura , Vladimir Ozerov , Alexey Kuznetsov , Yakov Zhdanov, Ilya Murchenko, Nikita Amelchev
Open Issues
There are several unresolved issues / open questions:
6. Unclear on how to prepare/update 6.3.11.GCE&AWSvirtualmachines
7. Steps to be done to upload RPM/DEB packages
Overview
Content Tools
Apps
2 Comments
Maxim Muzafarov
I've faced with incorrect caching credentials by GibBash console utility under Windows.
Maxim Muzafarov
In case of getting an error:
Do the following: