provided in the vote email thread

Note:  This document is a work in progress and heavily derived from the NiFi Release guide generated by the community.  The idea is that with processes fully established for each project we can consolidate at a later point.

Apache MiNiFi Release Guidelines

The purpose of this document is to capture and describe the steps involved in producing an official release of Apache NiFi MiNiFI for both Java and C++ versions. It is written specifically to someone acting in the capacity of a Release Manager (RM).

The objective

Our aim is to produce an official Apache NiFi MiNiFi C++ release from an existing release branch.

Background Material

Terms

  • Release Manager (RM) - the Apache NiFi PMC Member or Committer acting as Release Manager for a particular release of Apache NiFi MiNiFi C++.
  • Release Candidate (RC) - an iteration of the release process that is proposed for a vote by the Apache NiFi Community.
  • Community - the community of people with an interest in the improvement and advancement of Apache NiFi, and its associated projects, including end-users, developers, evangelists, and advisers.
  • PMC - within the Apache NiFi community, members of the PMC oversee the ongoing project.
  • Committer - with the Apache NiFi community, committers have gain the privilege to commit changes to the Apache NiFi codebase.

High level flow of a release

  • The Apache NiFi community is constantly contributing to JIRA tickets assigned to the next release.
  • At some point the number of tickets open/remaining for the next release begins to approach zero.
  • A member of the community suggests a release and initiates a discussion.
  • Someone volunteers to perform the Release Manager (RM) role for the release. (This can be a committer but Apache guides indicate a preference for a PMC member.)
  • The RM validate the proposed release and stages the source code, Maven artifacts, and distributable files for a Release Candidate (RC).
  • The RM initiates a vote on the RC by the NiFi community.
  • If the NiFi community rejects the RC, the issues noted are resolved and another RC is generated.
  • If the NiFi community accepts the RC, the staged source code, artifacts, and distribution files are moved to the appropriately locations for public release.

Variable reference substitutions

Throughout this guide, references must be made to names and values that will vary from release to release. For clarity those variable values have been written like Bash variable references. When a term like "/tmp/src/nifi-${MINIFI_VERSION}" is seen in an instruction or email template it should be replaced with "/tmp/src/nifi-0.7.0" when working the release of "Apache NiFi 0.7.0".

Substitutions used in tasks and email templates:
ReferenceExample valueDescription
${BRANCH}mainthe development branch on which the release is based
${MINIFI_VERSION}0.10.0the version currently in development on the release branch, to be released



${JIRA_TICKET}MINIFICPP-1234the JIRA ticket created by the release manager for the release tasks
${RC}1the Release Candidate index start at 1 for the first release candidate
${RC_TAG_COMMIT_ID}
the commit ID of the RC tag created during the Maven release process
${RM_USERID}johndoethe Apache account ID of Release Manager
${RELEASE_TAG}rel/minificpp-0.10.0the Git repository tag for the source code as released
${VOTE_THREAD_URL}[0.10.0 vote thread][0100-rc2-vote]the URL for the Apache Pony Mail archive of the release vote thread


To be practical but avoid confusion with future release details, these example values reflect the release details of MiNiFi C++ 0.10.0 RC2.

What to validate and how to validate a release

The following is a list of the sorts of things that will be validated and are the basics to check when evaluating a release for a vote.
  • Are the LICENSE and NOTICE files present in the source root and complete?
    • Specifically look in the nifi-minifi-cpp-${MINIFI_VERSION}-sources.tar.gz artifact and ensure these files are present at the root of the archive.
  • Evaluate the sources and dependencies.
    • Does the overall LICENSE and NOTICE appear correct?
    • Do all licenses fit within the ASF approved licenses?
  • Is there a README available that explains how to build the application and to execute it?
    • Look in the *-source.tar.gz artifact root for the readme.
  • Are the signatures and hashes correct for the source release?
    • Validate the hashes of the sources artifact do in fact match
    • Validate the signature of the source artifact.
    • Need a quick reminder on how to verify a signature?
  • Do all sources have necessary headers?
    • Extract the sources file into a directory and execute mkdir build && cd build && cmake .. && make package && make test && make linter
  • Are there no unexpected binary files in the release?
    • The only thing we'd expect would be potentially test resources files.
  • Does the app (if appropriate) execute and function as expected?
This list is reflected in the Release Vote and Release Helper Guide emails that are sent once the release has been staged in the Git and Nexus repositories.

The Release Process

The release process includes steps to be performed by the Release Manager as well as the Apache NiFi developer community.

Step 1. Configure the build environment (RM and community)

  1. Follow the steps outlined in the https://github.com/apache/nifi-minifi-cpp#getting-started to prepare the development system.
  2. Confirm that the local Git workspace is configured with an origin remote pointing to the RM's personal fork of the NiFi source and an "ASF" remote pointing to the Apache Git Repository for NiFi.
    $ git remote -v
    asf    https://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp.git (fetch)
    asf    https://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp.git (push)
    origin    https://github.com/${RM_USERID}/nifi-minifi-cpp.git (fetch)
    origin    https://github.com/${RM_USERID}/nifi-minifi-cpp.git (push)
    Additional remotes will not cause a problem if these two are correct. Other configurations are perfectly acceptable but the appropriate adjustments to the steps in this guide must be made by the release manager.
  3. Confirm that source code can be checked out for the branch being released.
    git checkout ${BRANCH}
  4. Confirm that the entire application builds correctly in the build environment.

Step 2. Prepare and stage the release (RM)

  1. Create a JIRA ticket for the release tasks for version ${MINIFI_VERSION}.
    _The resulting JIRA ticket number is referred to as  ${JIRA_TICKET}  in this guide.
  2. Create the next version in JIRA, if it doesn't already exist, so work can continue towards that release.  Versions can be managed at https://issues.apache.org/jira/projects/MINIFICPP?selectedItem=com.atlassian.jira.jira-projects-plugin%3Arelease-page
  3. Create meaningful release notes for this version if not already created.  Add them to the Release Notes - MiNiFi (C++) page on the MiNiFi wiki.
    1. May be useful to add links to processor documentation so it is easier to diff releases. 
  4. Create a new branch off ${BRANCH} named after the JIRA ticket.
    $ git checkout -b ${JIRA_TICKET}-RC${RC} ${BRANCH}
  5. Verify that you have the needed dependencies to build and run MiNiFi
  6. Ensure the the full application builds, all tests work, and source passes linting by executing the following:
    TODO/DISCUSS:  Since we don't have a dependency management framework we will include only base extensions.
    $ mkdir build && cd build && cmake -DENABLE_COAP=ON -DSKIP_TESTS= -DUSE_SHARED_LIBS=ON -DPORTABLE=ON -DBUILD_ROCKSDB=ON -DEXCLUDE_BOOST=ON -DBUILD_IDENTIFIER= -DCMAKE_BUILD_TYPE=Release -DFAIL_ON_WARNINGS= .. && make package_source && make -j$(nproc) package && make -j$(nproc) test && make -j$(nproc) linter && make docker && make docker-verify
  7. Startup and test the application with from the build folder:
    $ tar xvzf nifi-minifi-cpp-${MINIFI_VERSION}.tar.gz && ./nifi-minifi-cpp-${MINIFI_VERSION}/bin/minifi.sh start
  8. Ensure that python version compatibility tests pass by running the https://github.com/${RM_USERID}/nifi-minifi-cpp/actions/workflows/verify-python-compatibility.yml CI actions on the ${JIRA_TICKET}-RC${RC} branch
  9. Evaluate and ensure the appropriate license headers are present on all source files.

  10. Ensure LICENSE and NOTICE files are complete and accurate. (Developers should always be keeping these up to date as they go along adding source and modifying dependencies to keep this burden manageable.)

  11. If the validated artifacts all look good then create a tag and push the branch to the ASF repository.

    $ git tag -s "minifi-cpp-${MINIFI_VERSION}-RC${RC}" -m "${JIRA_TICKET} RC${RC} release candidate of NiFi MiNiFi C++ ${MINIFI_VERSION}" ${JIRA_TICKET}-RC${RC}
    $ git push --tag asf ${JIRA_TICKET}-RC${RC}

Generate the convenience binary:
$ make centos
$ mv nifi-minifi-cpp-${MINIFI_VERSION}-bin-{centos,linux}.tar.gz

Create the Windows MSI installer. The easiest way to do it is to use the python bootstrap batch file located at bootstrap/py_bootstrap.bat and select the "One Click Build" option. This builds and packages the binaries on Windows into an MSI installer using the default extension options and Release build type.


Create the signature and hashes for the source release and convenience binary files:
for artifact in $(find . -type f \( -name '*.tar.gz' -o -name '*.msi' \))
do
  echo $artifact;
  echo  " Generating ASCII armored GPG signature "
  gpg -a -b --digest-algo=SHA512 ${artifact};
  echo  " Generating sha512 sum hash "
  sha512sum  ${artifact} |  cut -d" " -f1 > ${artifact}.sha512
  echo  " Generating sha256 sum hash "
  sha256sum  ${artifact} |  cut -d" " -f1 > ${artifact}.sha256
done
    1. ASCII armored GPG signatures (--digest-algo=SHA512 select the SHA512 hash algorithm). Configure GPG to always prefer stronger hashes.
      $ gpg -a -b --digest-algo=SHA512 nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz    # produces nifi-minifi-cpp${MINIFI_VERSION}-source.tar.gz.asc
      $ gpg -a -b --digest-algo=SHA512 nifi-minifi-cpp-${MINIFI_VERSION}-bin.tar.gz          # produces nifi-minifi-cpp-${MINIFI_VERSION}-bin.tar.gz.asc
      $ gpg -a -b --digest-algo=SHA512 nifi-minifi-cpp-${MINIFI_VERSION}-bin.tar.gz          # produces nifi-minifi-cpp-${MINIFI_VERSION}-bin.tar.gz.asc
    2. Generate SHA256 hash summaries.
      $ shasum -a 256 nifi-${MINIFI_VERSION}-source-release.zip | cut -d" " -f1 >  nifi-${MINIFI_VERSION}-source-release.zip.sha256
      $ shasum -a 256 nifi-${MINIFI_VERSION}-bin.tar.gz | cut -d" " -f1 >  nifi-${MINIFI_VERSION}-bin.tar.gz.sha256
      $ shasum -a 256 nifi-${MINIFI_VERSION}-bin.zip | cut -d" " -f1 >  nifi-${MINIFI_VERSION}-bin.zip.sha256
    3. Generate SHA512 hash summaries.
      $ sha512sum nifi-${MINIFI_VERSION}-source-release.zip | cut -d" " -f1 >  nifi-${RELEASAE}-source-release.zip.sha512
      $ sha512sum nifi-${MINIFI_VERSION}-bin.tar.gz | cut -d" " -f1 >  nifi-${RELEASAE}-bin.tar.gz.sha512
      $ sha512sum nifi-${MINIFI_VERSION}-bin.zip | cut -d" " -f1 >  nifi-${RELEASAE}-bin.zip.sha512

For reviewing the release candidate, upload the source, the MSI installer and the convenience binary tarballs, along with their hashes and signatures, to the ASF distribution repo using subversion:

svn co https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp
cd nifi-minifi-cpp
mkdir ${MINIFI_VERSION}
cp nifi-minifi-cpp-* ${MINIFI_VERSION}/
svn add ${MINIFI_VERSION}
svn commit

(you need to be a committer to do this, and svn will prompt you for your Apache password).

Step 4. Error recovery (RM)

If anything isn't correct about the staged artifacts you can drop the staged repo from repository.apache.org and delete the local tag in git. If you also delete the local branch and clear your local maven repository under org/apache/nifi then it is as if the release never happened. Before doing that though try to figure out what went wrong so the Release Guide can be updated or corrected if necessary.
So, as has been described here you can test the release process until you get it right. The mvn versions:set and mvn versions:commit commands can come in handy to help do this so you can set versions to something clearly release test related.

Step 5. Release Vote (RM and community)

After the release source and artifacts are staged in the repositories it's time for the RM to send a release vote to the NiFi community.
Once the release vote is called for, members of the NiFi developer community have 72 hours to evaluate the RC and cast their vote by replying to the "[VOTE] Release ..." email sent by the RM.
NOTE: The release vote is majority rule vote that must include at least 3 binding +1 votes Apache NiFi PMC members and more positive than negative binding votes.
For the resolved issues the link can be generated here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12321520
  1. RM sends a vote request email to the NiFi Developers Mailing List.

  2. RM sends the following helper email to the NiFi Developers Mailing List.

    1. TO: dev@nifi.apache.org
    2. FROM: ${RM_USERID}@apache.org
    3. SUBJECT:  Apache NiFi MiNiFi C++ ${MINIFI_VERSION} Release Helper Guide

      Hello Apache NiFi community,

      Please find the associated guidance to help those interested in validating/verifying the release so they can vote.

      # Download latest KEYS file:
      https://dist.apache.org/repos/dist/release/nifi/KEYS

      # Import keys file:
      gpg --import KEYS

      # Pull down nifi-minifi-cpp-${MINIFI_VERSION} source release artifacts for review:

      wget https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/${MINIFI_VERSION}/nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz
      wget https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/${MINIFI_VERSION}/nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz.asc
      wget https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/${MINIFI_VERSION}/nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz.sha256
      wget https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/${MINIFI_VERSION}/nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz.sha512

      # Verify the signature
      gpg --verify nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz.asc

      # Verify the hashes (sha256, sha512) match the source and what was provided in the vote email thread
      sha256sum nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz sha512sum nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz

      # Extract nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz
      tar xvzf       nifi-minifi-cpp-${MINIFI_VERSION}-source.tar.gz

      # Verify the build works including tests and linter checks
      cd nifi-minifi-cpp-${MINIFI_VERSION}-source
      mkdir build && cd build && cmake .. && make package && make test && make linter
      # or:
      # ./bootstrap.sh && cd build && make package && make test && make linter

      # On Windows:
      # Install dependencies as documented: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=139627733
      # cd nifi-minifi-cpp-${MINIFI_VERSION}-source
      # win_build_vs.bat build /P

      # Verify the contents contain a good README, NOTICE, and LICENSE.

      # Verify the git commit ID is correct

      # Verify the RC was branched off the correct git commit ID

      # Look at the resulting convenience binary as found in build/nifi-minifi-cpp-${MINIFI_VERSION}-bin.tar.gz

      # Make sure the README, NOTICE, and LICENSE are present and correct

      # Run the resulting convenience binary and make sure it works as expected

      # Send a response to the vote thread indicating a +1, 0, -1 based on your findings.

      Thank you for your time and effort to validate the release!
  3. Developers in the community review the release candiate and reply to the vote email with their vote.

  4. After 72 hours if

    • at least 3 binding (PMC members) cast +1 votes, and
    • the positive binding votes out number any negative binding votes
  5. the vote passes and the release candidate is officially released. If the vote does not pass, corrections are made on the release branch and a new release candidate is put forward for a new vote.
  6. RM sends vote result email.

    • TO: dev@nifi.apache.org
    • FROM: ${RM_USERID}@apache.org
    • SUBJECT:  [RESULT][VOTE] Release Apache NiFi MiNiFi C++ ${NIFI_VERSION}

      Apache NiFi Community,

      I am pleased to announce that the ${MINIFI_VERSION} release of Apache NiFi MiNiFi C++ passes with
        X +1 (binding) votes
        Y +1 (non-binding) votes
        0 0 votes
        0 -1 votes

      Thanks to all who helped make this release possible.

      Here is the PMC vote thread: ${VOTE_THREAD_URL}

Step 6. Finalize the Release

After the vote is complete and the release is approved, these steps complete the release process.
  1. Move convenience binaries and related artifacts from dist/dev to dist/release: (a member of the PMC needs to run this)

    $ svn move -m " ${JIRA_TICKET}"   https://dist.apache.org/repos/dist/dev/nifi/nifi-minifi-cpp/ ${MINIFI_VERSION} https://dist.apache.org/repos/dist/release/nifi/nifi-minifi-cpp/ ${MINIFI_VERSION}

  2. Update the MiNiFi website to point to the new download(s) by creating a pull request on https://github.com/apache/nifi-site and following the instructions in README.md.  Remove older release artifacts from download page (leave the current release and the previous one). In addition to updating the download page as described delete artifacts other than the current/new release from the dist/nifi SVN storage. They are already in the archive location so no need to do anything else.

  3. Update the NiFi Web Page to indicate NEWS of the release as appropriate

  4. Create a proper signed tag of the released codebase based on the RC Tag created during the Maven release process.

    $ git tag -s rel/minifi-cpp-${MINIFI_VERSION} -m "${JIRA_TICKET} signed release tag for approved release of NiFi MiNiFi C++ ${MINIFI_VERSION}" ${RC_TAG_COMMIT_ID}
    For instructions on setting up to sign your tag see  here .

  5. Push the release tag to the official ASF repository.

    $ git push asf rel/minifi-cpp-${MINIFI_VERSION}

  6. Publish the convenience binaries to docker hub
    • To build a multiarch docker image you need the following:
      • An x86 architecture machine with docker available (in this example it will be the host machine)
      • An arm64 architecture machine with docker available (in this example it will be the remote machine)
      • Access to the dockerhub repository you want to push the image to with logged in credentials (here we will use the official apache/nifi-minifi-cpp repository)Update the release notes with the final date of the release.
    • Run the following commands:
      • $ ssh-add ~/.ssh/remote-host-public-key.pem  # ssh key for the remote host for the arm64 build
      • $ docker context create node-arm64 --docker "host=ssh://<remote host user>@<remote host ip>"  # create the new build context for arm64
      • $ docker buildx create --use --name multi-arch-build default   # create the new builder instance
      • $ docker buildx create --append --name multi-arch-build node-arm64   # add the arm64 build context for the new builder instance
      • $ cmake -DDOCKER_BUILD_ONLY=ON -DDOCKER_PUSH=ON -DDOCKER_TAGS=apache/nifi-minifi-cpp:${MINIFI_VERSION} -DDOCKER_PLATFORMS=linux/amd64,linux/arm64 .. && make docker  # build and push the multiarch docker image
  7. Update the release notes with the final date of the release.

  8. After the release has been complete for 24 hours send the release announcement.



  • No labels