This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Skip to end of metadata
Go to start of metadata

Building an RC

The Release Candidate artefacts we vote upon should be the exact ones we vote against, without any modification than renaming – i.e. the contents of the files must be the same between voted release canidate and final release. Because of this the version in the built artefacts that will become the official Apache releases must not include the rcN suffix.

  • Set environment variables 

    # Set Version
    export VERSION=1.10.2rc3
    
    
    # Set AIRFLOW_REPO_ROOT to the path of your git repo
    export AIRFLOW_REPO_ROOT=$(pwd)
    
    
    # Example after cloning
    git clone https://github.com/apache/airflow.git airflow
    cd airflow
    export AIRFLOW_REPO_ROOT=$(pwd)
  • set your version to 1.10.2 in airflow/version.py (without the RC tag)
  • Commit the version change.
  • Tag your release

    git tag ${VERSION}
  • Clean the checkout: the sdist step below will

    git clean -fxd
  • Tarball the repo

    git archive --format=tar.gz ${VERSION} --prefix=apache-airflow-${VERSION}/ -o apache-airflow-${VERSION}-source.tar.gz
  • Generate sdist
    NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included in the file produced.

    python setup.py compile_assets sdist bdist_wheel
  • Rename the sdist

    mv dist/apache-airflow-${VERSION%rc?}.tar.gz apache-airflow-${VERSION}-bin.tar.gz
    mv dist/apache_airflow-${VERSION%rc?}-py2.py3-none-any.whl apache_airflow-${VERSION}-py2.py3-none-any.whl
  • Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)

    ${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-source.tar.gz
    ${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-bin.tar.gz
    ${AIRFLOW_REPO_ROOT}/dev/sign.sh apache_airflow-${VERSION}-py2.py3-none-any.whl


  • Push Tags

    git push --tags
  • Push the artefacts to ASF dev dist repo

    # First clone the repo
    svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev
    
    # Create new folder for the release
    cd airflow-dev
    svn mkdir ${VERSION}
    
    # Move the artefacts to svn folder & commit
    mv ${AIRFLOW_REPO_ROOT}/apache{-,_}airflow-${VERSION}* airflow-dev/${VERSION}/
    cd ${VERSION}
    svn add *
    svn commit -m "Add artefacts for Airflow ${VERSION}" 

PyPI Snapshots (Optional)

At this point we have the artefact that we vote on, but as a convenience to developers we also want to publish "snapshots" of the RC builds to pypi for installing via pip. To do this we need to

  • Edit the airflow/version.py to include the RC suffix.
  • python setup.py compile_assets sdist bdist_wheel
  • Follow the steps in Publishing to PyPi
  • Throw away the change - we don't want to commit this: git checkout airflow/version.py

It is important to stress that this snapshot is not intended for users.

Make sure your public key is on id.apache.org and in KEYS

You will need to sign the release artifacts with your pgp key. After you have created a key, make sure you

# Create PGP Key
gpg --gen-key

# Checkout ASF dist repo
svn checkout https://dist.apache.org/repos/dist/release/airflow
cd airflow


# Add your GPG pub key to KEYS file. Replace "Kaxil Naik" with your name
(gpg --list-sigs "Kaxil Naik" && gpg --armor --export "Kaxil Naik" ) >> KEYS


# Commit the changes
svn commit -m "Add PGP keys of Airflow developers"


See this for more detail on creating keys and what is required for signing releases.

http://www.apache.org/dev/release-signing.html#basic-facts

Voting on an RC

  • Once the RC is built (both source and binary), put them in the dev SVN repository:
    https://dist.apache.org/repos/dist/dev/airflow/

  • Use the dev/airflow-jira script to generate a list of Airflow JIRAs that were closed in the release.

  • Send out a vote to the dev@airflow.apache.org mailing list:

    [VOTE] Airflow 1.10.2rc3
    Hey all,
    
    I have cut Airflow 1.10.2 RC3. This email is calling a vote on the release,
    which will last for 72 hours. Consider this my (binding) +1.
    
    Airflow 1.10.2 RC3 is available at:
    https://dist.apache.org/repos/dist/dev/airflow/1.10.2rc3/
    
    *apache-airflow-1.10.2rc3-source.tar.gz* is a source release that comes
    with INSTALL instructions.
    *apache-airflow-1.10.2rc3-bin.tar.gz* is the binary Python "sdist" release.
    
    Public keys are available at:
    https://dist.apache.org/repos/dist/release/airflow/KEYS
    
    Only votes from PMC members are binding, but members of the community are
    encouraged to test the release and vote with "(non-binding)".
    
    Please note that the version number excludes the `rcX` string, so it's now
    simply 1.10.2. This will allow us to rename the artifact without modifying
    the artifact checksums when we actually release.
    
    
    Changes since 1.10.2rc2:
    *Bugs*:
    [AIRFLOW-3732] Fix issue when trying to edit connection in RBAC UI
    [AIRFLOW-2866] Fix missing CSRF token head when using RBAC UI (#3804)
    ...
    
    
    *Improvements*:
    [AIRFLOW-3302] Small CSS fixes (#4140)
    [Airflow-2766] Respect shared datetime across tabs
    ...
    
    
    *New features*:
    [AIRFLOW-2874] Enables FAB's theme support (#3719)
    [AIRFLOW-3336] Add new TriggerRule for 0 upstream failures (#4182)
    ...
    
    
    *Doc-only Change*:
    [AIRFLOW-XXX] Fix BashOperator Docstring (#4052)
    [AIRFLOW-3018] Fix Minor issues in Documentation
    ...
    
    Cheers,
    <your name>
  • Once the vote has been passed, you will need to send a result vote to dev@airflow.apache.org:

    [RESULT][VOTE] Airflow 1.10.2rc3
    Hello,
    
    Apache Airflow 1.10.2 (based on RC3) has been accepted.
    
    4 “+1” binding votes received:
    - Kaxil Naik  (binding)
    - Bolke de Bruin (binding)
    - Ash Berlin-Taylor (binding)
    - Tao Feng (binding)
    
    
    4 "+1" non-binding votes received:
    
    - Deng Xiaodong (non-binding)
    - Stefan Seelmann (non-binding)
    - Joshua Patchus (non-binding)
    - Felix Uellendall (non-binding)
    
    Vote thread:
    https://lists.apache.org/thread.html/736404ca3d2b2143b296d0910630b9bd0f8b56a0c54e3a05f4c8b5fe@%3Cdev.airflow.apache.org%3E
    
    I'll continue with the release process and the release announcement will follow shortly.
    
    Cheers,
    <your name>

Publishing release

  • After both votes pass (see above), you need to migrate the RC artifacts that passed to this repository:
    https://dist.apache.org/repos/dist/release/airflow/
    (The migration should including renaming the files so that they no longer have the RC number in their filenames.)

Publishing to PyPi

  • Create a ~/.pypirc file:

    $ cat ~/.pypirc 
    [distutils]
    index-servers =
      pypi
      pypitest
    
    [pypi]
    username=your-username
    password=*********
    
    [pypitest]
    repository=https://test.pypi.org/legacy/
    username=your-username
    password=*********

    (more details here.)

  • Set proper permissions for the pypirc file:

    $ chmod 600 ~/.pypirc
  • Confirm that airflow/version.py is set properly.
  • Install the package in PyPi's test environment:

    python setup.py compile_assets sdist bdist_wheel upload -r pypitest
  • Verify that the test package looks good by downloading it and installing it into a virtual environment. The package download link is available at:
    https://test.pypi.org/project/apache-airflow/#files
  • Install the package in PyPi's production environment:

    python setup.py compile_assets sdist bdist_wheel upload -r pypi
  • Again, confirm that the package is available here:
    https://pypi.python.org/pypi/apache-airflow

Updating CHANGELOG.md

  • Get a diff between the last version and the current version:

    $ git log 1.8.0..1.9.0 --pretty=oneline
  • Update CHANGELOG.md with the details, and commit it.

Notifying developers of release

  • Notify dev@airflow.apache.org that the artifacts have been published:

    Airflow 1.10.2 is released
    Dear Airflow community,
    
    I'm happy to announce that Airflow 1.10.2 was just released.
    
    The source release, as well as the binary "sdist" release, are available
    here:
    
    https://dist.apache.org/repos/dist/release/airflow/1.10.2/
    
    We also made this version available on PyPi for convenience (`pip install apache-airflow`):
    
    https://pypi.python.org/pypi/apache-airflow
    
    The documentation is available on:
    https://airflow.apache.org/
    https://airflow.apache.org/1.10.2/
    https://airflow.readthedocs.io/en/1.10.2/
    https://airflow.readthedocs.io/en/stable/
    
    Find the CHANGELOG here for more details:
    
    https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
    
    Cheers,
    <your name>
  • Update the Announce page on the wiki

Publishing documentation

Bulk migrate JIRAs

Old stuff that doesn't apply anymore

 View legacy instructions

Committers should use the following commands and steps to create an Apache Airflow release:

  1. Run dev/airflow-jira on the test branch
    dev/airflow-jira compare 1.8.1

  2. Commit RCs here
    svn co https://dist.apache.org/repos/dist/dev/airflow

    Make sure CHANGELOG and UPDATING.md have been updated.

  3. Remove previous versions from  https://dist.apache.org/repos/dist/dev/airflow (`svn rm` -> `svn commit`)

  4. Run a travis build, confirm it passes (duh.)
  5. Build airflow
    python setup.py sdist

  6. Generate MD5
    gpg --print-md MD5 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.md5

  7. Generate the SHA512
    gpg --print-md SHA512 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.sha

  8. Upload your GPG public key to https://pgp.mit.edu

  9. Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of the file

  10. Sign Airflow tar.gz
    gpg --armor --output apache-airflow-1.8.1rc0+apache.incubating.tar.gz.asc --detach-sig apache-airflow-1.8.1rc0+apache.incubating.tar.gz

  11. Do votes in dev@airflow.incubator.apache.org and general@incubator.apache.org
    https://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRdcn4dkJwumYtUMsfFLq-gwjg%40mail.gmail.com%3E
    https://mail-archives.apache.org/mod_mbox/incubator-general/201705.mbox/%3CCABYbY7cnACrrsrFtdwtq_-4AokMAe8CBOcWSvNz%2BhgB9DfVEvQ%40mail.gmail.com%3E

  12. Commit final release here
    svn co https://dist.apache.org/repos/dist/release/airflow

  13. Merge the test branch into the stable branch
    git pull apache v1-8-test
    git pull apache v1-8-stable
    git co v1-8-stable
    git merge v1-8-test
    git tag 1.8.1
    git push apache v1-8-stable
    git push apache 1.8.1

  14. Follow these instructions to publish to PyPi
    Note: You must remove the +incubating from the airflow/version.py file.

Source releases:
  • ...
  • Generate MD5s
    gpg --print-md MD5 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.md5

  • Generate the SHA512
    gpg --print-md SHA512 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.sha

  • Upload your GPG public key to https://pgp.mit.edu

  • Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of the file

  • Sign Airflow tar.gz
    gpg --armor --output apache-airflow-1.8.1rc0+apache.incubating.tar.gz.asc --detach-sig apache-airflow-1.8.1rc0+apache.incubating.tar.gz



  • No labels