Adding a Python version

  1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well.
  2. Add a Beam Python container for the new python version.
    1. https://github.com/apache/beam/tree/master/sdks/python/container
  3. Add a new Python version to different test-suites.
    1. Tox test suites - link
    2. All the required Gradle tasks such as pre-commits, post-commits etc.
    3. Make sure all the runners support the new Python version.
    4. Fix any tests that fail. Typically, a new Python version requires updating Beam Type Inference code. https://github.com/apache/beam/issues/31047
  4. Add the Github actions suite for the new Python version.
    1. https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml
  5. Add the support for building wheels for the new Python version.
    1. https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml
  6. Update the upper limit in __init__.py with the next major Python version.
  7. Add new Python version in release validation scripts: https://github.com/apache/beam/pull/31415
  • If there is a new feature update or there is a regression when adding a new Python version, please file an issue.
    • All the unit tests and Integration tests must pass before merging the new version. 
    • If you are a non-committer, please ask the committers to run a seed job on your PR to test all the new changes.

For an example, see PRs associated with https://github.com/apache/beam/issues/29149, and commits on https://github.com/apache/beam/pull/30828 which add Python 3.12 support.

Removing a Python version

  1. Bump the python version in setup.py and update the python version warning in __init__.py. 
  2. Delete the unsupported Python version containers from sdks/python/container
  3. Remove the test-suites for the unsupported python version from
    1. Remove unsupported py{version} from the following files/directories
      1. sdks/python/test-suites/gradle.properties
      2. apache_beam/testing/tox
        1. Move any workflows that only exist for the deprecated Python version from tox/py3X to the next minimum Python version's folder
      3. apache_beam/testing/dataflow
      4. apache_beam/testing/direct
      5. apache_beam/testing/portable
    2. Remove the unsupported python version gradle tasks from
      1. beam/.test-infra/{DSL_SCRIPT}.groovy
      2. build.gradle.kts
      3. settings.gradle.kts
      4. buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
    3. Migrate GitHub actions from the deprecated Python version to the next one
      1. Example PR: https://github.com/apache/beam/pull/32429
      2. Make these changes on a branch in the main Beam repository if possible so you can execute the new workflows directly for testing.
    4. Remove the support for building wheels and source distributions for the unsupported Python version from .github/workflows/build_wheels.yml
    5. Remove the unsupported Python version from GitHub actions unit tests at .githubworkflows/python_tests.yml
    6. Remove the unsupported Python version from sdks/python/tox.ini
  4. Clean up any code that applies to unsupported Python version[Optional]


Example tracking issue for removing Python 3.8 support from Apache Beam: https://github.com/apache/beam/issues/31192

  • No labels