Apache Airflow chose GitHub Actions and build the whole CI solution on it. It turned out to be robust and seamless to run.
The architecture of the proposed solutions is shown below. It describes Apache Airflow Architecture, but it should be applicable to other projects:
UPDATE: Apache Airflow has changed the process since - there is no more automated push to DockerHub - all the CI images are only automatically published in GHCR.IO for the purpose of development and CI. Apache Airflow switched to manual process of publishing RC and RELASE images by the release manage (using semi-automation of course). Similarly as preparing the releases, building and pushing the images is now executed on the release manager-controlled hardware. Once the image is locally built it is pushed to DockerHub using personal authentication of the release manager.
The following components are part of the CI infrastructure
The following CI Job runs are currently run for Apache Airflow, and each of the runs have different purpose and context.
- Pull Request Run - Those runs are results of PR from the forks made by contributors. Most builds for Apache Airflow fall into this category. They are executed in the context of the "Fork", not main Airflow Code Repository which means that they have only "read" permission to all the GitHub resources (container registry, code repository). This is necessary as the code in those PRs (including CI job definition) might be modified by people who are not committers for the Apache Airflow Code Repository. The main purpose of those jobs is to check if PR builds cleanly, if the test run properly and if the PR is ready to review and merge. The runs are using cached images from the Private GitHub registry - CI, Production Images as well as base Python images that are also cached in the Private GitHub registry.
- Direct Push/Merge Run - Those runs are results of direct pushes done by the committers or as result of merge of a Pull Request by the committers. Those runs execute in the context of the Apache Airflow Code Repository and have also write permission for GitHub resources (container registry, code repository). The main purpose for the run is to check if the code after merge still holds all the assertions - like whether it still builds, all tests are green. This is needed because some of the conflicting changes from multiple PRs might cause build and test failures after merge even if they do not fail in isolation. Also those runs are already reviewed and confirmed by the committers so they can be used to do some housekeeping - for now they are pushing most recent image build in the PR to the Github Private Registry - which is our image cache for all the builds. Another purpose of those runs is to refresh latest Python base images. Python base images are refreshed with varying frequency (once every few months usually but sometimes several times per week) with the latest security and bug fixes. Those patch level images releases can occasionally break Airflow builds (specifically Docker image builds based on those images) therefore in PRs we always use latest "good" python image that we store in the private GitHub cache. The direct push/master builds are not using registry cache to pull the python images - they are directly pulling the images from DockerHub, therefore they will try the latest images after they are released and in case they are fine, CI Docker image is build and tests are passing - those jobs will push the base images to the private GitHub Registry so that they be used by subsequent PR runs.
- Scheduled Run - those runs are results of (nightly) triggered job - only for master branch. The main purpose of the job is to check if there was no impact of external dependency changes on the Apache Airflow code (for example transitive dependencies released that fail the build). It also checks if the Docker images can be build from the scratch (again - to see if some dependencies have not changed - for example downloaded package releases etc. Another reason for the nightly build is that the builds tags most recent master with nightly-master tag so that DockerHub build can pick up the moved tag and prepare a nightly public master build in the DockerHub registry. The v1-10-test branch images are build in DockerHub when pushing v1-10-stable manually
All runs consist of the same jobs, but the jobs behave slightly differently or they are skipped in different runs. Here is a summary of the run types with regards of the jobs they are running. Those jobs often have matrix run strategy which runs several different variations of the jobs (with different Backend type /Python version, type of the tests to run for example)
Pull Request Run
Direct Push/Merge Run
* Builds all images from scratch
|Static checks 1||Performs first set of static checks||Yes||Yes||Yes *|
|Static checks 2||Performs second set of static checks||Yes||Yes||Yes *|
|Docs||Builds documentation||Yes||Yes||Yes *|
|Build Prod Image||Builds production image||Yes||Yes||Yes *|
|Prepare Backport packages||Prepares Backport Packages for 1.10.*||Yes||Yes||Yes *|
Counts how many python files changed in the change.
Used to determine if tests should be run
|Yes||Yes (but it is not used)||Yes (but it is not used)|
|Tests||Run all the combinations of Pytest tests for Python code||Yes (if pyfiles count >0)||Yes||Yes*|
|Quarantined tests||Those are tests that are flaky and we need to fix them||Yes (if pyfiles count >0)||Yes||Yes *|
Checks if requirement constraints in the code are up-to-date
|Yes (fails if missing requirement)|
Fails if missing requirement
Eager dependency upgrade
Does not fail for changed requirements
|Pull Python from cache||Pulls Python base images from Github Private Image registry to keep the last good python image used in PRs||Yes||No||-|
|Push Python from cache||Pushes Python base images to Github Private Image registry - checks if latest image is fine and pushes if so||-||Yes||-|
|Push Prod image|
Pushes production images to GitHub Private Image Registry
This is to cache the build images for following runs.
|Push CI image|
Pushes CI images to GitHub Private Image Registry
This is to cache the build images for following runs.
|Tag Repo nightly|
Tags the repository with nightly tag
It is a lightweight tag that moves nightly
Triggers DockerHub build for public registry