WORK IN PROGRESS

Background

Airflow's CI runs on the shared GitHub-hosted runner pool. For a public repository, runner minutes are free and unlimited - so cost is not the problem.

The problem is concurrency: GitHub caps the number of concurrent jobs at the account level, shared across every repository in the account.
Because all Apache projects live in the single apache  GitHub organization, the entire foundation draws on one shared concurrency ceiling that the ASF cannot simply scale.

Airflow is one of the foundation's highest-volume repositories, so its large, bursty test matrix both queues behind itself and crowds out every other ASF project.

The ASF cannot price its way out of this - public-repo minutes are already free; the lever is more, controllable capacity, not more budget.

Meanwhile, AWS has donated credits to Apache Airflow, today used mainly to host the project's documentation (including archived versions) on S3.

The idea behind this proposal is to extend that existing, donated AWS footprint to CI - relieving the shared concurrency ceiling with capacity Airflow controls, and drawing on the donation rather than new foundation
spend.

Purpose

A proposal for introducing AWS CodeBuild-backed self-hosted GitHub Actions runners in Airflow CI.

Primary goals

• Reduce queue pressure on the shared, ASF-wide GitHub-hosted runner pool.
• Improve turnaround time for trusted CI runs by adding capacity Airflow can scale.
• Keep a clear trust boundary so untrusted code never runs on AWS self-hosted infrastructure.

Non-goals

• No one-step migration of all CI to self-hosted runners - this is additive, with GitHub-hosted as default and fallback.
• Windows stays on GitHub-hosted runners in this proposal (there's a single CI job that runs on Windows, apparently).

Why this matters now - qunatitative evidence

Airflow executes ~277 Tests (AMD) runs per day, consuming ~1,780 runner-hours/day. Routing only scheduled canaries and committer-authored runs to AWS-controlled capacity removes approximately 50–55% of this load from the ASF-shared runner pool. The scheduled canary alone accounts for ~11% of total compute and can be migrated immediately without executing PR code. Trusted runs also complete ~25% faster on average while avoiding queue delays caused by ASF-wide concurrency limits.

Scope and Definitions

Trust Classes

• Untrusted run - PRs from contributors without commit access and without explicit committer/PMC trust approval.
• Trusted run - scheduled runs; committer/PMC-owned runs; PRs from non-committers that a committer/PMC has explicitly approved.

Runners in Scope

• In scope:
  • Linux AMD self-hosted runners on AWS.
  • Linux ARM self-hosted runners on AWS.
  • Public GitHub-hosted runners as default/fallback path.
• Out of scope for now:
  • Windows self-hosted runner (we have a single job that runs on Windows).

What changes

Today, Airflow CI relies primarily on public GitHub-hosted runners.

This proposal introduces AWS self-hosted runners for Linux CI workloads and changes how runs are routed:

• build-info  always runs on AWS self-hosted runners.
• Scheduled runs use AWS self-hosted runners by default.
• Pull requests route either to AWS self-hosted runners or to public GitHub-hosted runners, depending on trust and approval rules.
• Public GitHub-hosted runners remain in use for untrusted runs and as a fallback path.

Proposed flow (high-level)

build-info  runs first on the AWS fleet, evaluates trust, and routes every downstream job to the AWS fleet (trusted) or to the public pool (untrusted), with public as the safe fallback.

Key properties

• Ephemeral & one-shot - each job is a fresh, isolated container; nothing to patch or babysit, no state bleed.
• Trusted-only routing - the decision is centralized in build-info / selective_checks, not scattered per workflow.
• In-region everything - images via ECR pull-through, caches in S3: no cross-cloud egress.

Routing Policy

1. Where trust decisions live. The allowlist used both for self-hosted-runner access and for approving untrusted runs is managed in the AWS Terraform infrastructure repository (apache/airflow-ci-infra), not in apache/airflow. It contains only active PMC members and committers. This separation is intentional: trust policy must not be modifiable from the code repository whose jobs are being evaluated.

2. build-info exception. build-info is the only job that always runs on AWS self-hosted runners -- it keeps the queue-advancing decision job on the fast path. It must use the job definition from the target branch (never PR-modifiable logic), must not expose secrets in logs, and must follow security best practices.

3. Default downstream routing. Trusted → AWS (AMD/ARM); untrusted → public. A public-runner override stays available for trusted runs.

4. Approval for non-committers. Expressed with a dedicated trust label (use self-hosted runner); without it, non-committer runs stay on the public runner.

5. Dual-gate enforcement.

   • Gate A (GitHub signal): the trust label is present.
   • Gate B (AWS validation): the label's approver is cross-validated against the AWS allowlist.
   • If either gate fails or is ambiguous → fall back to public runners.

6. Parity. All in-scope self-hosted runners follow the same trust-routing and approval rules.

Infrastructure - defined as code

All AWS-side resources are defined as code in Airflow's CI-infrastructure Terraform repository (the project's existing CI-infra home, e.g. apache/airflow-ci-infra): the fleet, caches, IAM, secrets, and the trust allowlist are reviewable, reproducible, and destroyable from version control -- and, crucially, the allowlist lives outside apache/airflow. The one piece that is not Terraform-managed is the GitHub App connection that links the WORKFLOW_JOB_QUEUED  webhook to CodeBuild: it is an org-level GitHub install and must be set up by the Apache Infra team.

Cost model. GitHub-hosted is "free" but capacity-constrained by the shared pool; the AWS fleet is pay-per-use but elastic and Airflow-controlled. Size from observed CloudWatch utilization; a capped budget bounds spend while the value is proven.

Required code adjustments

Running Airflow's CI on a fresh, root, container-based, in-AWS runner surfaces a handful of general incompatibilities - distinct from the throwaway scaffolding used to pin the PoC to one fork.

These are the items that matter for any self-hosted-on-AWS deployment and are candidates to upstream on their own merits.

Runner routing via selective_checks (the big one)

selective_checks already computes a runners output that workflows consume. Extend it to emit the AWS runner label for trusted events and the public label otherwise, driven by the dual-gate, and wire each runs-on: ${{ inputs.runners }} to it - replacing the fork's hard-coded labels. This is the precondition for any upstream rollout.

Runner-environment compatibility

The agent runs as root in a plain container, not as runner on a GitHub-hosted VM.

• $HOME, not /home/runner - tools installed by uv tool install / pip --user (breeze, prek) land in /root/.local/bin; PATH wiring must use $HOME.
• No systemd, no separate /mnt - GitHub-hosted capacity tricks (/var/lib/docker → /mnt, systemctl) must be guarded behind a self-hosted marker, or they corrupt the Docker daemon.
• Self-installers that probe the env (e.g. pnpm) crash as root → use deterministic installs.
• Environment-specific error strings in tests (EADDRNOTAVAIL vs "Connection refused") need broadened matchers.

Container images & DockerHub limits

Self-hosted AWS runners share NAT egress, so DockerHub pull limits hit immediately - at the host daemon, inside the breeze container, and inside KinD's containerd.

• ECR pull-through cache is the real fix - first pull populates ECR, the rest are in-region and unmetered; solves rate limits and egress at every layer, preferable to buying a higher DockerHub tier.
• Preload pinned test images into KinD (Celery redis broker, postgres, alpine) on the authenticated host, then kind load, so IfNotPresent pods never reach DockerHub.

e2e containers must run non-root

A root host defaulted the Docker-Compose e2e stack to AIRFLOW_UID=0; the image's pip wrapper refuses to run as root (exit 1), and set -e turns the startup dependency install into a crash-loop. Fall back to the image's airflow user (uid 50000) when the host is root, and make bind-mounts writable for it. (Harmless on non-root hosts - upstream-worthy.)

Kubernetes / Docker-in-Docker

• Worker-node kubelet won't start under DinD on cgroup v2 - run a single-node cluster (the control-plane is schedulable with no workers), or arrange host-level cgroup delegation.
• Image namespace must follow the repo - thread the actual github_repository through the K8S build/upload/deploy path so the locally-built image is found/loaded.

E. Secret handling - redact env dumps

Verbose breeze/prek runs print the full environment for reproduce-locally ergonomics; on a public repo that leaks secrets into world-readable logs. Redact values by variable name (TOKEN|SECRET|PASSWORD|CREDENTIAL|ACCESS_KEY|…) before printing. (Upstream-worthy hardening; rotate anything already exposed.)

F. Instance sizing & parallelism

Drive sizing from CloudWatch CPU/memory, not guesswork. pytest-xdist's -n auto = os.cpu_count(), so a bigger box silently spawns more workers and re-creates the same memory pressure - pin PARALLELISM when upsizing. Slower/contended hardware also pushes some serial suites past GitHub-hosted-tuned timeout-minutes; give budgets headroom.

Latent bugs surfaced by the environment (fix upstream regardless)

• prek mypy file-list race — CI mypy hooks wrote a shared files/mypy_files.txt; prek's concurrency scales with CPU count, so on a many-core box parallel hooks clobbered it mid-read and mypy failed with a mangled filename. Use a per-invocation file list.

Success criteria

• Trusted AMD/ARM runs default to AWS self-hosted paths; untrusted stay public unless approved.
• Dual-gate trust checks enforced and auditable; public fallback reliable.
• A maintainer runbook exists for approval, rollback, and incident handling.