NOTE: Many CI jobs have moved to use GitHub Actions, and all new jobs should be added in Actions. For more info on GitHub Actions in Beam, see https://github.com/apache/beam/blob/master/.github/workflows/README.md

Highlights

The location ./test-infra/jenkins contains the defined Jenkins jobs are defined in  These jobs are written using the Job DSL using Apache Groovy. Job definitions should be a simple as possible and ideally identify a single Gradle target to execute.

Testing Changes

The following are some Testing Changes tips that could help you:

• Use Local Dockerized Jenkins to test changes locally without breaking the project’s job definitions. 
• Test a PR that includes Jenkins changes, but it is potentially destructive because the job definitions are a shared resource.
• To update the job definitions on a PR we need to run the Seed Job. This will cause the job definitions to be read and parsed. Running a seed job is a high privilege workflow, so it can only be run by committers.
  
  • To run a seed job over a committer PR, one needs only to type "Run Seed Job" into the PR in Github.
  • To run a seed job over a non-commiter PR, one needs to follow a hacky workaround: https://drive.google.com/file/d/1Yk4TIF3Qw_ur4P8fTx1I1469TDFdCHHq/view
    
    • Another option is to have a committer clone the PR and type "Run Seed Job"

Triggering jobs

Beam committers can trigger a job with the Jenkins UI. Non-committers can trigger a job if there is a trigger phrase.

• To trigger a job on Jenkins UI, log in on https://ci-beam.apache.org/, find the job page with search, select Build with Parameters page from the menu and click Build (you may also specify a build parameter, for example a commit sha1 value).
• Alternatively, to reliably trigger at a specific PR, select Configure and add ghprbPullId  as a String parameter. Then set the default for sha1 to origin/pr/${ghprbPullId}/merge . Save.  Then select Build with Parameters, and put in the PR number.

Job suffixes

Each post-commit and pre-commit job file defines several jobs with different suffixes.

• For pre-commits, there are _Commit, _Phrase, and _Cron suffixes.
• The _Commit job happens with every push to a pull request.
• The _Phrase happens when the trigger phrase is entered as a comment in the pre-commit.
• The _Cron pre-commit happens post-commit on the master branch as a signal of whether the pre-commit would pass without any changes.

Job labels

Consider the following tips regarding Jenkins-jobs-related tags:

• Most Beam Jenkins jobs specify the label beam, which uses beam executors 1-15.
• Performance test jobs specify the label beam-perf, which uses beam executors 16.
  • Used for tests that run locally on the Jenkins worker such as Direct Runner. Configured to run a one job at a time to reduce the noisy neighbor problem
• Website publishing job specifies the label git-websites, which allows publishing generated documentation to the asf-site branch.

How to access a test history

To access a test history:

1. Look at this URL:
   https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/lastCompletedBuild/testReport/junit/org.apache.beam.runners.fnexecution.data/GrpcDataServiceTest/testMessageReceivedBySingleClientWhenThereAreMultipleClients/history/
2. Go to failed job -> test result -> navigate to failed test -> history

How to delete jobs that are no longer needed

To delete jobs that are no longer needed in Jenkins: 

1. Remove the `.groovy` file associated with the job. For more information, see 

   Removes Python 2 and Python 3.5 Postcommit jobs. #12891.

2. Delete the job history from Jenkins (committers only, if you are not a committer please ask the reviewer of your PR to help with this).
   1. Open https://ci-beam.apache.org/ and log in with your Apache Credentials.
   2. Select the job you want to remove.
   3. Click Delete project in the left side panel.

Jenkins Infrastructure Setup

16 GCE instances support Jenkins' continuous integrating jobs. The instance group 'apache-ci-beam-jenkins-nodes' manages the compute instances in the Google Cloud project 'apache-beam-testing'. Each instance has 16 CPUs and 104GB of memory. We used to run Jenkins nodes using Apache Jenkins Build cluster (https://builds.apache.org/), but we have since migrated to our dedicated cluster: https://ci-beam.apache.org/, and there is an old group of workers 'apache-beam-jenkins-jnlp' that is now stopped but not yet deleted.

Current installations

• apache-maven:3.5.4
• Docker:20.10.23
• Go:1.16
• kubectl
• open-jdk-11
• open-jdk-1.8
• Pip:8.1.1
• Python:2.7.12
• Python:3.5.2
• Python:3.6.13
• Python:3.7.10
• Python:3.8.9
• Python:3.9.4
• Python:3.11.2

• Virtualenv:16.4.3

How to install and upgrade software on Jenkins workers

All software updates must be performed and verified in a temporary experimental compute instance. To install or upgrade tools for Jenkins instances:

1. Create a compute instance with the most recently created boot image. 

   gcloud compute instances create $USER-jenkins-upgrade --project=apache-beam-testing --zone us-central1-b --image-family=jenkins-worker-boot-image --machine-type=n1-highmem-16

   You can create the instance through VM Instances in Cloud Console or by running the gcloud command.

2. Connect using ssh to the instance via Cloud Console or gcloud command.

   gcloud compute --project "apache-beam-testing" ssh --zone "us-central1-b" $USER-jenkins-upgrade

3. Login as 'jenkins' user.

   sudo su jenkins

4. Install or upgrade the required tools.

5. Verify the tool by running corresponding Beam tests.

6. Verify backwards compatibility by running Beam PostCommits against all SDKs.

   • Clone Beam repository from Git:

     git clone https://github.com/apache/beam.git && cd beam

   • Run your tests by Gradle wrapper. For example: 

     ./gradlew :PythonPostCommit --continue --<args>

   • Remove beam repository after verification.

7. Create a new image from the your instance and name it as jenkins-worker-boot-image-YYYYMMDD. Put the image in the family of 'jenkins-worker-boot-image', and add a description of your changes.

   gcloud compute instances stop $USER-jenkins-upgrade --project "apache-beam-testing" --zone "us-central1-b" 
   
   gcloud  compute images create \
     --project apache-beam-testing \
     --source-disk $USER-jenkins-upgrade \
     --source-disk-zone us-central1-b \
     jenkins-worker-boot-image-<YYYYMMDD> \
     --family jenkins-worker-boot-image \
     --description "Updated XYZ, see BEAM-1234." 

To reboot Jenkins executors with a new image, do the following:

1. Go to the page for the Jenkins agent you want to update (for example) and click "Mark this node temporarily offline", leaving a reason such as "Updating X dependency." 
2. Wait until there are no more tests running in that agent (under "Build Executor Status" on the left of the page). You can also prioritize the update on agents currently not running any builds.
3. Restart the VM with a new disk:

   OLD_IMAGE_DATE=20231110
   NEW_IMAGE_DATE=YYYYMMDD
   PROJECT=apache-beam-testing
   ZONE=us-central1-b
   
   # Select the workers currently offline and not processing any builds,
   # that you want to upgrade. 
   # For example: for CI_INSTANCE_NUMBER {1..4} {10..12} 16; do 
   
   for CI_INSTANCE_NUMBER in {1..4}; do
     CI_INSTANCE=apache-ci-beam-jenkins-${CI_INSTANCE_NUMBER}
     gcloud compute instances stop $CI_INSTANCE --project $PROJECT --zone $ZONE
     gcloud compute instances detach-disk $CI_INSTANCE --project $PROJECT --zone $ZONE --disk ${CI_INSTANCE}-${OLD_IMAGE_DATE}
     gcloud compute disks create ${CI_INSTANCE}-${NEW_IMAGE_DATE}  --project $PROJECT --zone $ZONE --image=jenkins-worker-boot-image-${NEW_IMAGE_DATE}
     gcloud compute instances   attach-disk $CI_INSTANCE   --project $PROJECT --zone $ZONE  --disk ${CI_INSTANCE}-${NEW_IMAGE_DATE}  --device-name=${CI_INSTANCE}-${NEW_IMAGE_DATE}  --boot
     gcloud compute instances start ${CI_INSTANCE}   --project $PROJECT --zone $ZONE
   done  

   You could also do these steps via Cloud Console instead of CLI:

   1. Stop the VM, then click edit.

   2. Delete the existing boot disk by clicking the 'X'.

   3. Create a new boot disk by using the newly generated image.

   4. Name disk as 'apache-ci-beam-jenkins-[1,16]-YYYYMMDD'.

   5. Make sure the boot disk 'Model' is 'Boot, read/write'.

   6. Start the VM.

4. Mark the node in Jenkins as online again and click "Launch agent" button if there is one.
5. Verify that inventory job for this node did not start while the worker was offline, sample build for node #10: https://ci-beam.apache.org/job/beam_Inventory_apache-beam-jenkins-10. If there is a stuck execution of the inventory job, cancel it manually. TODO (BEAM-13666): we should cancel inventory stuck jobs automatically. 
6. (Optional) If this is the first agent being updated and you would like to test your changes by running a particular Jenkins job, do the following:
   1. Go to the page for the job you want to run (example).
   2. Click "Configure" on the left menu.
   3. Find the checkmark "Restrict where this project can be run" and change the restriction from "beam" to the specific name of the agent (ex. "apache-beam-jenkins-1").
   4. Save and apply that change.
   5. Back on the page for the job, click "Build with Parameters" on the left menu.
   6. Run the build on "master".
   7. Once you're done checking the results, change the restriction for the job back to "beam".
7. Repeat for each worker. You can also do it in batches of workers, but avoid bringing all of the workers offline at once.  

8. After confirming that update was successful, we can go to Cloud Console Disks and clean up deprecated Jenkins worker disks, or do this via cli:

   OLD_IMAGE_DATE=20231110
   PROJECT=apache-beam-testing
   ZONE=us-central1-b 
   
   for CI_INSTANCE_NUMBER in {1..16}; do
     CI_INSTANCE=apache-ci-beam-jenkins-${CI_INSTANCE_NUMBER}
     gcloud compute disks delete ${CI_INSTANCE}-${OLD_IMAGE_DATE}  --project $PROJECT --zone $ZONE  --quiet 
   done