Page History

...

• apache-maven:3.5.4
• Docker:1820.0910.423
• Go:1.16
• kubectl
• open-jdk-11
• open-jdk-1.8
• Pip:8.1.1
• Python:2.7.12
• Python:3.5.2
• Python:3.6.13
• Python:3.7.10
• Python:3.8.9
• Python:3.9.4
• Python:3.11.2

• Virtualenv:16.4.3

How to install and upgrade software on Jenkins workers

...

1. Go to the page for the Jenkins agent you want to update (for example) and click "Mark this node temporarily offline", leaving a reason such as "Updating X dependency." 
2. Wait until there are no more tests running in that agent (under "Build Executor Status" on the left of the page). You can also prioritize the update on agents currently not running any builds.
3. Restart the VM with a new disk:

   
   

   Code Block
   language bash
   OLD_IMAGE_DATE=2020072920230217 NEW_IMAGE_DATE=20211011YYYYMMDD PROJECT=apache-beam-testing ZONE=us-central1-b # Select the workers currently offline and not processing any builds, # that you want to upgrade. # For example: for CI_INSTANCE_NUMBER {1..4} {10..12} 16; do for CI_INSTANCE_NUMBER in {1..4}; do CI_INSTANCE=apache-ci-beam-jenkins-${CI_INSTANCE_NUMBER} gcloud compute instances stop $CI_INSTANCE --project $PROJECT --zone $ZONE gcloud compute instances detach-disk $CI_INSTANCE --project $PROJECT --zone $ZONE --disk ${CI_INSTANCE}-${OLD_IMAGE_DATE} gcloud compute disks create ${CI_INSTANCE}-${NEW_IMAGE_DATE} --project $PROJECT --zone $ZONE --image=jenkins-worker-boot-image-${NEW_IMAGE_DATE} gcloud compute instances attach-disk $CI_INSTANCE --project $PROJECT --zone $ZONE --disk ${CI_INSTANCE}-${NEW_IMAGE_DATE} --device-name=${CI_INSTANCE}-${NEW_IMAGE_DATE} --boot gcloud compute instances start ${CI_INSTANCE} --project $PROJECT --zone $ZONE done

   
   

   You could also do these steps via Cloud Console instead of CLI:

   1. Stop the VM, then click edit.

   2. Delete the existing boot disk by clicking the 'X'.

   3. Create a new boot disk by using the newly generated image.

   4. Name disk as 'apache-ci-beam-jenkins-[1,16]-YYYYMMDD'.

   5. Make sure the boot disk 'Model' is 'Boot, read/write'.

   6. Start the VM.

4. Mark the node in Jenkins as online again and click "Launch agent" button if there is one.
5. Verify that inventory job for this node did not start while the worker was offline, sample build for node #10: https://ci-beam.apache.org/job/beam_Inventory_apache-beam-jenkins-10. If there is a stuck execution of the inventory job, cancel it manually. TODO (BEAM-13666): we should cancel inventory stuck jobs automatically. 
6. (Optional) If this is the first agent being updated and you would like to test your changes by running a particular Jenkins job, do the following:
   1. Go to the page for the job you want to run (example).
   2. Click "Configure" on the left menu.
   3. Find the checkmark "Restrict where this project can be run" and change the restriction from "beam" to the specific name of the agent (ex. "apache-beam-jenkins-1").
   4. Save and apply that change.
   5. Back on the page for the job, click "Build with Parameters" on the left menu.
   6. Run the build on "master".
   7. Once you're done checking the results, change the restriction for the job back to "beam".
7. Repeat for each worker. You can also do it in batches of workers, but avoid bringing all of the workers offline at once.  

8. After confirming that update was successful, we can go to Cloud Console Disks and clean up deprecated Jenkins worker disks, or do this via cli:

   Code Block
   language bash
   OLD_IMAGE_DATE=2020072920230217 PROJECT=apache-beam-testing ZONE=us-central1-b for CI_INSTANCE_NUMBER in {1..16}; do CI_INSTANCE=apache-ci-beam-jenkins-${CI_INSTANCE_NUMBER} gcloud compute disks delete ${CI_INSTANCE}-${OLD_IMAGE_DATE} --project $PROJECT --zone $ZONE --quiet done