General Process

  1. Stop puppet agent
  2. Stop buildbot service
  3. Backup /x1/buildmaster
    1. backups are labelled buildmaster-$buildbot_version-YYYYMMDDNN where NN is an incremented backup id starting with 01. 
  4. Backup asfbuildbot2 mysql database as root
    1. Credentials may be found in /root/.my.cnf
    2. from /x1/; mysqldump -uroot -p --databases asfbuildbot2 > asfbuildbot-$version-YYYYMMDDnn.sql 
  5. Edit https://github.com/apache/infrastructure-p6/blob/production/modules/buildbot_asf/manifests/init.pp
    1. Bump buildbot version
    2. Ensure requisite packages are being installed
    3. Ensure the buildmaster service is set to `ensure => stopped`.
  6. Start puppet to pull in changes on buildmaster node
  7. Stop puppet on buildmaster node
  8. As buildmaster run `buildbot upgrade-master` in the /x1/buildmaster/master1' dir
  9. Fix any configuration errors that occur (See note 1) and re-run the upgrade-master step until it completes successfully.
  10. Start the Buildbot service.
  11. Repeat 1 to 11 as needed for multi step upgrades.
  12. change puppet back to ensure service running
  13. After a few days soak in, remove the local backups

2.4 to 3.2.0 Upgrade Notes

As advised in Buildbot Documentation, we upgraded in steps.
From 2.4 to 2.5.10 to 3.0.3 and finally to 3.2.0
Local backups were taken at the start and end of each upgrade.

(note 1) From 2.4 to 2.5.10 we had to fix builders to be a list. (pelican conf etc)
(note 2) From 2.5.10 to 3.0.3 we had to fix the mail notifiers.
(note 3) From 3.0.3 to 3.2.0 all went well with no configuration changes needed.