TeamCity

JetBrains TeamCity is the main CI/CD service for building, testing and releasing Apache Ignite and it's components.

TeamCity 1

URL: https://ci.ignite.apache.org/

TeamCity 2

URL: https://ci2.ignite.apache.org/

TC1 → TC2 syncing

Current scheme of keeping consistency between 2 instances of TeamCity instances.

Build configuration settings

  1. TC1 is set to keep it's settings via enabled Versioned Settings option.
    Storage format is XML, editing in UI is allowed (for continuous usual usage of the system by users).
  2. The code currently is stored in separate GitHub repository. Main branch is tc1.
  3. Each edit in TC1 results in commit to repository with corresponding change set.
  4. After each commit the build configuration in TC2 [Tools] / TC1 to TC2 sync is triggered, which takes code from tc1, modifies it with tc1-to-tc2-sync.sh script and pushes updated code to tc2 branch.
    This build exists on TC1 too for editing, but VCS trigger is disabled in TC1 and enabled in TC2, TC2 runs this build.
  5. TC2 is set to keep it's settings via enabled Versioned Settings option too, but from different branch as main (tc2).
    Storage format is XML too, editing in UI is not allowed (to keep TC2 as "replica only" mode)
  6. TC2's configuration is re-rendered according changes from tc2 branch after each run of sync build.

Secrets

  1. After 1st sync it is required to enter secrets value represented by Tokens (in Versions Settings option).
    Current list of secrets:
  2. Instead of entering values in Tokens, it is possible to copy values from sources TeamCity instances as follows:

    1. Search for credentialsJSON settings in repository with settings

      grep -R 'credentialsJSON:' .teamcity/ | \
  sed -r 's|(.):.param name="(.)".value="(.)" .|\1 :: \2=\3|' | \
  sed -r 's| spec=.||'

    2. Search for occurrences of UUIDs in <source-teamcity-host>:/<teamcity-data-dir>/config/project

      cd /<source-teamcity-data-dir>/config/project
find . -name credentials.json | while read file; do
    echo "${file}"
    grep --color -E '(<UUID-1>|UUID-2|...|<UUID-n>)|$' "${file}"
done
    3. Remove not matched UUIDs (they will not be colored) by editing files manually.

    4. Copy updated credentials.json files to some folder

      cd /<source-teamcity-data-dir>/config/project
mkdir -pv /tmp/credentialsJSON
find . -name credentials.json | while read file; do
    cp -v --parents "${file}" /tmp/credentialsJSON
done

    5. Pack secrets in archive

      cd /tmp/credentialsJSON
zip -r ../credentialsJSON.zip *
    6. Copy archive to target TeamCity instance <source-teamcity-host>:/<teamcity-data-dir>/config/project

    7. Find local secrets and remove them

      cd /<target-teamcity-data-dir>/config/project
find . -name credentials.json -exec rm -v {} \;
unzip credentialsJSON.zip *

    8. Restart TeamCity instance

      cd /<target-teamcity-server-dir>
bash bin/teamcity-server.sh stop
...
bash bin/teamcity-server.sh start
    9. Check /<target-teamcity-server-dir>/logs/teamcity-server.log that TeamCity instance started correctly

Users / Groups / Roles

Synchronization of users, groups and roles requires separate export and import procedure.

  1. Dump specific tables from source TeamCity database

    mysqldump --no-tablespaces -u teamcity -p teamcity ids_group \
                                                   ids_group_entity_id \
                                                   project \
                                                   project_mapping \
                                                   user_blocks \
                                                   user_build_parameters \
                                                   user_build_types_order \
                                                   usergroup_notification_data \
                                                   usergroup_notification_events \
                                                   usergroup_roles \
                                                   usergroups \
                                                   usergroup_subgroups \
                                                   usergroup_users \
                                                   usergroup_watch_type \
                                                   user_notification_data \
                                                   user_notification_events \
                                                   user_projects_visibility \
                                                   user_property \
                                                   user_roles \
                                                   users \
                                                   user_watch_type \
                                                   vcs_username > teamcity.sql
  2. Copy /<source-teamcity-data-dir>/config/roles-config.xml

  3. Pack dump and configuration fail into archive

    zip -r users-groups-roles.zip teamcity.sql roles-config.xml
  4. Copy archive to target TeamCity instance <source-teamcity-host>

  5. Unpack archive

    unzip users-groups-roles.zip
  6. Place roles-config.xml to /<target-teamcity-data-dir>/config/roles-config.xml

  7. Restore specific tables (TeamCity and/or DB service stop is not required)

    mysql -u teamcity -p teamcity < teamcity.sql
  8. Restart TC (restart is required to re-read some DB data which is not updated on-the-fly).

    1. Stop queue

      curl -s \
     -u "<tc_username>:<tc_password>" \
     -X POST \
     "https://<tc_url>/queueStartStop.html?stateChangeReason=TeamCity+daily+maintenance&newQueueState=false"

    2. Wait for non-composite running builds to finish

      while true; do
    runningBuildsCount="$(curl -s \
                               -u "<tc_username>:<tc_password>" \
                               -X GET \
                               "https://<tc_url>/app/rest/builds/?locator=state:running,composite:false" | \
                            xmllint --format \
                                    --xpath "string(/*[local-name()='builds']/@count)" -)"
    if [ "$runningBuildsCount" -gt 0 ]; then
	echo "Running builds left: ${runningBuildsCount}"
	sleep 60
    else
        break
    fi
done

    3. Restart server

      curl -s \
     -u "<tc_username>:<tc_password>" \
     -X POST \
     "https://<tc_url>/admin/serverRestart.html?start=1" 1>/dev/null 2>&1
sleep 10

    4. Wait for TeamCity to boot

      while true; do
    returnCode="$(curl -s \
                       -I \
                       -u "<tc_username>:<tc_password>" \
                       -o /dev/null \
	                   -w "%{http_code}" \
	                   -X GET \
	                   "https://<tc_url>/app/rest/server")"
    if [ "${returnCode}" != "200" ]; then
	echo "Current TeamCity HTTP status: ${returnCode}"
        sleep 60
    else
	break
    fi
done

    5. Resume queue

      curl -s \
     -u "<tc_username>:<tc_password>" \
     -X POST \
     "https://<tc_url>/queueStartStop.html?stateChangeReason=&newQueueState=true"

Backup

TeamCity server has embedded backup tool which can backup database, project files and settings and build logs and artifacts.

Backup can be done in two ways:

  1. Via UI: https://ci.ignite.apache.org/admin/admin.html?item=backup

  2. Via REST API:

    1. (optional) Remove old backups

      find "<teamcity-data_dir>/backup/" -type f -ctime +<days> -exec rm -rfv {} \; | sed -r 's|^| - |'

    2. Initiate backup

      curl -s \
     -u "<tc_username>:<tc_password>" \
     -X POST \
     "https://<tc_url>/app/rest/server/backup?includeConfigs=true&includeBuildLogs=false&fileName=<tc_backup_name>"

Travis CI

Backlog issues

  1. TC: Introduce automated instructions for preparing Linux and Windows agents.
  2. ✅ TC: Remove obsolete secret tokens.
  3. TC: Introduce Kotlin DSL based settings (instead of XML).
  • No labels

Apache Ignite