Gradle Rollout

Meeting schedule:

• Every Thursday until December 22nd - check your calendar! (or ask for an invite)

Meeting notes:

• Oct 13th:
  • Decision by ASF to do the following type of installation in production:
    • Deployment into EKS based on provided Helm chart
    • Connecting to external RDS Postgres database
    • Using S3 as build scan persistent storage
  • Installation by Gavin and Drew based on the installation tutorials provided by Gradle

• Oct 20th:
  • Etienne is in touch with Greg to come together on an agreement
    
    • No builds to be published to GE 'til we have the written agreement signed + sponsorship clarified
    • We've made all the progress we reasonably can until then
  • Infra has made excellent progress, we have an up and running instance
    • can we switch to postgresql?
    • can switch from nginx to apache for the ingress?
    • once httpd is on there we can use LE + mod_md for certificates
      • G: k8s has extraction for SSL certificates will configure the ingress definition
  • Meet again tomorrow to go over GE configuration in admin section
  • Next steps:
    • Configure admin section of GE
    • Configure valid SSL certificate
    • Use external DB (RDS) instead of internal DB
    • Use S3 for build scan storage
    • Replace nginx
    • Connect example ASF Maven project to GE to verify build scan publishing works
      • See example config of the extensions.xml and the gradle-enterprise.xml files here
      • Run from Jenkins job
      • Run from GitHub actions job
      • Run fro local develper machine

• Oct 21st:
  • Configure admin section of GE
    • Access control: LDAP and email server config will happen post-meeting
    • Access control: Setting up temporary local users for Drew and Gavin will happen post-meeting
    • Access control: Setting up local users for CI-Jenkins and CI-GitHub-Actions will happen post-meeting
  • RDS setup still in progress
• Oct 27th:
  • Gavin was OOO and not able to attend the meeting 
  • Setup of certificate, external DB, LDAP, and email server config in admin section still in progress
  • Question from Etienne:
    • Are the Jenkins servers used by the ASF on a recent version?
• November 3rd:
  • Gavin, Drew, and Rus were in attendance
  • LDAP Bind account has been created, LDAP can now be set up in coming week
    • LDAP has been connected, with no search filter.
    • LDAP users were granted build scan view , build scan publish  and build cache read
  • Additionally, Email will be handled by the LDAP account (once we verify that the user in question has SMTP auth rights)
  • SSL, RDS and S3 have all been set up
• November 10th:
  • Gavin, Drew, Rus, and Etienne were in attendance
  • Discussed how to configure the infrastructute-jenkins project to verify successful publication of build scans
• November 17th:
  • Gavin and Etienne were in attendance
  • Build scan publishing for local build on developer machine confirmed to work by both Gavin and Etienne
  • Build scan publishing for CI build on GitHub Actions confirmed to work by Gavin
  • Next steps:
    • Log in as system user, delete local user for Gavin, Gavin to re-login, and user will be taken from LDAP, then add admin permissions
    • Drew to do the same (local user already deleted by Gavin today)
    • Set up Jenkins job for infrastructure-jenkins project and inject access token of the dedicated, local Jenkins user created already in GE
    • Enable Jenkins Gradle plugin on the Jenkins controller owned by the infrastructure team
    • Have a call with the GE team to configure the Jenkins Gradle plugin to auto-configure the builds with publishing of a build scan
    • Wait for feedback from Greg on the agreement before tackling rollout of the build scan capturing for all CI builds
• November 24th:
  • Gavin, Rus, and Etienne were in attendance
  • Agreement has been signed between Apache (Greg) and Gradle (Rolf)
  • All build scan publishing experiments for GitHub Actions, Jenkins, and Buildbot have been completed successfully
  • Progress on installation automation via Terraform  has been paused while waiting for the agreement, but will be picked up again by Gavin soon
  • Next step:
    • Verify that the GE-enablement in Jenkins (via Gradle Jenkins plugin) works on the Infra controller
    • Decide on what controller to active the GE enablement next
    • Discuss creation of a GE/Apache convention plugin (Gradle) and extension (Maven) to centralize the build-side configuration of GE 
• December 8th:
  • Gavin, Drew, and Etienne were in attendance
  • Next step:
    • ASF to reach out to Apache Beam project and confirm they are ok/happy to get GE build scans for all their CI builds
    • Once ok has been received, enable GE injection on Apache Beam Jenkins controller (similar to what has been done for the Infra team Jenkins controller)
    • Gradle to provide PR to enable GE via configuration in the build, such that they get build scans not only for CI builds but also for all local builds
• December 15th:
  • Drew and Etienne were in attendance
  • Etienne indicated that they may have solved the workspace id problem
    • Root cause fix: Gradle raised an issue on the Apache RAT plugin because it should ignore the .mvn directory including sub-directories as part of the default excludes
    • Workaround: Gradle will add the missing exclude dynamically, available in the upcoming GE Maven extension 1.16.1 (will be released this Friday)
  • Next Steps:
    • ASF to upgrade to GE 2022.4
      • Upgrade Gradle Enterprise server to the latest version in the k8s cluster
    • ASF to upgrade Jenkins Gradle plugin
      • New version will be released early next week 
      • The new version will ship with GE Maven extension 1.16.1 that fixes the RAT plugin issue regarding default excludes
      • For any Apache project using the RAT plugin, this will no longer lead to build failures related to a missing file header in the workspace id file
    • ASF to reach out to the Apache Beam project maintainers
      • Also mention that Gradle had already optimized their build sometimes in the past, via multiple PRs that all got merged back then
      • Gradle team will re-verify that the Apache Beam Gradle build is fully optimized and create PRs if needed (this and next week)

Next steps:

• Set up GE in EKS using internal DB
• Configure admin section of GE
  • Incl. local users setup
  • Incl. LDAP config
  • Incl. mail server config
• Configure valid SSL certificate
• Use external DB (RDS) instead of internal DB
• Use S3 for build scan storage
• Replace nginx - not going to do this.
• Connect example ASF Maven project to GE to verify build scan publishing works
  • See example config of the extensions.xml and the gradle-enterprise.xml files here
  • Run from local developer machine
  • Run from GitHub actions job
  • Run from Jenkins job
• Enable GE auto-configuration on the Jenkins controller owned by the Infrastructure team
• Upgrade to GE 2022.4
• Update the Infra team experimental projects to GE Maven extension 1.16.1 (once available)
• Upgrade to Jenkins Gradle plugin 2.2 (once available)
• Prepare Apache Beam project for GE integration
  • Set up separate remote build cache node, exclusively to be used by the Apache Beam project
  • Set credentials on the new remote build cache node (username/password, not access key)
  • Configure Jenkins with Beam project-specific configuration to inject build cache credentials (username/password) via environment variables (Jenkins functionality described by Gavin)
• Prepare Apache convention plugin/extension
  • Convention plugin/extension makes it super easy and very non-invasive to enable GE on a project, via just a few lines of configuration in the project's build
  • Gradle will provide a PR for a project that builds an Apache GE convention plugin for Apache projects built with Gradle
  • Gradle will provide a PR for a project that builds an Apache GE convention extension for Apache projects built with Maven

Candidate projects:

• Apache Tika
• Apache Commons
• Apache Solr (build optimized by Gradle team some time ago), see CI at https://ci-builds.apache.org/job/Solr/job/Solr-NightlyTests-main/
• Apache Lucene (build optimized by Gradle team some time ago)
• Apache Beam (build optimized by Gradle team some time ago)
• Apache Kafka (build optimized by Gradle team some time ago)
• Apache Groovy (build optimized by Gradle team some time ago)

Implementation Notes

• EKS resources will live in AWS zone us-west-2
  
  • they are a separate group of nodes using k8s with their own resources, own AIM , own security groups
  • we do not want EKS configuration to cross contaminate our big collection of VMs.
    

Initial Local Environment setup and test.

reference document: https://docs.gradle.com/enterprise/tutorials/aws-kubernetes/

Steps	Gavin	Drew	$yourname	Useful Commands	Notes
Install the AWS CLI
Configure the AWS CLI					(Ensure your default region is set to us-west-2) OOC: is there a particular reason we're not using us-west-1? - Drew Foulks
Install kubectl					must use AWS kubectl v1.23
Install eksctl
Install helm
Create a cluster				eksctl create cluster --name gradle-test-$yourname	Create a TEST cluster! (not gradle-enterprise)
Retrieve Nodes Info				kubectl get nodes
Delete test Cluster				eksctl delete cluster --name gradle-test-$yourname
Local Environment Setup and test complete.

Create production Cluster Environment

Steps	Gavin	Drew	$yourname	Useful Commands	Notes
Create a cluster		N/A	N/A	eksctl create cluster --name gradle-enterprise	Done - Is located in us-west-2 (Oregon)
Expand from 2 node to 3 node Cluster		N/A	N/A	eksctl scale nodegroup \ --cluster gradle-enterprise \ --nodes 3 \ --nodes-max 3 \ - -name $(aws eks list-nodegroups --cluster-name gradle-enterprise --query 'nodegroups[0]' --output text)	Done (Ensure your default region is set to us-west-2)
Install Nginx Ingress Controller		N/A	N/A	kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.0/deploy/static/provider/aws/deploy.yaml	Done - What about an apache2 Ingress instead?
Check AWS NLB		N/A	N/A	kubectl get \ -n ingress-nginx service ingress-nginx-controller \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'	ad70e0387e8e64aa78d35a94d34445b3-fedf4a44e4c40f6a.elb.us-west-2.amazonaws.com
Create a high performance storage class		N/A	N/A
Configure the hostname		N/A	N/A	edited our internal inventory.conf to add the CNAME, confirmed via dig.	ge.apache.org
Installing Gradle Enterprise		N/A	N/A	Created a values.yaml (also saved the gp3.yaml helm repo add gradle https://helm.gradle.com/ helm repo update gradle helm install --create-namespace --namespace gradle-enterprise ge gradle/gradle-enterprise --values values.yaml --set-file global.license.file=gradle-enterprise.license	Output to helm install Expand source NAME: ge LAST DEPLOYED: Thu Oct 20 10:18:24 2022 NAMESPACE: gradle-enterprise STATUS: deployed REVISION: 1 TEST SUITE: None Where to put these config files? In puppet? ge.apache.org is running Need to sort out an SSL certificate
Check Pods Status		N/A	N/A	kubectl -n gradle-enterprise get pods	check pods status Expand source NAME READY STATUS RESTARTS AGE gradle-build-cache-node-5f68b4fc6d-jktnr 1/1 Running 0 14m gradle-database-8674dfd4fb-2vhpd 2/2 Running 0 14m gradle-enterprise-app-0 1/1 Running 0 14m gradle-enterprise-operator-6768786566-28cls 1/1 Running 0 14m gradle-keycloak-0 1/1 Running 0 14m gradle-metrics-75d4b845dc-g5rk9 1/1 Running 0 14m gradle-proxy-0 1/1 Running 0 14m gradle-test-distribution-broker-78764546df-9p4h2 1/1 Running 0 14m