NameAssociation
ASF Infra

ASF Infra

Gradle

Gradle

Vincent Mayers 

Gradle
Rus HartGradle

Meeting schedule:

  • Every Thursday until December 22nd - check your calendar! (or ask for an invite)

Meeting notes:

  • Oct 13th:
    • Decision by ASF to do the following type of installation in production:
      • Deployment into EKS based on provided Helm chart
      • Connecting to external RDS Postgres database
      • Using S3 as build scan persistent storage
    • Installation by Gavin and Drew based on the installation tutorials provided by Gradle
  • Oct 20th:
    • Etienne is in touch with Greg to come together on an agreement
      • No builds to be published to GE 'til we have the written agreement signed + sponsorship clarified
      • We've made all the progress we reasonably can until then
    • Infra has made excellent progress, we have an up and running instance
      • can we switch to postgresql?
      • can switch from nginx to apache for the ingress?
      • once httpd is on there we can use LE + mod_md for certificates
        • G: k8s has extraction for SSL certificates will configure the ingress definition
    • Meet again tomorrow to go over GE configuration in admin section
    • Next steps:
      • Configure admin section of GE
      • Configure valid SSL certificate
      • Use external DB (RDS) instead of internal DB
      • Use S3 for build scan storage
      • Replace nginx
      • Connect example ASF Maven project to GE to verify build scan publishing works
        • See example config of the extensions.xml and the gradle-enterprise.xml files here
        • Run from Jenkins job
        • Run from GitHub actions job
        • Run fro local develper machine
  • Oct 21st:
    • Configure admin section of GE
      • Access control: LDAP and email server config will happen post-meeting
      • Access control: Setting up temporary local users for Drew and Gavin will happen post-meeting
      • Access control: Setting up local users for CI-Jenkins and CI-GitHub-Actions will happen post-meeting
    • RDS setup still in progress
  • Oct 27th:
    • Gavin was OOO and not able to attend the meeting 
    • Setup of certificate, external DB, LDAP, and email server config in admin section still in progress
    • Question from Etienne:
      • Are the Jenkins servers used by the ASF on a recent version?
  • November 3rd:
    • Gavin, Drew, and Rus were in attendance
    • LDAP Bind account has been created, LDAP can now be set up in coming week
      • LDAP has been connected, with no search filter.
      • LDAP users were granted build scan view , build scan publish  and build cache read
    • Additionally, Email will be handled by the LDAP account (once we verify that the user in question has SMTP auth rights)
    • SSL, RDS and S3 have all been set up
  • November 10th:
    • Gavin, Drew, Rus, and Etienne were in attendance
    • Discussed how to configure the infrastructute-jenkins project to verify successful publication of build scans
  • November 17th:
    • Gavin and Etienne were in attendance
    • Build scan publishing for local build on developer machine confirmed to work by both Gavin and Etienne
    • Build scan publishing for CI build on GitHub Actions confirmed to work by Gavin
    • Next steps:
      • Log in as system user, delete local user for Gavin, Gavin to re-login, and user will be taken from LDAP, then add admin permissions
      • Drew to do the same (local user already deleted by Gavin today)
      • Set up Jenkins job for infrastructure-jenkins project and inject access token of the dedicated, local Jenkins user created already in GE
      • Enable Jenkins Gradle plugin on the Jenkins controller owned by the infrastructure team
      • Have a call with the GE team to configure the Jenkins Gradle plugin to auto-configure the builds with publishing of a build scan
      • Wait for feedback from Greg on the agreement before tackling rollout of the build scan capturing for all CI builds
  • November 24th:
    • Gavin, Rus, and Etienne were in attendance
    • Agreement has been signed between Apache (Greg) and Gradle (Rolf)
    • All build scan publishing experiments for GitHub Actions, Jenkins, and Buildbot have been completed successfully
    • Progress on installation automation via Terraform  has been paused while waiting for the agreement, but will be picked up again by Gavin soon
    • Next step:
      • Verify that the GE-enablement in Jenkins (via Gradle Jenkins plugin) works on the Infra controller
      • Decide on what controller to active the GE enablement next
      • Discuss creation of a GE/Apache convention plugin (Gradle) and extension (Maven) to centralize the build-side configuration of GE 
  • December 8th:
    • Gavin, Drew, and Etienne were in attendance
    • Next step:
      • ASF to reach out to Apache Beam project and confirm they are ok/happy to get GE build scans for all their CI builds
      • Once ok has been received, enable GE injection on Apache Beam Jenkins controller (similar to what has been done for the Infra team Jenkins controller)
      • Gradle to provide PR to enable GE via configuration in the build, such that they get build scans not only for CI builds but also for all local builds
  • December 15th:
    • Drew and Etienne were in attendance
    • Etienne indicated that they may have solved the workspace id problem
      • Root cause fix: Gradle raised an issue on the Apache RAT plugin because it should ignore the .mvn directory including sub-directories as part of the default excludes
      • Workaround: Gradle will add the missing exclude dynamically, available in the upcoming GE Maven extension 1.16.1 (will be released this Friday)
    • Next Steps:
      • ASF to upgrade to GE 2022.4
        • Upgrade Gradle Enterprise server to the latest version in the k8s cluster
      • ASF to upgrade Jenkins Gradle plugin
        • New version will be released early next week 
        • The new version will ship with GE Maven extension 1.16.1 that fixes the RAT plugin issue regarding default excludes
        • For any Apache project using the RAT plugin, this will no longer lead to build failures related to a missing file header in the workspace id file
      • ASF to reach out to the Apache Beam project maintainers
        • Also mention that Gradle had already optimized their build sometimes in the past, via multiple PRs that all got merged back then
        • Gradle team will re-verify that the Apache Beam Gradle build is fully optimized and create PRs if needed (this and next week)

Next steps:

  • Set up GE in EKS using internal DB
  • Configure admin section of GE
    • Incl. local users setup
    • Incl. LDAP config
    • Incl. mail server config
  • Configure valid SSL certificate
  • Use external DB (RDS) instead of internal DB
  • Use S3 for build scan storage
  • Replace nginx - not going to do this.
  • Connect example ASF Maven project to GE to verify build scan publishing works
    • See example config of the extensions.xml and the gradle-enterprise.xml files here
    • Run from local developer machine
    • Run from GitHub actions job
    • Run from Jenkins job
  • Enable GE auto-configuration on the Jenkins controller owned by the Infrastructure team
  • Upgrade to GE 2022.4
  • Update the Infra team experimental projects to GE Maven extension 1.16.1 (once available)
  • Upgrade to Jenkins Gradle plugin 2.2 (once available)
  • Prepare Apache Beam project for GE integration
    • Set up separate remote build cache node, exclusively to be used by the Apache Beam project
    • Set credentials on the new remote build cache node (username/password, not access key)
    • Configure Jenkins with Beam project-specific configuration to inject build cache credentials (username/password) via environment variables (Jenkins functionality described by Gavin)
  • Prepare Apache convention plugin/extension
    • Convention plugin/extension makes it super easy and very non-invasive to enable GE on a project, via just a few lines of configuration in the project's build
    • Gradle will provide a PR for a project that builds an Apache GE convention plugin for Apache projects built with Gradle
    • Gradle will provide a PR for a project that builds an Apache GE convention extension for Apache projects built with Maven

Candidate projects:

  • Apache Tika
  • Apache Commons
  • Apache Solr (build optimized by Gradle team some time ago), see CI at https://ci-builds.apache.org/job/Solr/job/Solr-NightlyTests-main/
  • Apache Lucene (build optimized by Gradle team some time ago)
  • Apache Beam (build optimized by Gradle team some time ago)
  • Apache Kafka (build optimized by Gradle team some time ago)
  • Apache Groovy (build optimized by Gradle team some time ago)

Implementation Notes

  • EKS resources will live in AWS zone us-west-2
    • they are a separate group of nodes using k8s with their own resources, own AIM , own security groups
    • we do not want EKS configuration to cross contaminate our big collection of VMs.

Initial Local Environment setup and test.

reference document: https://docs.gradle.com/enterprise/tutorials/aws-kubernetes/

StepsGavinDrew$yournameUseful CommandsNotes

Install the AWS CLI

(tick)(tick)


Configure the AWS CLI

(tick)(tick)

(Ensure your default region is set to us-west-2)

OOC: is there a particular reason we're not using us-west-1? - Drew Foulks 

Install kubectl

(tick)(tick)

must use AWS kubectl v1.23

Install eksctl

(tick)(tick)


 Install helm

(tick)(tick)


Create a cluster

(tick)(tick)
eksctl create cluster --name gradle-test-$yournameCreate a TEST cluster! (not gradle-enterprise)
Retrieve Nodes Info(tick)(tick)
kubectl get nodes
Delete test Cluster(tick)(tick)
eksctl delete cluster --name gradle-test-$yourname
Local Environment Setup and test complete.

Create production Cluster Environment

StepsGavinDrew$yournameUseful CommandsNotes

Create a cluster

(tick)N/AN/Aeksctl create cluster --name gradle-enterpriseDone - Is located in us-west-2 (Oregon)
Expand from 2 node to 3 node
Cluster
(tick)N/AN/Aeksctl scale nodegroup \
--cluster gradle-enterprise \
--nodes 3 \
--nodes-max 3 \ -
-name $(aws eks list-nodegroups --cluster-name gradle-enterprise --query 'nodegroups[0]' --output text)
Done
(Ensure your default region is set to us-west-2)
Install Nginx Ingress Controller(tick)N/AN/A
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.0/deploy/static/provider/aws/deploy.yaml
Done - What about an apache2 Ingress instead?
Check AWS NLB(tick)N/AN/Akubectl get \ -n ingress-nginx service ingress-nginx-controller \ -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'ad70e0387e8e64aa78d35a94d34445b3-fedf4a44e4c40f6a.elb.us-west-2.amazonaws.com

Create a high performance storage class

(tick)N/AN/A

Configure the hostname

(tick)N/AN/Aedited our internal inventory.conf to add the CNAME, confirmed via dig.

ge.apache.org

Installing Gradle Enterprise

(tick)N/AN/A
  • Created a values.yaml (also saved the gp3.yaml
  • helm repo add gradle https://helm.gradle.com/
  • helm repo update gradle
  • helm install --create-namespace --namespace gradle-enterprise ge gradle/gradle-enterprise --values values.yaml --set-file global.license.file=gradle-enterprise.license
Output to helm install
NAME: ge
LAST DEPLOYED: Thu Oct 20 10:18:24 2022
NAMESPACE: gradle-enterprise
STATUS: deployed
REVISION: 1
TEST SUITE: None

Where to put these config files?
In puppet?

ge.apache.org is running

Need to sort out an SSL certificate

Check Pods Status(tick)N/AN/Akubectl -n gradle-enterprise get pods
check pods status
NAME                                               READY   STATUS    RESTARTS   AGE
gradle-build-cache-node-5f68b4fc6d-jktnr           1/1     Running   0          14m
gradle-database-8674dfd4fb-2vhpd                   2/2     Running   0          14m
gradle-enterprise-app-0                            1/1     Running   0          14m
gradle-enterprise-operator-6768786566-28cls        1/1     Running   0          14m
gradle-keycloak-0                                  1/1     Running   0          14m
gradle-metrics-75d4b845dc-g5rk9                    1/1     Running   0          14m
gradle-proxy-0                                     1/1     Running   0          14m
gradle-test-distribution-broker-78764546df-9p4h2   1/1     Running   0          14m