You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

The Apache Beam project tracks a set of community and project health metrics, with targets to ensure a healthy, sustainable community (ex: test timing and reliability, pull request latency). The dashboards are available at: https://s.apache.org/beam-community-metrics.

Resources

Updating and Deploying

Instructions for developing and making updates to the metrics pipelines and dashboards are checked-in: README.md.

Local development and deployment is managed by docker-compose, while the production environment is deployed via Kubernetes and Cloud SQL on Google Cloud.

Analytics Database

Metric data is imported to and queried from a Postgresql analytics database. Our database is deployed on Google Cloud SQL,  under project apache-beam-testing, instance beammetrics. The database tables are automatically maintained by the import scripts.

Migrating Jenkins Job History

In some cases, it will be necessary to migrate historical data. For example, jobs can be renamed in Jenkins. Jenkins will correctly migrate history to refer to the new job name, but already imported job runs need to be migrated in the postgres database. Follow the steps below in order to migrate data:

  1. Backup the database. Login to the Database Backups page for beammetrics and create a backup in case anything goes wrong.

  2. Connect to the database. From a console, login using gcloud command:
    gcloud sql beta connect beammetrics --database=beammetrics --user=<your_username>

  3. Verify the data to migrate. Craft a SELECT query for the data you plan to update. For example:

    SELECT *
    FROM jenkins_builds jb
    WHERE jb.job_name = 'beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle'
      AND NOT EXISTS (
        SELECT *
        FROM jenkins_builds
        WHERE job_name = 'beam_PostCommit_Java_ValidatesRunner_Gearpump'
          AND build_id=jb.build_id
      );

    Note that the import scripts will import recent history with the migrated job name from Jenkins; the NOT EXISTS clause above ensures that the duplicate history is not migrated from the previous name (it will be removed later). The UPDATE command will fail without this condition.

  4. Update the data. After validating the query targets the intended rows, modify it to update the necessary fields:

    UPDATE jenkins_builds jb
    SET job_name = 'beam_PostCommit_Java_ValidatesRunner_Gearpump'
    WHERE <query-from-above>;

  5. Remove duplicate history. The import script will import recent history with the migrated job name from Jenkins. The final step is to remove the redundant history with the old job name:

    DELETE jenkins_builds
    WHERE job_name = 'beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle'

Kubernetes Cluster

A Kubernetes cluster hosts the data ingestion scripts and the Grafana frontend. Our cluster is deployed on Google Cloud Platform, under project apache-beam-testing, cluster metrics, workload beamgrafana. The following sections cover instructions for deploying and updating the Kubernetes cluster. Only accounts with the container.deployments.update and container.deployments.create permissions are able to do this. If you need to deploy or update the cluster, request that someone with the appropriate permissions follow these instructions.

First Time Prep

If you haven't worked on the apache-beam-testing GCP project before, there is some configuration work required first.

Deploy the Cluster

Execute the following from the .test-infra/metrics directory of the Apache Beam repository:

  • Add secrets for grafana: kubectl create secret generic grafana-admin-pwd --from-literal=grafana_admin_password=<pwd>
  • Create persistent volume claims:
kubectl create -f beam-grafana-etcdata-persistentvolumeclaim.yaml
kubectl create -f beam-grafana-libdata-persistentvolumeclaim.yaml
kubectl create -f beam-grafana-logdata-persistentvolumeclaim.yaml
  • Build and publish sync containers ./build_and_publish_containers.sh
  • Create deployment kubectl create -f beamgrafana-deploy.yaml

Update the Cluster

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/


## Update deployment from yaml file 
## Update yaml file and build containers
./build_and_publish_containers.sh
## Deploy new version
kubectl replace -f beamgrafana-deploy.yaml

## Manual partial update
# Build and publish sync containers
cd sync/jenkins
docker build -t gcr.io/${PROJECT_ID}/beammetricssyncjenkins:v1 .
docker push -t gcr.io/${PROJECT_ID}/beammetricssyncjenkins:v1

# If needed check current pod status
kubectl get pods
kubectl describe pod <pod_id>

# Update container image via one of the following.
## update image for container 
kubectl set image deployment/beamgrafana container=<new_image_name>

Grafana UI

The Grafana dashboarding frontend is deployed as part of the Kubernetes cluster. The website is exposed at public IP 104.154.241.245.

When you deploy a new Grafana instance, there is some one-time setup:

  1. Log-in at http://localhost:3000 with username admin and the value specified for GF_SECURITY_ADMIN_PASSWORD in docker-compose.yml.
  2. Add Postgres as a data source:
    1. Click the 'Add data source' button.
    2. Fill out the following config:
      • Name: BeamPSQL
      • Type: PostgreSQL
      • Host beampostgresql:5432
      • Database: beam_metrics
      • User: admin
      • Password: POSTGRES_PASSWORD in docker-compose.yml.
      • SSL Mode: Disable
  3. Change default organization name to "Beam" to let anonymous users browse dashboards without logging in. This must be done manually, since it's impossible to do it via configuration (see issue: https://github.com/grafana/grafana/issues/2908).
    1. Log in as an administator
    2. Go to 'Server Admin' / 'Orgs'
    3. There should be one organization called 'Main Org.' Click the name and change it to "Beam"
  4. Change home dashboard. Just like in the previous step, this must be done manually, due to https://github.com/grafana/grafana/issues/10266.
    1. Log in as an administrator
    2. Click at top-left on "Home". Find a dashboard called "Home / Getting Started" and mark it as favourite (give it a star)
    3. Go to 'Configuration' / 'Preferences'
    4. Choose "Home / Getting Started" from a drop-down list next to the 'Home Dashboard' label

Updating Dashboards

The Grafana dashboards are exported as json files in the codebase, and must be re-imported when deploying or making updates:

  1. Run local version of grafana and update/create dashboard
  2. Select share, see json file.
  3. Save/update file in .test-infra/metrics/grafana/dashboards/
  4. Publish new version of binaries.

Appendix

Useful Kubernetes commands and hints

# Get pods
kubectl get pods

# Get detailed status
kubectl describe pod <pod_name>

# Get logs
kubectl log <PodName> <ContainerName>

# Set kubectl logging level: -v [1..10]
https://github.com/kubernetes/kubernetes/issues/35054


# Update kubernetes secret
kubectl create secret generic production-tls --from-file=./tls.key --from-file=./tls.crt --dry-run -o yaml | kubectl apply -f -

Useful docker commands and hints

  • Connect from one container to another
    • curl <containername>:<port>
  • Remove all containers/images/volumes

sudo docker rm $(sudo docker ps -a -q) sudo docker rmi $(sudo docker images -q) sudo docker volume prune

Enabling the legend on dashboards

If you have a graph that doesn't have the legend, you can enable it by clicking on Graph name → More → ToggleLegend.  


  • No labels