Date, Time & Location
UTC time 1900 hrs
Meeting space UPDATED: https://meet.google.com/ntz-wita-vns
Attendees (Add your cwiki name below)
This event is open to all committers and guests
- Chris Thistlethwaite
- Sander Striker
- Jarek Potiuk
- Vladimir Sitnikov
- Kaxil Naik
- Tobiasz Kędzierski
- Paul Angus
- Justin Erenkrantz
- Ed Baunton
- Zach Hoffman
- Drew Foulks
Format
The meeting is yours to discuss whatever you want to. Aim to discuss with 'each other' as
in others subscribed to the builds@ mailing list, from your project or from other projects.
There will also be Infra representation for infra related discussion items. I'd like to be clear
though, this is an informal meeting arranged for the benefit of individuals and projects to
talk about anything they like in relation to the ASF Build services and related integrations,
and is not a projects vs infra panel discussion.
Topics of interest
Add here anything you would like a chance to talk about this week.
| Item | Who | Notes |
|---|---|---|
| Dockerhub | ||
| GitHub Actions, Docker Hub, GitHub Container Registry | ||
| GitHub Actions, (Gradle) Build Cache, resource consumption per project | It would help if there was a way to analyze resource consumption for the project. For instance, if TLP is in topN, then it might be time to optimize its build. Currently, it is not clear if the project I participate in cause issues for other projects. | |
| GitHub Actions | Tobiasz Kędzierski | |
| Discoverability of resources, clarity of SLAs / capacity | ||
| Remote Execution, worth exploring? | If there is not enough time for this topic this time, happy to defer to a next session. | |
Meeting Notes (Summary)
Meeting started with a slight hiccup, Jitsi crashed on us so we moved to a Google Meet organised quickly by Sander Striker (thanks Sander!).
Gavin started off by letting everyone know that Docker had upgraded our DockerHub account and that download limits and image retention limits
would not apply to any project under the 'apache' org namespace on DockerHub. Gavin also mentioned an upcoming meeting with Github to talk about usage
stats , container registry etc. (Update on that meeting, GH cant provide stats as we want them yet, but are able to provide 'snapshots').
Sander talked about his Board level action item, mainly surrounding Infra documentation on how projects find the CI resources that are available to them. Sander
later sent an email asking for those on the builds@ list to get involved in that initiative.
Sander introduced guest Ed Baunton, who together talked about the advantages of remote execution, Buildgrid and Buildbox, caching of results of build steps to avoid
repeating steps of a build that have already passed, just send along the result for the next step, and much more. An action item came of this to actually build a beta
test service to investigate how this can be of benefit to ASF projects. Maven folks in particular are a bit skeptical as to whether it could work for their use case.
Chris Lambertus asked for interest in setting up a K8s cluster of nodes to add to our Jenkins offering, interest was positive and a coupe of people put their hands
up to assist in the creation of this.
Jarek, Vladimir and others were interested in talking about Github Actions, and most of the rest of the meeting was centered around this topic. Mainly, how do we know
who is using what resources, where are the bottlenecks, how can we make better use and be fairer to all. Our account has unlimited minutes, but the contention is with
the 'runners' of which we have 180. Obviously some visible stats from Github in this area would help, that is on the list of things that has been asked for, and hopefully
will be become a reality at some point soon. To ease the load of the provided runners, some projects want to investigate the possibility of self-hosted runners. For
open source projects however, this is a potential avenue of attack for example via PRs. Infra does not endorse self hosted runners at this time - but also is not blocking
on any project that wants to investigate further. (Perhaps look into restricting what PRs can trigger a build). Infra will also look into the security aspects of self-hosted
runners and any possible solutions/workarounds.
Edit: Since the meeting, and the meeting that took place with Github afterwards, it was suggested that projects should start using a Max Parallel Setting that is
available for Actions, this would reduce the amount of runners an action can consume at any one time. Infra suggests to set that at '5'.
During the builds meeting, Gavin pointed people to https://infra-reports.apache.org/cistats/ ; and suggestions to make this URL more widely known be adopted.
Whilst unofficial and created originally for Infra internal usage, we can see its usefulness beyond this and so will get added to docs.
Some good talk was had about Yetus and how it could prove useful for other projects, beyond what it already does for the Hadoop ecosystem. Allen could perhaps
talk more about that at a future meeting, maybe a demo of some kind? I think I recall Allen and Chris talking about Yetus use for Infra, needs clarification.
Action Items
- Gavin McDonald Set up a Buildgrid instance to investigate if it would be useful for projects
- Gavin McDonald Set up a Gradle Build Cache on asf949
- (open) - Remote Build Cache Server (clarify type and usage, seperate from Gradle Cache) , suggested by Vladimir Sitnikov
- Chris Lambertus Extend Docker image retention limits ( per request by Allen Wittenauer )
- (open) - Investigate Apache Superset Dashboard for CI usage stats
- (open) - Investigate Apache Cloudstack for possible use.
- (open) - Spin up on demand nodes, such as GPU etc.
- Chris Lambertus to follow up on K8s cluster setup, with volunteers.
- Andrew Wetmore Add https://infra-reports.apache.org/cistats/ to docs for higher visibility.
Next Meeting
Interest was fantastic and so I aim to hold a meeting once a month on the 2nd Thursday,
Next meeting date is 17th December at 1900 UTC via Jitsy. (Pushed out by a week due to accommodating for Github guest)