Page History

...

To solve these issues we have decided to create individual jobs for each module. Given that Sling has a large number of modules, it is unrealistic to manage them all manually. As such, we have started using We currently use a GitHub folder that includes all Sling GitHub repositories with a Jenkinsfile present. Adding a file to a repository automatically creates a new job in that folder.

We have previously used the Jenkins Job DSL Plugin . This allows us to programatically create jobs based on a groovy script.

Views

We currently have the following views defined:

, but moved away from it due to the better GitHub integration of the Multibranch Pipeline job type.

Jenkins top-level entry points

The following Jenkins entry points are available:

• Sling Modules  - GitHub organisation folder (provided by https://github.com/jenkinsci/github-branch-source-plugin, documentation at https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/github-branch-source-plugin) that Sling - holds all Sling jobs
• Sling-Dashboard - holds all Sling jobs that need attention ( failed, unstable )Monitor - View that displays all master branch builds that have not completed successfully
• Open PRs - a listing of all pull request open against Sling repositories

To see the details on how the Sling job generation is configured, click Configure  after visiting the Sling GitHub organisation folder .

New repositories should appear after at most 1 day, after they have been discovered by the GitHub organization folder. If needed, the Scan Organization Now  button can be  used to force a scan for new projects.

New jobs are created with the following metadata

Managing jobs

Since all the job management is now done via a Groovy scripted pipelines script manual job creation is discouraged. Altering an existing job is also discouraged not possible since the changes will be undone when the jobs are next regeneratedautomatically created and updated.

Jobs Job configurations are managed by using a 'seed' job, which processes the DSL script. The job is currently located at https://builds.apache.org/view/S-Z/view/Sling/job/sling-seed-build . Typically there is no need to modify this job.The script controlling the job management is located at https://github.com/apache/sling-tooling-jenkins/blob/master/create_jobs.groovy . To add a new job, the module must be available on GitHub and listed in the manifest file at https://github.com/apache/sling-aggregator/blob/master/default.xml .generated by the Jenkins pipeline library helpers under sling-tooling-jenkins/vars. The recommended Jenkinsfile to use in Sling projects is

/**
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

slingOsgiBundleBuild()

The reusable logic is contained in classes under sling-tooling-jenkins/src and configured on the GitHub organization folder level. By default the Maven build being triggered by Jenkins is activating a profile with id ci. This allows triggering specific steps inside your pom.xml only for CI builds (SLING-9749).

This allows modules that do not follow the standard job pattern to reuse the main logic ( error handling, Sling module descriptor parsing, notifications ), while having their own separate stage definitions. The only module which does this currently is the Sling IDE Tooling ( see the Sling IDE Tooling Jenkinsfile ). It is recommended to first try and apply the default slingOsgiBundleBuild build before using non-standard Jenkinsfiles.

Per-job customisations can be applied by creating a Sling module descriptor in the git repository root, see Sling module descriptor for details.If any of these things change, the seed job must be executed to update the generated jobs

Changes to sling module descriptors are taken into account in the next build. Changes applied to the Sling pipeline library will take effect in the job after. The reason is that the first build applies the updated settings after it's complete, so the next build will see the updates.

SNAPSHOT parent versions

...

Testing more complex changes with Docker

If you wish to perform more complex changes, affecting the logic of the script, it's recommended to test these changes on a separate Jenkins instance. This way you minimise the chances that something goes wrong on the ASF Jenkins instance and you get very quick feedback. As of 2016-10-01, a run on sling-seed-build on builds.apache.org takes between 10 and 90 minutes.

The fastest way to setup Jenkins locally ( until we fix

...

If you wish to test the actual job execution, label the master with the "Ubuntu" label. Otherwise, only the seed job will run.

Limitations

We rely on the Jenkins Maven Plugin to automatically detect relationship between jobs, e.g. from org.apache.sling.distribution.core to org.apache.sling.distribution.api . However, this does not take into account the provisioning model. To work around this, we currently set up some very broad rules, such as 'all entries under bundles/ or installer/ trigger a launchpad build'. Also, we manually set some dependencies between some of the launchpad testing jobs. We might revisit this to see if we can do something more fine-grained.

...

GitHub API usage

The ASF Jenkins instance is configured to use the GitHub API using a Personal Access Token. This token allows us to get decent rate limits and also to perform write actions on GitHub - for instance setting a commit's status or adding a pull request comment. Currently the used OAuth scope is repo.

For legal reasons the ASF Infra team does not allow usage of bot accounts, so currently the token is tied to Robert Munteanu 's GitHub account. This also has the consequence of showing the avatar of the user that owns it, which may be a bit surprising.

Validating PRs

Even PRs might trigger Jenkins Jobs. The hook used for that is described at GitHub webhook relay service. The event will only fire though for actions being triggered by ASF committers (not by outside contributors).

References

1. Jenkins Job DSL Plugin - https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin

2. Jira
   server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key SLING-6061

3. dev@sling.apache.org - sling ci builds per module : http://markmail.org/message/l6wc4bu5pzw5td3d
4. dev@sling.apache.org - CI alternatives for Sling:  http://markmail.org/message/mdn4anwe6kxqxa2z
5. builds@apache.org - Splitting a large build into _many_ smaller builds : http://mail-archives.apache.org/mod_mbox/www-builds/201609.mbox/%3C1472811065.24075.8.camel%40apache.org%3E
6. Multibranch Pipeline recipes

7. Jira
   server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key SLING-7245