WORK IN PROGRESS
| Table of Contents |
|---|
Overview
This document describes a draft proposal for NuttX continuous integration.
Email context
Part of the info in this document is related to threads in the dev@nuttx.apache.org mailing list. Some related threads by subject are the following:
- Subject "Jenkins CI": generic talk about the Jenkins CI and this document
- Subject "Build testing": info about the build testing scripts
- Subject "Code review tool": info about the code review tool (
nxstyle) - Subject "Test repository": info about were to put the CI/testing stuff.
Authors
- Miguel Herranz (miguel@midokura.com)
- ... (collaborate and put your name here too
)
Goals
The goals of continuous integration are the following:
...
| Case | Maximum duration | Commit message format & code style | Build of nuttx and/or apps | Minimal testing | Full testing |
|---|---|---|---|---|---|
| Rebase from master in local developer environment | ~1-5 minutes | always | always | optional | not recommended |
| Prepare local code for pull request | ~1 hour | always | always | recommended | optional |
| Continuous integration triggered by a pull request update | ~2-3 hours | always | always | always | optional |
| Nightly build | ~12 hours | always | always | always | always |
Pipeline
The pipeline term is used here in a loose way. It means whatever arrange of tools and scripts to execute the CI. Specifically it does not restrict to the Jenkins CI pipelines.
...
Each one of these steps should be usually reflected by a step in the pipeline of the chosen CI tool, so that its UI allows easy identification of the step. They also should have easy access to
Admit
This step will basically check that the commit is well formed and that the source code follows the NuttX coding standards.
...
The output of this step will be a report that will include all the detected problems, so that the contributor could address them all: in this case it should not fail-fast until all the errors are detected. For example, if 2 files have been modified, the tool should not stop after finding one error in one of the files. The idea of fail-fast is to save resources, but in this case, the compute resource wasted is minimal, and the weight of the "contributor's time resource" is higher.
C language
...
nxstyle: this is a tool available undertools/nxstyle.cinincubator-nuttxrepository.
Build
This step will check that source code can be build in all or at least the main architectures.
...
The output of this step should be any artifact that could be used by the test step. They should be complete enough to be able to be downloaded and successfully tested on local developer environment, if desired.
Test
Static code Analysis
Currently no static code analysis tests are done in NuttX, so this will kind of tests are out of the scope for the initial CI pipeline.
Unit test
These tests should test individual functions, like simple API calls, etc.
Some of them are under testing/ directory in incubator-nuttx-apps repository.
Functional tests
It seems most of the current test available in NuttX are in this category, despite this classification could be discussed. Most of them are under testing/ directory in incubator-nuttx-apps repository.
Integration tests
These tests should test NuttX integration with external and third-parties code.
Currently no integration tests are done in NuttX (to be checked), so this will kind of tests are out of the scope for the initial CI pipeline.
Performance tests
Currently no performance test are done in NuttX, so this will kind of tests are out of the scope for the initial CI pipeline.
Coverage tests
These tests should gather the coverage reports of previous test steps, and generate a report about the coverage (absolute and relative to the just code in the contribution (GIT commit) under test).
Currently no coverage output is generated in previous tests, so this will kind of tests are out of the scope for the initial CI pipeline.
Implementation
Current proposal for implementation is to generate three Jenkins pipeline jobs: incubator-nuttx, incubator-nuttx-apps, incubator-nuttx-testing (specially the first one).
...
Using a multibranch pipeline allows to properly track the master, release and other branches and also the ones created due to pull requests. It also integrates smoothly with new Jenkins interfaces (Blue Ocean) for better/simpler visualization of the pipelines.
Pipeline versioning
As commented, a Jenkins multibranch pipeline defined by the contents of tools/Jenkinsfile file in the root each repository. But importantly, as demanded by explicit requirement, that file will not contain the logic of the pipeline itself. Instead, it will use a pipeline library defined in the incubator-nuttx-testing repo.
This way, the new changes to the CI logic will not need changes in the main repos, like incubator-nuttx or incubator-nuttx-apps and all the changes will go to incubator-nuttx-testing.
Triggers
The triggers of the pipelines will be:
...
NOTE: the Jenkins multibranch pipeline jobs will be configured to restrict pull requests that modify the tools/Jenkinsfile to committers, to avoid potential security issues by abuse of not trusted users.
Environment
The jobs in the pipeline will have the following filesystem as part of their environment:
nuttx/will contain the contents of themasterbranch (or release/topic branch) ofincubator-nuttxrepository, or the trigger commit, in case it belongs to that repository.apps/will contain the contents of themasterbranch (or release/topic branch) ofincubator-nuttx-appsrepository, or the trigger commit, in case it belongs to that repository.testing/will contain the contents of themasterbranch (or release/topic branch) ofincubator-nuttx-testingrepository, or the trigger commit, in case it belongs to that repository.
Pipeline example
Here is a draft of what could be the pipeline script:
...
|
|---|
TODO
This is a TODO list with questions and issues to discuss:
...