WORK IN PROGRESS
Overview
This document describes a draft proposal for NuttX continuous integration.
Email context
Part of the info in this document is related to threads in the dev@nuttx.apache.org mailing list. Some related threads by subject are the following:
- Subject "Jenkins CI": generic talk about the Jenkins CI and this document
- Subject "Build testing": info about the build testing scripts
- Subject "Code review tool": info about the code review tool (
nxstyle
) - Subject "Test repository": info about were to put the CI/testing stuff.
Authors
- Miguel Herranz (miguel@midokura.com)
- ... (collaborate and put your name here too
)
Goals
The goals of continuous integration are the following:
- Avoid "integration hell" by provide a quick but thoughtful automated check of new code addition so that the Code Contribution Workflow can be done seamlessly.
- Definition of quick is difficult, but some hints are:
- At maximum, a few minutes to rebase local developer work with latest master. Usually a developer get a lot more code than he produces, so is important the rebase to be easy to do, or the developer will prefer to just keep working for a few days, leading to potential "integration hell".
- At maximum, a couple hours to do minimum set of checks in the CI server. This is needed to not saturate CI service.
- Less than a day for a full regression including all the tests (nightly tests). This is needed to have a daily stable base.
- Fail fast approach: the fastest tests should be the first one to be tested, and they should provide feedback right away, stopping any further processing until they are addressed so that CI resources could be freed for testing other changes. Nevertheless, the CI pipeline should be reproducible by local developer environment, allowing the developer to test before submit, so minimizing the developer coding loop.
- Definition of thoughtful is even more difficult, but some hints are:
- Check the source code style
- Check it builds and executes in the main architectures
- Check it don't break the main functionalities
- Check that all the new code is covered by the existing or new added tests.
- Provides useful logs, errors and reports, so that is easy for the developer to fix the problems.
- The results are consistent (the result of a given commit does not change over time or execution environment)
- By automated, it would be executed in a CI like ASF Jenkins service. However, the CI platform specific part should be clearly separated from the parts that should be common to the testing in local developer environment. For example, if there is a Jenkins pipeline that is executed, it should use some external script that can be use without a Jenkins environment, and a tool to run the script locally in a similar way to the Jenkins pipeline should be provided.
- Definition of quick is difficult, but some hints are:
- Use standardized tools and methods whenever it is possible and practical so that more people can contribute.
In general, other common practices should be considered as goals too.
To some extension, the following table should give a hint about what to include in each situation:
Case | Maximum duration | Commit message format & code style | Build of nuttx and/or apps | Minimal testing | Full testing |
---|---|---|---|---|---|
Rebase from master in local developer environment | ~1-5 minutes | always | always | optional | not recommended |
Prepare local code for pull request | ~1 hour | always | always | recommended | optional |
Continuous integration triggered by a pull request update | ~2-3 hours | always | always | always | optional |
Nightly build | ~12 hours | always | always | always | always |
Pipeline
The pipeline term is used here in a loose way. It means whatever arrange of tools and scripts to execute the CI. Specifically it does not restrict to the Jenkins CI pipelines.
In principle, all the code for the pipeline should be stored in the separate repository, so that the main NuttX code is not "contaminated" by it (see thread "Test repository" in dev@nuttx.apache.org mail list). Currently the repository to use is: https://github.com/apache/incubator-nuttx-testing
The continuous integration pipeline consist in the following steps and substeps.
- Filter
- Build
- Test
- Static code analysis
- Unit tests
- Functional tests
- Integration tests
- Performance tests
- Coverage tests
Each one of these steps should be usually reflected by a step in the pipeline of the chosen CI tool, so that its UI allows easy identification of the step. They also should have easy access to
Admit
This step will basically check that the commit is well formed and that the source code follows the NuttX coding standards.
The intention is to discard as early as possible that the code is not fulfilling the coding standards. Building and testing stages are way more costly in resources, so it would be wasteful to test all the pipeline just to discover that the code is not acceptable to start with.
The output of this step will be a report that will include all the detected problems, so that the contributor could address them all: in this case it should not fail-fast until all the errors are detected. For example, if 2 files have been modified, the tool should not stop after finding one error in one of the files. The idea of fail-fast is to save resources, but in this case, the compute resource wasted is minimal, and the weight of the "contributor's time resource" is higher.
C language
Proposed tools:
nxstyle
: this is a tool available undertools/nxstyle.c
inincubator-nuttx
repository.
Build
This step will check that source code can be build in all or at least the main architectures.
Proposed tools:
tools/testbuild.sh
inincubator-nuttx
repository
In principle not all the architectures should be tested, so at least one configuration for minimal test and another for full one is expected.
The output of this step should be any artifact that could be used by the test step. They should be complete enough to be able to be downloaded and successfully tested on local developer environment, if desired.
Test
Static code Analysis
Currently no static code analysis tests are done in NuttX, so this will kind of tests are out of the scope for the initial CI pipeline.
Unit test
These tests should test individual functions, like simple API calls, etc.
Some of them are under testing/
directory in incubator-nuttx-apps
repository.
Functional tests
It seems most of the current test available in NuttX are in this category, despite this classification could be discussed. Most of them are under testing/
directory in incubator-nuttx-apps
repository.
Integration tests
These tests should test NuttX integration with external and third-parties code.
Currently no integration tests are done in NuttX (to be checked), so this will kind of tests are out of the scope for the initial CI pipeline.
Performance tests
Currently no performance test are done in NuttX, so this will kind of tests are out of the scope for the initial CI pipeline.
Coverage tests
These tests should gather the coverage reports of previous test steps, and generate a report about the coverage (absolute and relative to the just code in the contribution (GIT commit) under test).
Currently no coverage output is generated in previous tests, so this will kind of tests are out of the scope for the initial CI pipeline.
Implementation
Current proposal for implementation is to generate a Jenkins pipeline job, called nuttx
.
Pipeline versioning
The execution will consist on a Jenkins Pipeline defined by the contents of Jenkinsfile
file in the root the ci-configuration
branch of incubator-nuttx-testing
repository (see https://jenkins.io/doc/book/pipeline/getting-started/#defining-a-pipeline-in-scm).
Triggers
The triggers of the pipeline will be:
- Pull requests master branch (or release/topic branch) of
incubator-nuttx
repository - Pull requests master branch (or release/topic branch) of
incubator-nuttx-apps
repository - Pull requests master branch (or release/topic branch) of
incubator-nuttx-testing
repository
WARNING: the special branch in incubator-nuttx-testing
repository, called ci-configuration
, will have a special functionality as explained before, and must not be used as trigger source. This branch will not trigger any automated job from its pull request. Instead, it should be tested using another Jenkins pipeline job, called nuttx-testing-ci-configuration
, that should be triggered manually by Apache NuttX committers with enough privileges. This will allow to test in a more controlled way the changes in the pipeline itself.
Environment
The jobs in the pipeline will have the following filesystem as part of their environment:
nuttx/
will contain the contents of themaster
branch (or release/topic branch) ofincubator-nuttx
repository, or the trigger commit, in case it belongs to that repository.apps/
will contain the contents of themaster
branch (or release/topic branch) ofincubator-nuttx-apps
repository, or the trigger commit, in case it belongs to that repository.testing/
will contain the contents of themaster
branch (or release/topic branch) ofincubator-nuttx-testing
repository, or the trigger commit, in case it belongs to that repository.
Pipeline example
Here is a draft of what could be the pipeline script:
|
---|
TODO
This is a TODO list with questions and issues to discuss:
- Is there a "POSIX compliance" test suite? Making a quick web search reveals some projects exists (http://posixtest.sourceforge.net/, https://github.com/juj/posixtestsuite, etc) aside the official one (http://www.opengroup.org/testing/testsuites/posix.html) so maybe it could be used in testing NuttX compliance to standards.
- In order to test NuttX in more platforms and real boards, specialized CI tools could be added. For example, like open source ones like https://kernelci.org/ (that leverages https://lavasoftware.org/).