Table of Contents
...
1. What is Github Checks
Github checks consists of a series of checking tasks, designed to verify whether the basic functions of doris Doris pass. The main test branch is the pull requests that the developers of the doris open source community synchronize to the doris master branch。Checking tasks are tested separately through 3 pipelines of teamcity: Doris FE UT, Doris BE UT, Doris P0 Regression,and the final test results will be pushed to github.
How to trigger Checks
No manual triggering required. Every commit of every active pull request related to the Doris master branch will be detected by teamcity server after 10 minutes, teamcity automatically generates 3 pipeline tasks (Doris FE UT, Doris BE UT, Doris P0 Regression) related to the commit, waiting for scheduling execution.
Teamcity Pipeline Guidance
Whats Teamcity Server Address
Before you start, you need to know the open source community pipeline address: http://43.132.222.7:8111. Ordinary users can view it as a guest (Log in as guest), or jump directly to the pipeline details page from github.
How to check pipeline results in github
1. Find the target pull request
Find the target pull request in the git repository of apache/incubator-doris
2. Check teamcity build results in github
On the details page of the target pull request, drop down to the "Review required" section, slide the gray bar on the right, and find the FE UT (Doris FE UT), BE UT (Doris BE UT), and P0 Regression (Doris P0 regression) tasks. A green tick indicates that the build task succeeded, and a red cross indicates that the build task failed. The build result of pull/8901 is shown here
3. Redirect to teamcity to check the specific build task information
When a Pull Request is submitted, Github Checks are automatically triggered.
The following Checks are REQUIRED, and only after passing these checks can the code be merged:
- BE Code Formatter: BE C++ code format check(sh build-support/check-format.sh)
- FE Code Style Checker: FE JAVA code format check (cd fe/ && mvn clean install -DskipTests)
- BE UT (Clang): Run BE unit test with Clang (DORIS_TOOLCHAIN=clang ./run-be-ut.sh --run --clean)
- BE UT (macOS): Run BE unit test on macOS (./run-be-ut.sh --run --clean)
- BE UT (Doris BE UT): BE unit tests will be executed: sh run-be-ut.sh --run
- FE UT (Doris FE UT): FE unit tests will be executed: sh run-fe-ut.sh --run
- License Check: License check, will check whether all source files contain Apache License Header
- ShellCheck: Check all shell scripts format. (sh build-support/shell-check.sh)
- Build Extensions / Build Broker: Build Broker (cd fs_brokers/apache_hdfs_broker/ && sh build.sh)
- Build Extensions / Build Documents: Build Documents (cd docs && /bin/bash build_help_zip.sh)
- COMPILE (DORIS_COMPILE): Will execute the compilation of FE and BE (BUILD_TYPE=ASAN sh build.sh --fe --be), then trigger P0/P1/External Regression
- P0 Regression (Doris Regression): Will build the Doris environment of 1 FE and 1 BE, and execute regression p0 test(sh run-regression-test.sh --run -g p0)
- P1 Regression (Doris Regression): Will build the Doris environment of 1 FE and 1 BE, and execute regression p1 test(sh run-regression-test.sh --run -g p1)
- External Regression (Doris External Regression): Will build the Doris environment of 1 FE and 1 BE, and execute regression external test(sh run-regression-test.sh --run -g external_docker)
- performance (Doris Performance): Will execute the compilation of FE and BE (BUILD_TYPE=RELEASE sh build.sh --fe --be), build the Doris environment of 1 FE and 1 BE and execute some performance cases(see regression-test/pipeline/performance/)
- Build Third Party Libraries (Linux): Build the third party libraries on Linux x86_64 (cd thirdparty && ./build-thirdparty.sh)
- Build Third Party Libraries (macOS): Build the third party libraries on macOS x86_64 (cd thirdparty && ./build-thirdparty.sh)
The other Checks are NOT REQUIRED. Which means, even if these checks fail, it does not affect the merge of the Pull Request.
2. How Checks are triggered?
1.The Checks 5,6,11,12,13,14,15 are triggered by TeamCity.
2.The other Checks are triggered by Github Action. They will be triggered immediately after PR is submitted or updated.
The pipeline trigger on teamcity to include the run and following special keyword comments in the PULL REQUEST COMMENT section,The keyword run is also included to avoid error firing.
Key words:
buildall: This keyword will trigger all pipelines on teamcity
compile: This keyword will trigger the pipeline of COMPILE (DORIS_COMPILE)
feut: This keyword will trigger the pipeline of FE UT (Doris FE UT)
beut: This keyword will trigger the pipeline of BE UT (Doris BE UT)
p0: This keyword will trigger the pipeline of P0 Regression (Doris Regression)
p1: This keyword will trigger the pipeline of P1 Regression (Doris Regression)
external: This keyword will trigger the pipeline of External Regression (Doris External Regression)
performance: This keyword will trigger the pipeline of performance (Doris Performance)
arm: This keyword will trigger the pipeline of P0 Regression (ARM pipeline)
cloudut: This keyword will trigger the test pipeline of Cloud UT (Doris Cloud UT)
cloud_p0: This keyword will trigger the test pipeline of cloud_p0 (Doris Cloud Regression)
cloud_p1: This keyword will trigger the test pipeline of cloud_p1 (Doris Cloud Regression)
Note:
To avoid error firing, you need to include both run and the specified keyword,like "run p0" , "run buildall",this is so important!
- p0,p1, and external pipeline all depend on compile-phase output, so when you want to retry one of these pipelines separately, you need to make sure that the corresponding branch has compiled output. If not, please first run the compile phase by adding the keyword with compile in the comment area of the corresponding pull request. Build the dependency pipeline after compiling.
- When you use run buildall, it means that you have triggered compile feut beut p0 p1 multiple pipelines. you need to wait for the pipeline result. It is best not to trigger the same pipeline again while waiting for the result unless the pipeline fails due to some accident
For example:
- When developers add a comment with a buildall keyword to the Conversation area of the PULL REQUEST, for example: "run buildall cases", git action will detect this behavior and trigger FE single test, BE single test, and code compilation pipeline. when the code compilation pipeline. p0 and p1 pipeline. if you want to execute data lake related use cases simultaneously, You can have a comment like "run buildall and external", and the external pipeline will automatically start the data lake environment to perform the relevant tests.
- When developers find a pipeline running error but want to retry it, they can add a comment with the pipeline keyword in the Conversation area of the PULL REQUEST, such as "run p1". If p1 fails the test due to lack of compilation output, So we need to add a comment with the "run compile" keyword first, and after successful compilation, p0,p1 pipeline will be automatically triggered.
3. How to View Checks
3.1 Github Action
Github Action use Github hosted VM to run the Checks.
You can click "Details" button behind the Checks to see the build log of the corresponding Check.
3.2 TeamCity Pipeline Guidance
TeamCity Pipelines are running on self-hosted VM.
3.2.1 Login to TeamCity Server
Find the target Pull Request:
There are several status of TeamCity Pipeline:
- Expected — Waiting for status to be reported: The pipeline is not started, waiting in the queue.
- Pending — TeamCity build started: The pipeline is running.
- TeamCity build finished: The pipeline finished successfully.
- TeamCity build failed: The pipeline finished with errors.
- Skip teamCity build: The pipeline does not need to run and just skiped.
You can click "Details" to go to the TeamCity Web UI. Or visit http://43.132.222.7:8111/login.html directly via web browser.
Anyone can click "Log in as guest" to login to TeamCity Server.
3.2.2 Check the specific pipeline status
If you jump to TeamCity by clicking the "Details" button in Pull Request. You will be lead to the details part of specified pipeline:
Click "Details" on the right side of the construction task, you can jump to teamcity to check the specific information of the build task. The navigation bar of the construction information page contains 5 parts:
- Overview
...
- : Display a simple report of task execution
...
- .
- Changes: Display that the build task involves repository, branch, commit history and other information
...
- .
- Tests: Display the result of the case execution. If you want to view the failure details, you can click "Failure" on the left side of the failure case to jump to the specific error information page
...
- .
- Build Log: Display the specific log of the task execution, supports retrieval, and supports
...
- log download
...
- .
- Artifacts: Useless for now.
Basically, you can just visit Overview to see which tests are failed and visit Build Log to see the complete build logs.
3.2.3 Check pipeline history
If you visit TeamCity via http://43.132.222.7:8111/login.html, you can also see all other pipelines waiting or running in TeamCity
1. Choose "Doris" project
2. Get the specific pull request implementation
2.1 Get the latest execution task of the pull request
Filter the pull request you want to get on the right side of the Doris project, and expand the sub-project to see the latest record details built under the pull request.
Green indicates that the build was successfully executed, Red indicates that the build task failed, and the reason for the failure is displayed.
Taking pull/8901 as an example here, the latest build task of pull/8901 in each subproject is queried in the Doris project. The execution of FE UT fails, and the execution of BE UT and P0 Regression succeeds.
2.2 Get the historical execution tasks of the pull request
If you want to view the historical task records built by the pull request, you can select a branch under each subproject, and you can get the historical records of the branch execution.
3. Get the details of a specific task
Click the task id to jump to the task details page, and select the information you want to view in the navigation bar. Same as described in Section 3.2.2.
If you haven't found a build task matching the target branch here, you can continue to step 4.
4. checking pending builds
If the branch construction task has not been found in the previous steps, the target task is still in the queue and has not been scheduled.
You can go to "Queue" and use the target pull request to perform a global search to check whether the task is in the queue.
4. Jenkins Daily Test
4.1 What is Jenkins?
Essentially, Jenkins is an automation engine that supports many automatic modes. Users can define pipeline on Jenkins to automatically perform a series of repetitive tasks.
The ASF Build Services provides a Jenkins server for any ASF project to run build and test tasks for their project from this server. (See the Jenkins wiki page).
And all user will be able to view results and logs of those builds on the Web UI.
4.2 How the Test Triggered?
There are two pipeline on Jenkins for Doris:
- doris_daily_disable_vectorized: Disable the vectorized engine before running test.
- doris_daily_enable_vectorized: Enable the vectorized engine before running test.
All of the pipeline automatically triggered once a day in Jenkins server.
4.2 How to View Test Results?
You can find two badges named "VectorizedEngine" and "OriginEngine" on the Apache Doris GitHub home page.
Anyone can click those badges to go to the Jenkins pipeline's Web UI. Or visit https://ci-builds.apache.org/job/Doris directly via web browser.
4.2.1 Check the pipeline build history
You can check the build history of the pipeline on the left side of the pipeline page.
4.2.2 Check the build logs via Blue Ocean UI
Blue ocean can greatly improve the visual experience of Jenkins.
You can click the button in the left list to enter the Blue Ocean UI.
In Blue Ocean UI, you can check the specific build logs, and also can see all the success, failed and passed cases and other build status like time, code change.
5. FAQ
1. Why have my PR Checks never started running?
For Github Action Checks, they should be triggered and run immediately.
But If you are "first-time contributor", you need to ask one of Doris Committer to approve the checks running.
For TeamCity Pipeline, first of all, the trigger interval of TeamCity is 10 minutes, so please wait at least 10 minutes.
Secondly, because there are only 3 servers running these pipelines, only one pipeline can be run on the server at the same time every day,
so when there are many PRs, the pipeline will enter the waiting queue. See checking pending builds to view the pending builds.
2. How to re-trigger Teamcity pipeline?
1. Submit a new commit to your Pull Request
Any update to the Pull Request will trigger a new Pipeline.
2. Contact the Doris PMC to re-trigger
Occasionally, the check may fail because of a problem with the Pipeline itself, rather than a problem with the code.
At this point the Doris PMC can be contacted via the dev@doris.apache.org for manual re-triggering.
3. The check fails, but the cause of the error is not caused by my code.
In some cases, some problematic code may be incorrectly merged into the trunk,
which may cause all subsequent PRs developed based on the trunk to encounter check failures.
Please refer to the following process to solve:
- Email dev@doris.apache.org for help with error messages.
- The community will help to fix related issues.
- After the related problem code is merged into the master, rebase the master code and update the Pull Request to re-trigger Checks and pass the check.
4. How to get detail logs and debug info from TeamCity?
1. Download Build Log
2. At the end of Build Log, you can find a download link to download all FE/BE log and coredump file
5. How to modify the configuration files used in the TeamCity pipelines?
- the pipeline of P0 Regression (Doris Regression) uses the conf in https://github.com/apache/doris/tree/master/regression-test/pipeline/p0/conf
- the pipeline of P1 Regression (Doris Regression) uses the conf in https://github.com/apache/doris/tree/master/regression-test/pipeline/p1/conf
- the pipeline of External Regression (Doris External Regression) uses the conf in https://github.com/apache/doris/tree/master/regression-test/pipeline/external/conf
- the pipeline of performance (Doris Performance) uses the conf in https://github.com/apache/doris/tree/master/regression-test/pipeline/performance/conf
- the pipeline of cloud_p0 (Doris Cloud Regression) uses the conf in https://github.com/apache/doris/tree/master/regression-test/pipeline/cloud_p0/conf
- the pipeline of cloud_p1 (Doris Cloud Regression) uses the conf in https://github.com/apache/doris/tree/master/regression-test/pipeline/cloud_p1/conf
6. Suggestions
If you have any suggestions and comments, welcome to email dev@doris.apache.org to discuss.
...