Abstract

The Open Climate Workbench (OCW) project has basic unit test coverage in the form of a test suite written using the nose framework. In order to improve test coverage, functionality, and quality of tests the test framework should be updated. OCW would greatly benefit from a continuous integration (CI) infrastructure and automated test environment with a focus on large scale integration testing of climate model analysis. The current testing makes the process of new code integration tedious and manual with potential regressions going unnoticed, this can lead to bugs and backward incompatibility. This GSoC project proposes to build an improved test framework (smoke tests and unit tests) and set up continuous integration tools.

Design/description of work

The key deliverable of this project involves improving the existing unit tests before designing and developing a brand new smoke test suite which will then enable CI testing for OCW. In the second half of the project, the continuous integration infrastructure will be built and fully automated. This involves configuring the dedicated build clusters at Apache Builds (and also at Travis-CI). The continuous integration server would automatically build and test the source code for each pull request that is submitted and post the results. All documentations related to the project will also be created. If time permits, I would also like to improve the overall project by fixing the numerous issues the project currently has.

Results for the Apache community

This project will result in a robust and effective test suit for the OCW project along with a working continuous integration environment that tests the source code and provides useful results. Also, there will be documentations for the new additions to OCW.Objectives:

1. Improve Unit Tests

2. Develop Smoke Tests

3. Configure continuous integration environment

Deliverables

1. An improved unit test suite.

2. A newly developed smoke test suite.

3. A fully set up continuous integration environment.

4. Documentation related to the project.

Scheduling

Community Bonding Period (April 22, 2016 - May 22, 2016)

During the community bonding period, I will solicit the OCW community for feedback on designing of the smoke tests and what the current unit tests lack. I will also try to develop a prototype for the tests in this period which would be useful during the coding period. The OCW project has some automated installation scripts but they have some issues. During this period I’ll also focus on fixing these issues so that the continuous integration environment can be set up effectively.

May 22, 2016 - May 28, 2016

I will be having my end sessional examinations which start from 9th May and end on 28th May and so I won’t be able to work.

May 29, 2016 - June 20, 2016 (Week 1 - 3)

 The aim of week 1 is to start writing unit tests. The current unit tests do not cover the entire project and thus the unit tests will be improved to increase the test coverage. During the weeks 2, 3 unit test coding will be continued.

June 21, 2016 - July 14, 2016 (Week 4 - 6)

 By the end of week 3, the unit test framework should be completed and coding for the smoke tests can be started. The weeks 4, 5 and 6 will be used to develop the smoke testing framework for the OCW project. The design and implementation (prototype) of this framework will be done during the community bonding period.

July 15, 2016 - July 30, 2016 (Week 7 - 8)

The aim of weeks 7 and 8 is to integrate the project into a continuous integration environment. Build(s) job(s) will be established on builds.apache.org to run the new large scale integration tests. Jenkins is the main integration environment which will be configured to automatically execute the test and build the project.

July 31, 2016 - August 7, 2016 (Week 9)

By the end of week 9, the project should be ready as per this proposal. A lot of new code will be added to the project by this time and the documentation will need to be updated. This period will be used to create documentation (the incomplete ones) about the continuous integration tools used and the new testing framework that is added.

August 8, 2016 - August 17, 2016 

This is the buffer period in case things don’t turn out as planned. All code that remains incomplete or has bugs will be fixed in this period.

Every Sunday I will write documentation for all code that is created along with the weekly progress report.

Reports

Weekly Reports

Community Bonding Period

• Week 1 (23^rd April - 30^th April) 

  ---

  Previous Action Items: None
  Weekly Activity:

  Week 1 was used to find where the existing unit tests lack and which modules need to be tested. A test build has been set up at Codecov to generate test coverage reports which will be used for further analysis and development of tests. Also, Jenkins build (which would always fail) has been fixed partially and now It completes successfully. 

  Issues found - CLIMATE-785, CLIMATE-786, CLIMATE-787
  Issues fixed  - CLIMATE-785, CLIMATE-786
  
  Next Week's Agenda: Study and analyze dataset.py and test_dataset.py file in the ocw module.

  Mentor's Comments:Community bonding and engagement has been good. Lots of activity

  
  

• Week 2 (1^st May - 7^th May)

  ---

  Previous Action Items: Fix CLIMATE-787
  

  Weekly Activity:
  The Open Climate Workbench module and the test suite were studied further in this week, especially the dataset.py and test_dataset.py files.

  Multiple issues were found in these files which will be addressed in the coming weeks.   

  Issues found - CLIMATE-788, CLIMATE-788

  Issues fixed -  CLIMATE-789

  Next Week's Agenda: Study and analyze dataset_processor.py and test_dataset_processor.py file in the ocw module.

  Mentor's Comments: Community bonding and engagement has been good. Lots of activity
   

• Week 3 (8^th May - 14^th May)

  ---

  Previous Action Items: Fix CLIMATE-787 and CLIMATE-788
  Weekly Activity:

  The study of test suite was continued in this week. The dataset_processor.py file and test_dataset_processor.py file were studied specifically. A prototype for the functions that are not covered by the tests was developed. Also, the existing test files did not comply with PEP8. All files in the test directory have been updated to comply with PEP8 guidelines.
  
  Issues found - CLIMATE-791, CLIMATE-792
  Issues fixed -  CLIMATE-791, CLIMATE-792

  Next Week's Agenda: Study and analyze evaluations.py and test_evaluations.py file in the ocw module.

  Mentor's Comments: PR's were sent in for a number of issues. Ibrahim has been in contact with me and things are going well. He also logged an issue with INFRA regarding pre-commit build hooks for OCW, Jenkins + Github integration.
   

• Week 4 (15^th May - 21^th May)

  ---

  Previous Action Items: Fix CLIMATE-787 and CLIMATE-788
  Weekly Activity:

  The aim of the week 4 was to study the evaluation.py file and tests related to it, which was achieved. I managed to trace out which tests need to be updated and which functions are not covered currently. I couldn't contribute much during this week as I was having my end sessional examination. I've conveyed the same to my mentor.
  
  Issues found - 
  Issues fixed -

  Next Week's Agenda: Add tests to test_dataset_processor.py file to improve test coverage. 

  Mentor's Comments:  I have been working with Ibrahim Jarif getting him to commit directly to the master canonical source repository hosted at Apache. I am happy with progress so far but would like to see Ibrahim Jarif start thinking about larger scale integration tests. He should also be following up with Infrastructure on getting the Jenkins build issues sorted out as well as documenting the CI infrastructure on the Climate wiki.
  

  
  
  

• Week 5 (22^nd May - 28^th May)

  ---

  Previous Action Items: Fix CLIMATE-787 and CLIMATE-788
  Weekly Activity:
  I managed to add new test cases for dataset processor module. The tests coverage has been increased. Some more tests need to be added which will be completed in the following week. The build at Apache Builds has been fixed. The build is triggered every time a PR is submitted to the climate repository. The builds are failing due to some permission issue with conda package manager. I've been following up with the Apache Infra team for support and the build should be fixed in the next week. Also coveralls integration as been added which keeps track of code coverage.

  Issues found - CLIMATE-793, CLIMATE-794, CLIMATE-795
  Issues fixed - CLIMATE-788, CLIMATE-793, CLIMATE-794

  Next Week's Agenda: Complete remaining dataset processor tests and add tests for evaluation and dataset module.

  Mentor's Comments: A good solid week of work. I am happy with progress.
  

   

• Week 6 (29^th May - 4^th June)

  ---

  Previous Action Items: Fix CLIMATE-787 and CLIMATE-795
  Weekly Activity:
  Completed the tests related to dataset processor module, evaluation module, and the dataset module.
  Improved dataset processor tests coverage from 44.66% to 62%, evaluation tests coverage from 87.13% to 94%, dataset module tests coverage from 84.21% to 100%. Some more tests will be added to these modules after discussing with the mentor. Multiple bugs were fixed in these modules while developing the tests.

  Issues found - CLIMATE-796, CLIMATE-799, CLIMATE-800, CLIMATE-801, CLIMATE-802
  Issues fixed - CLIMATE-795, CLIMATE-796, CLIMATE-799, CLIMATE-800, CLIMATE-801, CLIMATE-802

  Next Week's Agenda: Add tests for utils module and local module.

  Mentor's Comments: 
  
  

• Week 7 (5^th June - 11^th June)

  ---

  Previous Action Items: Fix CLIMATE-787
  Weekly Activity:
  Completed the tests related to utils module and local module. More tests will be added for the local module after CLIMATE-787 has been resolved.
  Improved utils module tests coverage from 69.93% to 100%, local module tests coverage from 29.84% to 51%.

  Issues found - CLIMATE-803, CLIMATE-804, CLIMATE-805, CLIMATE-806
  Issues fixed - CLIMATE-804, CLIMATE-805, CLIMATE-806

  Next Week's Agenda: Develop Smoke tests prototype.

  Mentor's Comments: 
  

   

• Week 8 (12^th June - 18^th June)

  ---

  Previous Action Items: Fix CLIMATE-787
  Weekly Activity:
  The aim of this week was to develop smoke tests prototype which unfortunately remains pending. However, I managed to fix some issues and update ocw module code base. I added integration for landscape.io and requires.io. Also, all the files in the ocw module now comply with PEP8. In the next week, I'll complete the smoke tests prototype.

  Issues found - CLIMATE-807, CLIMATE-808, CLIMATE-809, CLIMATE-811, CLIMATE-812, CLIMATE-813, CLIMATE-814, CLIMATE-815, CLIMATE-816     
  Issues fixed - CLIMATE-807, CLIMATE-808, CLIMATE-809, CLIMATE-811, CLIMATE-812, CLIMATE-813, CLIMATE-814, CLIMATE-815, CLIMATE-816

  Next Week's Agenda: Develop Smoke tests prototype.

  Mentor's Comments

   

• Week 9  - Till Completion (From 19^th June)

  ---

  Activity:
  The smoke test prototype was developed in this period. Along with it, the remaining unit tests were improved and the continuous integration infrastructure was developed. The reports related to a few weeks are missing as the project was completed ahead of the expected timeline. 

  Mentor's Comments

Final Report

The project was successfully completed with the following outcomes:

1. An improved set of Unit Tests
2. Newly developed Smoke Tests and 
3. A working Continuous Integration Infrastructure.