This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • Guidelines for test developers
Skip to end of metadata
Go to start of metadata

Every developer who is adding test cases is expected to follow below guidelines. These guidelines will help the regression and basic validation tests to be of highest quality and it will eliminate the need to modify the test cases or fix them later in time. Some of the old test cases do not abide by below guidelines and efforts are going on to make them consistent. But it is important that any new test addition should not be done by copying any old test case and modifying it. That will add to inconsistencies.


Before adding pull request/patch, please go through below guidelines and make sure your tests follow these rules.


1. Import * should always be avoided. The imports must be specific.
When the imports are specific, it eliminates the possibility of test case failure due to invalid import when specific import is removed from dependent module.
E.g. If your test case has following import.
from A import *
And it uses time module which is not imported explicitly in test case and comes from module A. Then the test case will start failing when "import time"is removed from module A. You certainly don't want this to happen.

2. Maintaining Pep8 standards for python code.

The code is read more often that it is written. Pep8 standards improve the readability of the code making it consistent in style. There is a tool named "autopep8" which you can install with pip install and then you can run following command on your test file.
autopep8 -i -a -a
This will make the file pep8 consistent and will also remove the white spaces. But some issues need human intervention and can't be fixed with tool. For fixing those, check the pep8 issues with "pep8" and fix manually.

3. Keep only imports which are used in the test suite and remove unwanted imports.

4. Keep all the configuration parameters (such as data which is passed to API while creating Network offering, Service offering, account etc...) in tools/marvin/marvin/config/ file. Don't include them in test suite itself.

Many of the dictionaries are reusable and if you are adding a new test, there are only a few dictionaries you will have to add in the file.
If any of the data contains URLs or any data which should be changed according to setup/env, then include the dict in "configurableData" section in file. This makes it easier to identify which data needs to be dynamic according to the setup and which data doesn't need to be touched when env is changed.

5. Before committing a test case, run it with the latest branch against which you are adding the test case and attach the results in Pull Request. If in case change is very small, and doesn't need to be run, then at least check syntactical errors with python command and also with the help of tools such as pyflakes.


6. If you add a new function in your test case and you think it can be used in future by other test cases, then please add that function to common or utils file in Marvin. Don't keep it local to test case. This will prevent multiple contributors adding same functions in their test case to achieve a particular goal.

7. Please make sure all the resources created through the test cases are deleted when test case execution completes, or even when the test case fails.


8. If same test case is to be run with different configuration or setting, you can make use of ddt library. For example, if you have added test case for isolated networks, and you need to run the same code for shared and VPC networks, then you don't need to add 3 test cases. Just add relevant tags to the test case and you are good to go. Although you will need to write code for handling those tags. It is already used in few test cases. A simple grep over component folder and you can see how it is used.

This blog explains how it works.
9. Try to use base library methods for API calls, instead of calling API methods directly from the test case.
E.g. When you need to stop a Virtual Machine, then you can do this in two ways.
You have "virtual_machine" as object of VirtualMachine in class in
from marvin.cloudstackAPI import stopVirtualMachine

cmd = stopVirtualMachine.stopVirtualMachineCmd() =






Second way (b) is simpler and should be preferred because

  • Has lesser number of lines
  • Less ambiguous
  • Makes use of existing code
  • Easier to maintain


There was an instance when after stopping the VM, the virtual machine state was asserted to be Stopped by listing the VM explicitly, after calling stop() method. This code was in many test cases and to remove the redundant code, verifications steps were made as a common part of stop() method. This was a huge boost in terms of maintenance of code and removing duplicate lines. For availing such advantages, always prefer method (b).

This is possible when you have an object of the base library class. In case you don't have the object, then you can use method (a).


Feel free to add more guidelines or let the community know if you don't have edit access.

  • No labels