TL;DR

• We use git-flow, but keep release branches around for minor versions as long term support branches.
• Go through the git-flow cheat sheet: http://danielkummer.github.io/git-flow-cheatsheet
• Create nice commits, be responsible, use review board to submit patches.
• https://git-wip-us.apache.org/repos/asf/cloudstack.git is master which is synced to github so it's recommended to fork on github.
• Up to 4.4.0 cloudstack used cherry-picking in favor of merging so there's a transition in progress.

Quick Start

Here is how to kick start feature development:

1. Create an account on GitHub.
   See Setting Up Git
2. Fork the Apache Repo:
   1. Go to https://github.com/apache/cloudstack.
   2. Click Fork.
      See Fork a Repo
3. Clone your fork:
       git clone --recursive https://github.com/<username>/cloudstack.git
4. If you use git-flow, initialize it with the defaults
       git flow init
5. Configure remote:
       git remote add upstream https://git-wip-us.apache.org/repos/asf/cloudstack.git
   The GitHub repo is a mirror and not the definitive source, so we use https://git-wip-us.apache.org/repos/asf/cloudstack.git and not https://github.com/apache/cloudstack
6. Do your work on a feature or bugfix branch
       git checkout -b <feature-branch>
or if you use git-flow
       git flow feature start <feature>
7. Sync from upstream periodically (for long running features, read:daily).
   This sets your local working set to the branch you want to pull the latest changes into:
       git checkout <feature-branch>
This fetches remote changes into the git index but not yet into the checkout:
       git fetch upstream
   This rewrites your local changes on top of the latest upstream:
       git rebase --ff-only upstream/<feature-branch>
   Don't rebase if you have pushed your local changes somewhere, instead merge:
       git merge upstream/<feature-branch> --no-ff
8. Submit your changes
   1. As a non-committer, when you have some code to contribute, submit changes to the Review Board
      1. Create an Apache Review Board account at https://reviews.apache.org
      2. Install RBTools
      3. Set the default reviewboard.url, e.g. git config reviewboard.url https://reviews.apache.org
      4. Call post-review
      5. Edit the resulting URL to assign a reviewing group and to describe the change.
         RBTools automates creating a diff file that captures the changes you've made and publishing them to the ReviewBoard site.
      6. Reviewers follow the Code Submission and Review Guidelines
   2. As a committer, you can follow the above process for larger commits, if you haven't gotten review from someone else yet. Otherwise, submit changes upstream
          git push upstream <feature-branch>
9. Finish the feature
   
   1. After testing of the new feature has finished, you can merge it back to develop
          git checkout develop
    git pull --rebase
    git merge --edit --no-ff <feature-branch>
    git push upstream develop 
   2. Once the feature has been integrated into develop (and any release branches), and integration tested, and once you are confident the feature branch won't be needed anymore, you can remove it locally
          git branch -d <feature-branch>
   3. git-flow combines steps a. and b. You can do this, but it assumes that integration testing has been done on the feature branch (see below)...
          git flow feature finish <feature>

Committers will not be looking at your forked repo by default. Therefore, as a non-committer, you should alert the cloudstack-dev mailing list of your work and periodically request the incremental commits you make be reviewed.

It is also best to work on feature branches rather than working directly on develop, even in your private fork. When the patch is ready for submission push the squashed patch through reviewboard. If the patch is sizeable and touches many projects in the repo, best to break it down into smaller chunks for easier and quicker review by multiple committers. Don't use github's pull request feature, but in the reviewboard patch notes, you can include the upstream you request a committer to pull from to preserve useful history.

Holler for help on the lists if you're stuck and ever need help.

Using Git in CloudStack

Git Config

# Set the name of the user for all git instances on the system
git config --global user.name "Firstname Lastname"
# Set the email-address of the user for all git instances on the system
git config --global user.email "your_email@youremail.com"

Create clean commits

Here are some key guidelines for how to make a clean commit. See the linux kernel patch guidelines for more.

It is more important to create clean commits than it is to follow all the branching rules. If your commits are clean, branching/merging mistakes can be resolved by looking at them. The reverse is not true.

Commit early and often, push periodically

Coming from centralized version control like cvs or svn, you may be used to committing you work only once a task is finished. An svn commit goes to the remote repository immediately, so committing work in progress breaks flow for your colleagues that then have to update their working copies. There can also be a race to get your work in first so that you don't have to be the one to sort out the merge.

Leave that mindset behind! Git is truly distributed version control, so you can and should commit without push. In a git workflow, your code-test-code cycle should become

• write some code
• test that code
• did the test work? Is my tree in a committable state?
  If so, commit it.

Then when you end a session (when you finish a task or feature, or you reach a milestone in its development), you think about pushing your changes.

• run git log to remember what commits you did
• run git rebase -i to clean up your history (optional; see below)
• run git commit --amend to clean up your commit message (see below)
• run git fetch upstream/...; git pull --rebase to replay your changes against the latest upstream
  • in the rare case of a conflict or problem, investigate and fix it
• run git push to send your commits upstream

Because we use feature branches, none of this involves scary merging. Only once a feature is complete does it get merged into the develop branch, and at that point you want to spend some time to check that your changes don't conflict.

Create idempotent commits

 Apply, basically, the same principles to commits that apply to source code.

Commits should do one thing and do that thing well.

Either move files, or refactor/rename some method, or fix license headers, or change the build, or change a version number in a build file. Don't combine multiple such things into one commit.

This helps make sure that the many different changes going on at the same time can be merged together without issues, and make it possible to go back in time to figure out what happened.

In the case of documentation files, it can make the job of the translation teams more complicated, as it becomes difficult for them to determine exactly what content changes need to be translated.

Commit Messages

Please make your commit messages descriptive. Avoid committing changes to many files in one go with a generic, vague message. Instead, commit each file (or small, related groups of files) with tailored commit messages.

A good commit message should look like this:

Header line: Explain the commit in one line of 50 chars

After a blank line, the body of commit message is a few lines of text, explaining things
in more detail, possibly giving some background about the issue being fixed, etc etc.
 
Use bullets if possible:
 - Line 1
 - Line 2 etc.
 * Start works too, instead of -

The body of the commit message can be several paragraphs. Use proper word-wrap
and keep columns shorter than 72. That way "git log" will show things
nicely even when it's indented.

Include the CLOUDSTACK-xyz jira issues related to this commit, if any, somewhere in the
message.
Reviewed-by: Individuals who reviewed, or link to review on review.apache.org
Reported-by: whoever-reported-it, if applicable (usually this is recorded in the Jira bug)
Submitted-by: give credit to other individual for any patch submission committed by you
Signed-off-by: Your Name <youremail@yourhost.com>

Note that there exists a git hook to prepare a commit form for you, located at tools/git/prepare-commit-msg. One simply needs to link to it in order to use it:

ln -s ../../tools/git/prepare-commit-msg .git/hooks/prepare-commit-msg

Please use a logical prefix, which should be:

• CLOUDSTACK-XXX-bug-id prefix: Explain the bugfix in one line; This is for only bug fixes.
• Maven: If this changes the build system
• Doc: If this is doc related change
• awsapi: If the code changes several things inside a subproject, use artifactId sans the cloud- prefix such as: plugin-netapp; this should be lowercase
• NameOfTheClass: If this patch purely changes to ThisClass, this should be CamelCase

Keep history

For a feature that takes several weeks to flesh out, these guidelines may result in dozens of commits. That's absolutely fine, git's implementation means that there is basically no cost at all to keeping all that history around. Having the history available of what it took to get to a finished feature is very useful; the branch/merge history for that future provides a summary of all those different commits.

For experimental work where you have to do a lot of undo/redo work, perhaps abandoning part of a solution completely and replacing it with another, you may choose to hide some of the 'mess' of that history, creating a cleaner alternate history, pretending you got everything right the first time. Cloudstack currently doesn't have a strong rule for or against that. If you want to do it, two key commands are git rebase -i and git checkout -p. When you do it, remember: do not rebase on branches you have pushed upstream.

Workflow

Be mindful of the branch that you are working in. Please use feature branches for bigger changes, and support/hotfix branches for bugfixes that have to go into multiple releases. Please be mindful that just because you have commit privileges does not relieve you of the obligation to develop in the open, which means communication on the cloudstack-dev list.

Using the git-flow command line tools is recommended but completely optional. Here's the git-flow cheat sheet: http://danielkummer.github.io/git-flow-cheatsheet/

Committer: handling patches

As a committer one of your responsibilities is to review and commit patches from others. The patches must pass through reviewboard, the bug tracker or the mailing list so they can be tracked and so they can be considered a CONTRIBUTION per the apache license/CLA.

Patches should be applied via git-am (in other words, don't just use a diff, it records no author information).

Reviewer guidelines

See the Code Submission and Review Guidelines.

Move to git-flow

Discussion thread | Vote thread

After the 4.4.0 release, cloudstack has decided to move to (a variant of) the git-flow branching model. The switch will be made on 7th august, 2014 TBD.  The setup is as follows:

This is the proposal for first cut

1. branch develop from master
2. branch release/4.5 from the develop
3. 4.3.x workflow does not change and continues to built using cherry-picks from 4.4/master/develop
   (4.3.1 should be last 4.3.x) 
4. 4.4.x workflow considers 4.4 the long-term support branch

Changes to continuous integration

Ideally the continuous integration setup will evolve so that it builds and tests all branches, including all feature and bugfix branches. However that may require more resources than are currently available.

So initially, mirroring current practice, CI should build the develop branch and all active release branches.

Creating a hotfix

A hotfix is a bugfix that is / has to be done to an existing release and/or release branch.

Hotfix for 4.4.x

Since 4.4 can't be merged to/from master, bugfixes to 4.4 need to be done twice, once for 4.4 and once for 4.5 and beyond:

1. branch hotfix/4.4-<jira-ticket> from 4.4
2. write fix
3. branch hotfix/<jira-ticket> from release/4.5
4. write/port fix (use git cherry-pick or do by hand)
5. push both fixes, integration test both
6. call for 4.4 release manager (Daan) to merge hotfix/4.4-<jira-ticket> into 4.4
7. TBC do all committers merge to release, one of

1. 1. call for the 4.5 release manager to merge hotfix/<jira-ticket> into release/4.5
   2. merge hotfix/<jira-ticket> into release/4.5 and push
2. merge release/4.5 into develop

For bugfixes that don't need to go into 4.5.x, normal git-flow can be followed.

Hotfix for 4.5.x and beyond

In git-flow it is customary to delete release branches after a release. We won't do that. Instead minor releases get a long-term support (LTS) release branch. This means it will be a bit easier to create patch releases for multiple releases.

Imagine we have released 4.5.0 and 4.6.0, and we want to do a hotfix that is to be part of 4.5.1 and 4.6.1:

1. branch hotfix/<jira-ticket> from release/4.5
       git flow hotfix start <jira-ticket> release/4.5  
2. write fix, smoke test, commit, push, integration test
       mvn -P ... install && ... smoke test ...
    git commit -a
    git push upstream hotfix/<jira-ticket>
3. merge to release/4.5
   TBC do all committers merge to release, one of

1. 1. call for 4.5 release manager to merge hotfix/<jira-ticket> into release/4.5
   2. merge hotfix/<jira-ticket> into release/4.5 and push
          git checkout release/4.5
    git pull --rebase
    git merge --no-ff --edit hotfix/<jira-ticket>
    git push upstream release/4.5
2. merge to release/4.6
   same procedure as for release/4.5 
3. merge release/4.6 to develop
    git checkout develop
    git pull --rebase
    git merge --no-ff --edit release/4.6
    git push upstream develop
while it should be possible to merge your hotfix branch into develop, the possibility exists that other hotfixes were already merged into the release but not yet into develop. If you merge your hotfix but not those previous hotfixes, comparing the history of release and develop branches gets a bit more difficult. So, we merge all the hotfixes on 4.6 to develop.

For bugfixes that don't need to go into 4.5.x, normal git-flow can be followed (see below).

Calling git merge with --no-ff ensures a merge commit. This commit documents the history that there was a specific hotfix.

Calling git merge with --edit allows editing the merge commit message. Use this to add any comments about the application of this hotfix to this release.

All fixes to micro releases (x.y.1, x.y.2, x.y.3, etc) must be brought in and merged as hotfixes. The only commits on micro releases that are made directly on the branch itself are those that involve making the release itself, i.e. changing the version number or the release notes.

Applying a hotfix that didn't start as a hotfix

If a bug got fixed on the develop branch, and then later on you decide that this fix has to make it into an already existing release branch, things are a little different. You do not want to merge from the develop branch into the release branch.

    DON'T EVER DO THIS
    git checkout release/4.5
    git merge develop

This use case is what cherry-picking is for . Use git log (or look at github, or ...) to find the SHA1 checksum of the bugfix commit(s) that need to go onto the release branch. Then, cherry-pick it

    git checkout release/4.5
    git pull --rebase
    git cherry-pick --edit abcdef12345
    git cherry-pick --edit bcdabc54321
    git push upstream release/4.5

Now, release/4.5 does need to be merge-able back into develop. This will often work out ok, but if there have been many changes to develop since the cherry-picked commits, it can be harder. It's a good idea to resolve this immediately.

    git checkout develop
    git pull --rebase
    git merge --no-ff --edit release/4.5
    git push upstream develop 

If that merge went without issues, you're done. Make sure to edit the merge commit message to explain what you did, since otherwise history will look just a tad weird (it will have two commits with the same commit message but different commit ids). If that merge has issues, it will need some thinking to resolve. If the conflict is just with these commits, you can probably just fix it by hand and finish the merge. If there's other problems...first abort the merge

    git merge --abort

then merge release/4.5 into develop up to but excluding the changes you just made

    git merge --no-ff --edit defabc543412 

then merge just your recent changes into develop, using merge strategy 'ours'

    git merge --no-ff --edit -s ours release/4.5

this says to resolve any conflicts by just choosing 'our' version of everything, where 'our' is the development branch. So, it is very important to apply this strategy to only the cherry-picked commits, because we do not want to miss merging any of the other hotfixes to the development tree.

...if that doesn't work, it is time to consult git merge --help, stack overflow, or the mailing list. Because you make all these changes locally, first, it is quite safe to try and experiment and see if you get the results you want. You can always roll back your changes locally. But, if things aren't working, it's best to ask for help (from a release manager, for example) rather than brute-force the change in.

Writing a new feature

See above under 'Quick Start'. This should be the default way of working for most changes.

Writing a new test for existing features

New features should come with tests and those should go onto the feature branch.

When improving test coverage for existing functionality, you should still create a feature branch if there's a lot to the test (moving code around, refactoring to reuse existing code, changes to test tools, etc). For simply adding a new test, see below.

Writing a new feature for a micro release

In the past, cloudstack has released new micro releases (x.y.1, x.y.2) that contain new features or that enable previously enabled features.

In the git-flow model, that ideally shouldn't happen. Rather the ambition should be to make the release cycle short and predictable enough that there is no pressure to push features into micro releases. Version numbers are cheap; feasible release management is more important than the psychology of the version number.

However git-flow provides a workflow model and getting to that way of working is a transition that will take time, and investment into continuous integration.

During the transition, if a decision is made to add new features to an existing release branch, the technical process for doing so is either

• for features created on develop and then backported, basically the same as above under 'applying a hotfix that didn't start as a hotfix';
• for features that were decided at development to be part of a micro release, basically the same as above under 'hotfix for 4.5 and beyond'

Following this route should be considered an exception to the rule, and should be discussed in the community. TBC a policy for when it is ok to do this.

Simple bugfix or improvement or test case

Fixing a simple bug that

• does not need to be backported to any other release
• consists of one reasonable-sized (reviewable) commit
• is unlikely to destabilize the build

Can be done directly on develop.

However, since when you start the bugfix you typically don't know exactly what the fix will be, it's simply good practice to create a local bugfix branch anyway. Basically, whenever you do a context-switch to pick up a new task or JIRA issue, you probably want to branch. The workflow difference in this case is that the branch you create is local to your machine; you never push it upstream. For example:

Start work:

    git checkout develop
    git pull --rebase
    git checkout -b bugfix/<jira-ticket>
    ...write test...
    git commit -a
    ...code code code...
    git commit -a
    ...test fix test...
    git commit -a

Check that you're done:
    mvn -P ... install && ... smoke test ...

Cleanup and make neat clean commit(s)

    git pull
    git rebase -i develop
    git commit --amend
    git checkout develop
    git merge --ff-only bugfix/<jira-ticket>
    git branch -d bugfix/<jira-ticket>

Push the changes:

    git push upstream develop

Release management

Besides the normal git-flow commands, remember to update version numbers (in maven pom.xml, in marvin setup.py, in systemvm scripts, ...should have a confluence page...).

Starting a new x.y.0 minor release

    git checkout develop
    git pull --rebase
    git flow release start x.y develop
    ...update version number to x.y.0-SNASPHOT...
    git push upstream release/x.y
    git checkout develop
    ...update version number to x.y'-SNAPSHOT...
    git commit -a
    git push upstream develop

Finishing a x.y.0 release

Don't use git flow release finish, it will delete the release branch.

    git checkout release/x.y
    git pull --rebase
    ...update version number to x.y.z...
    git push upstream release/x.y
    git checkout master
    git pull --rebase
    git merge --no-ff --edit release/x.y
    git tag -a vx.y.0
    git push upstream master
    git push --tags
    git checkout release/x.y
    ...update version number to x.y.z'-SNAPSHOT...
    git push upstream release/x.y 

Freezing a release

When a release branch nears release status, the release manager can declare the branch frozen. Once the branch is frozen, no-one but the release manager(s) should do any merges (or any other commits) to the branch. Committers can still create and nominate bugfix branches to merge into the release branch, but from this point on they should be extra careful to only work on those bugs for which it has been agreed they will still get fixed in that release. That agreement is reached on the development mailing list. For example, the agreement may be that fixes for JIRA blockers with a fix version of that release will be accepted, and nothing else.

Starting a minor x.y.z (patch/hotfix) release

This is very similar to starting minor release, but you start from the release branch, not from the develop branch.

    git checkout release/x.y
    git pull --rebase
    git flow release start x.y.z release/x.y
    git push upstream release/x.y.z
    git checkout release/x.y
    ...update version number to x.y.z'-SNAPSHOT...
    git push upstream release/x.y

Finishing a minor x.y.z (patch/hotfix) release

This is very similar to finishing a minor release. You shouldn't have to update version numbers in release/x.y in this case (since you did that at patch release start; see above), but it's probably good to check.

    git checkout release/x.y.z
    git pull --rebase
    ...update version number to x.y.z...
    git push upstream release/x.y.z
    git checkout master
    git pull --rebase
    git merge --no-ff --edit release/x.y.z
    git tag -a vx.y.z
    git push upstream master
    git push --tags
    git checkout release/x.y
    ...version number is already at x.y.z'-SNAPSHOT...

Aborting or cleaning up

While changes are only in your local repository, you can do many different things to undo, amend, or rewrite commits (see git reset --help). However, once commits / tags / branches are pushed to upstream, there is no more real 'undo' and the only direction for change is forward.

To undo (a) commit(s) then, is to apply the inverse of that/those commit(s). For simple cases, see git revert --help. For complex cases, amazing amounts of tips are found on stackoverflow and elsewhere. Here's one reference.

When you have to do any work like this, the most important is that you write good commit messages detailing what you intend to do. That way, if something goes wrong while cleaning up, others can help unpick the problem.

Patches

Patches encapsulated all the changes you want to make to another branch. Patches are used to communicate the changes that you want incorporated into Apache CloudStack.

Creating patches

Patches should be created with git-format-patch - and you should specifically be basing that against the branch you are targeting for inclusion, so for instance using our above example from the <feature-branch> branch we'd run:

$ git format-patch -s develop

The above would produces a patch file for each commit in the directory (unless you specified one with -o /path/to/patch/folder).
If you've a series of commits that you want to put into a single file, try

$ git format-patch -s develop --stdout > ./mycommits.patch

Sending patches

CloudStack currently doesn't utilize Github pull requests. We ask that you post patches on Review Board.

1. Create an Apache Review Board account at https://reviews.apache.org
2. Install RBTools, for example if you already have a recent python installed
   easy_install RBTools
3. Configure reviewboard details by writing a .reviewboardrc, for example
   touch .reviewboardrc
chmod go-rwx .reviewboardrc
cat >>.reviewboardrc <<END
REPOSITORY = "cloudstack-git"
REVIEWBOARD_URL = "https://reviews.apache.org/"
USERNAME = "your-review-board-username"
PASSWORD = "your-review-board-password"
OPEN_BROWSER = True
TARGET_GROUPS = "cloudstack"
REPOSITORY_TYPE = "git"
# TRACKING_BRANCH = upstream/develop
END 
4. Post your patch, for example
   rbt post --tracking-branch=upstream/develop
5. Edit the resulting review URL to assign a reviewing group and to further describe the change.
   If you know a person or people that's likely to know the area of the codebase your patch applies to, add them as a reviewer.

RBTools automates creating a diff file that captures the changes you've made and publishing them to the ReviewBoard site. The drawback is that you loose the meta data associated with each commit that you've made. Howevever, Apach Review Board can't handle a commit that references the same file multiple times.

We can and occasionally do accept patches send via e-mail to the development list, or patches attached to JIRA issues, but Review Board is the standard. Please use it.

For big changes that involve multiple commits, review board unfortunately does not support patch sets. When sensible, it's preferable to create multiple reasonably-sized, reviewable commits, and multiple reviews on review board. If you've made really small/clean commits, that may be too much hassle for you and/or the reviewer. Don't spam reviewboard! In those cases, please do submit the squashed patch using post-review anyway, but then edit the description to include a link (i.e. to a branch in a cloudstack fork on github, or to a git am mbox somewhere) for communicating the full history.

Apply patches

 If the patch to be applied gets the commit details in the header, then use

• git am < <patch> to apply the patch, which will automatically add newly added files to tracked files.
• git commit -as --author='Foo Bar <foo@bar.com>' for proper commit attribution

If it's just a raw patch, then use

• git apply <patch>
• git add <new files>
• git commit -as --author='Foo Bar <foo@bar.com>' for proper commit attribution

Words of wisdom

Never use the --force

Git assumes that if you have commit privileges that you are to be trusted, and generally this is a good thing. However, occasionally people are lured by the dark side of the force and when something doesn't merge cleanly, or their push doesn't work - they are tempted to whip out the --force and make it work. PLEASE NEVER DO THIS. It's is almost universally wrong. Please tell us about your problem on cloudstack-dev and let us help you fix it. 

Line endings

The coding standard for CloudStack says that unix line endings (LF) are used instead of CRLF (Windows line endings). You should set your editor and git configuration to behave properly. 

For more information: 

http://help.github.com/line-endings/

http://git-scm.com/docs/git-config

Fixing things for folks

In short, please don't. If non-committers have a patch 95% of the way - comment and tell them what is necessary to make the patch acceptable. Let them fix their own patch and resubmit. Yes you can probably fix things more quickly, but it's important for community growth (both in number and experience) that folks do this themselves.

git hooks

To install the git hooks, download and put them in the $CLOUDSTACK_SOURCE_ROOT/.git/hooks

chmod u+x may be required.

more information about git hooks is available @ http://git-scm.com/book/en/Customizing-Git-Git-Hooks

1. pre-push-hook -> pre push hook to reject any direct commits other than merges to master