Local History Operations for Subversion
References:
issue SVN-3625 "Commit shelving"
issue SVN-3626 "Commit checkpointing"
SavePoints wiki page
The Problem
This is the problem we are looking to solve. We will use this to determine the scope of design and implementation.
Shelving
While developing one change, you need to stop and work on an urgent fix or you want to commit a quick and easy change or try a different approach in the same code. You need to store the unfinished work and return to it later. Existing options: server-based branches, patch files, extra Working Copies.
Priorities: fast, offline.
Checkpointing
While developing a change, you reach a state which you want to save. A test passes or a subtask is complete, and the next thing you do might break it. If so, you will want to roll back to this state. When the work is finished, you will commit it as usual, or if it did not work out you may discard it all.
Priorities: fast, offline.
Future extension possibilities:
roll back and forth (undo/redo) along the series, e.g. to "bisect" for a bug
checkpoint automatically, e.g. before 'update' (like auto-versioning, auto-save, undo systems)
view your working state as a diff against the last checkpoint, instead of against the original base version
The Solution
Shelving
One-click patch management on top of 'svn diff' and 'svn patch'.
similar to "git stash"
backward-compatible extension to WC format
(old clients still work, just not seeing the shelves)
supports exactly the same kinds of change as 'svn patch' and 'svn diff'
we will extend these later to cover currently missing kinds of change
Functions:
shelve: save a WC diff as a patch and revert it from WC
unshelve: apply specified patch to WC and delete the patch
list shelves and the files that they affect
delete a shelf
Compared to manual patch management:
one-click simplicity
don't need to choose a filename or remember where it is
still can't save & restore WC base state (in v1)
Future upgrades:
checkpointing, by storing multiple versions of each shelf
supporting the initially unsupported kinds of change (e.g. binary content, copies, moves, directories)
integration with the "changelist" feature
A note about "reliability". 'svn diff' and 'svn patch' have historically been incomplete with well known omissions (for example: binary content, copies, moves, directories) and incompatibilities. Developing a 'shelving' feature based on diff and patch will force us to adopt the mindset that diff and patch must interoperate reliably, and so fix those deficiencies. The result will be that using 'svn diff' and 'svn patch' manually will then be a reliable solution on its own for those who have reason to continue using it. The additional 'shelving' user interface will remain an added convenience.
Checkpointing
Simple management of a series of versions of a patch. Each patch version captures the WC state relative to the original WC base revision. The kind of checkpointing initially described in issue SVN-3626 "Commit checkpointing". Accessed by new 'svn' commands.
Aim: to streamline the manual method, which I use myself, of using 'svn diff' and 'svn patch' to develop successive versions of a patch, named 'feature-v1.patch', 'feature-v2.patch', etc.
Main functionality:
save the WC modifications as a new version of a shelf (don't revert);
roll back to an intermediate state and re-work from there;
discard all versions of the patch.
To roll back to an intermediate state, the user will first 'revert' the modifications currently in the WC and then use a 'restore' command to apply a chosen version of the patch.
The user can use the regular 'commit' functionality to commit the final version of the work, or can restore an earlier version and commit that.
Existing features (such as diff) will not directly act on checkpoints. All existing commands operate against the original base revision. Especially:
'commit' commits current WC local mods as usual
('commit' offers to discard all the checkpoints?)
'update' updates the WC as usual
'update' does not rebase or do anything special to checkpoints
'revert' reverts to the original base
Key points:
key benefit: roll back to a checkpoint
key drawback: cannot (initially) view current work against last checkpoint
backward-compatible extension to WC format
(old clients still work, just not seeing the checkpoints)
Feature/Benefit Matrix; Comparison with Existing Options
Shelving (& Checkpoints) | server branch | patch files | extra WCs | Shelving (patches) | Checkpt |
works offline | no | yes | yes | yes | yes |
one-click simplicity | no | no | no | yes | yes |
fast | no | yes | medium | yes | yes |
low disk space requirement | yes | yes | no | yes | yes |
incremental re-build * | yes | yes | no | yes | yes |
save & restore WC base state | no | no | yes | no | no |
save & restore uncommittable state (conflicts, etc.) | no | no | yes | no | no |
can discard the entire work | no | yes | yes | yes | yes |
back-compatible WC format | n/a | n/a | n/a | yes | yes |
Checkpoints (additional) | server branch | patch files | extra WCs | Checkpt |
roll back to any checkpoint | yes | yes | yes | yes |
commit as a whole | yes | yes | yes | yes |
commit all the separate steps | yes | awkward | awkward | awkward |
incremental view in WC | yes | no | no | no |
automatic checkpoints to enable rolling back a conflicting update * | no | no | no | no |
* Note: "incremental re-build" means it touches only the relevant source files in the working copy, so in a typical compiled language build system only the modified files would be re-compiled.
* Note: "Automatic checkpoint" is called "Destructive command rollback" in "Design: SavePoints" [5].
Role Models
With Shelving, we are catching up with a feature that most current VCSs already provide. We should make it easy for users to apply what they have learnt in those systems, and avoid needless differences. We should also support Subversion's unique characteristics such as directories being versioned objects.
Headings are linked to corresponding documentation.
SHELVING | ||||||
---|---|---|---|---|---|---|
general | well integrated | simple patch | ||||
stored where? | local repo | WC/.hg/ | WC/.bzr/ | the depot (shareable) | local (project dir by default) | ~/.netbeans/ |
stored how? | special commit | git patch | special commit? | special commit (changelist) | svn+git patch | patch + MIME binary support |
changelist integration | n/a | tight: a shelf is a p4 change | cl <==> shelf strong association | no | ||
pop vs. apply | pop or keep | backup or keep | pop or keep | pop | backup or keep | pop or keep |
access in other cmds | yes | diff,diff2,files,print | (no) | (no) | ||
merge & conflicts | regular merge | temp. commit + merge + continue | regular merge |
Stashing can also be achieved in a DVCS, in general, by creating a local branch and committing the local change there and then switching back to the previous branch; later being able to merge the change or switch back to it and (depending on the system) throw away that commit.
CHECK- POINTING | DVCS local branch | |||
---|---|---|---|---|
general | normal commits | manage a patch series: combine, split, rebase | simple: text changes; size limit | simple: text changes; limited undelete |
The Design: Shelving & Checkpointing (for Subversion 1.11?)
(For earlier versions, see Shelving-v1 in Svn-1.10.)
A shelf is like a set of changes moved out of the WC into a special "shelf" storage area.
Saving a checkpoint is like creating a new shelf that is related to a previous one, having the same name and a different version number.
Supports working on multiple independent change-sets at the same time, by specifying the WC path(s) to be saved, as long as each WC path only belongs to one change-set at a time.
Functionality
Shelving operations:
shelve
move selected changes from WC to a named shelf
reverts the successfully shelves from the WC
unshelve
copy changes from specified (or newest) shelf to WC
can apply to any revision or branch
list shelves
delete a shelf
Checkpointing operations:
save a new checkpoint
svn shelf-save NAME [PATH...]
similar to 'shelve' except doesn't revert the working copy
restore an older checkpoint, discarding all newer versions
svn shelf-restore NAME [VERSION]
assumes the working copy has already been reverted
similar to 'unshelve' except can specify an older version
list the versions of a shelf
svn shelf-log NAME
output a shelf version as a patch
svn shelf-diff NAME [VERSION]
Other:
- optional log message (and other revprops) per shelf
- shelves stored in WC metadata dir '.svn/shelves/'
(The command-line syntax shown here is just for illustration.)
Implementation
- in libsvn_client
- supported in 'svn' command-line UI
- supported in TortoiseSVN (for Windows), Cornerstone (for Mac)
Kinds of change that can be shelved
WC State or Change | Minimal (in Svn 1.10) | Reasonable (for Svn 1.11?) | Comprehensive (possible future?) |
---|---|---|---|
committable changes | |||
file text, file delete/add, most properties | yes | yes | yes |
mergeinfo changes | yes | yes | yes |
copies and moves | no | yes | |
directories (mkdir/rmdir/...) | no | maybe | yes |
binary files & properties | no | yes | yes |
uncommittable state | |||
unresolved conflicts | no | no | yes |
changelist assignment | no | no | yes |
unversioned items (git stash -u/-a) | no | no | no |
inconsistent WC states (missing/obstructed) | no | no | no |
base state | |||
WC base state (rev, URL, switched, depth) | no | no | yes |
Shelving and Checkpointing Commands
Command-Line UI Design:
- see Shelving Command-Line UI Design [2]
Shelving and Checkpointing together have the following generic interface. X is a shelf name.
operation | CLI syntax example | recipe |
---|---|---|
shelve and revert | svn shelve X [PATH...] | assert(any PATH is modified) patch-save(X, PATH...) # creates a new version patch-unapply(X) |
save new version ('checkpoint') | svn shelf-save X [PATH...] | assert(any PATH is modified) patch-save(X, PATH...) |
restore X [N] unshelve X | svn unshelve X [N] | assert(not any patch-get-paths(X) is modified) # or warn or allow override patch-apply(X, N) patch-delete-versions-newer-than-N(X, N) |
export to patch | svn shelf-diff X N | # export version N of shelf X as a patch |
delete shelf | svn shelf-drop X | patch-delete-all-versions(X) |
list shelves | svn shelf-list svn shelves | list-all-shelves() |
list shelf versions | svn shelf-log X | list-shelf-versions(X) |
Commit Log Message
A log message, and any other revision properties, can be attached to a shelf (not to each version).
Discussion
Ability to include a short description, or a longer commit log message, could be helpful both per shelf and per checkpoint.
In some situations a user will want to attach a carefully written log message to a shelf. In other cases, a user will want to quickly shelve or checkpoint their current working state without thinking about a description. For checkpointing, this would be rather like auto-save in a word processor -- and in a GUI could be performed automatically based on time intervals or before/after certain events, even as far as checkpointing before/after every svn mkdir/delete/move/copy, etc.
We need to be prepared to support both styles of working. A shelf may very well contain both some carefully constructed checkpoints (with log messages) and many of the auto-save kind (without, or with simple auto-generated messages). Therefore we should allow a log message but also make it easy for the user to distinguish checkpoints where no message was given. To help with the latter, we can display metadata such as the date (or age) of the checkpoint and a short summary of what changes it contained. (The command-line client prototype currently does that.)
If a user stores a big, carefully written log message in a shelf, then they will want an easy way to transfer this to the final commit log message. Until that is made easy, the UI should perhaps encourage the user to write only a short message.
Path spec [PATH...]
- applies only to 'save' commands (shelve, checkpoint save)
- defaults to "." like in most svn commands
- is restrictive (restricts operation to PATH...)
- is not tracked (the shelf doesn't remember what top-level path(s) you specified, or other associated inputs, it only remembers which individual paths had changes)
- clashes (same path in more than one applied patch) are not managed
Extensions to Consider
These possible extensions are not supported initially.
allow showing a shelf's content -- like 'svn log [-v] [--diff]', 'svn status', 'svn diff [--summarize]'
shelve: if no name given, automatically generate a name (so more like 'git stash')
use shelf's log message when committing
see the "Commit Log Message" discussion section above
consider allowing restricting paths on 'apply' commands
when checkpointing, warn if PATH... excludes any paths that were in the previous version
- once the base state is recorded: warn or otherwise try to avoid accidentally unshelving to a different branch or unrelated path
Roll Forward, as in Undo/Redo
As a starting point, rollback is destructive, deleting versions newer than the target version. As an enhancement, it could be made to keep the newer versions and allow rolling forward to them. It could operate like the 'undo stack' model commonly found in editing applications, where roll-forward (often named 'redo') is possible up until a different change is saved to the stack, at which time the possibility is lost.
Rebasing Checkpoints
There are cases where it is potentially desirable to be able to rebase a series of checkpoints when updating (pulling in new changes from the repository). One case where this would be useful is if the user updates, goes offline, then wishes to roll back to a checkpoint saved before the update. Another is if the user wishes to make separate commits from several checkpoints in the series, some of which were saved before the last update; but that scenario is not a supported use case in this design.
In trunk-based development usually other people's changes have to be pulled before we can commit. In branch-based work flows that is uncommon.
Whenever we try to restore a checkpoint that was made against an older base, there is a chance that the patch application will run into conflicts. If and when we upgrade from traditional patch application to 3-way merge, conflicts would be reduced, but still possible. Rebasing a series of checkpoints would involve merging once into each checkpoint, with the possibility of conflicts at each step.
As conflicts require tedious manual recovery, these scenarios are best avoided as far as possible.
Rebasing a single checkpoint needs no special support. It is achieved by applying the checkpoint to the WC and then running 'svn update'. The result can then be saved as a new checkpoint or committed.
Therefore this potential extension is low priority.
Integrate Shelving with Changelists
I hate to add complexity without simplifying something at the same time. Considering the relationship between shelves and changelists:
a changelist is a named set of file-paths in the WC
a shelf is a named set of changes that is not currently applied to the WC
Integrate them like this:
shelves and changelists share the same namespace
shelving means converting a changelist to a shelf of the same name
- or, if the desired changes are not already managed in a changelist, specify them explicitly
unshelving means converting a shelf to a changelist of the same name
Benefits
- managing shelves is easier:
- after unshelve, even if there are other changes elsewhere in the WC, a changelist will remember which changes belonged to that shelf, so you can later shelve or "checkpoint" it without having to specify which files to include
clear and simple relationship with no overlap
changelists remain backwards-compatible
Main issues
a changelist can include unmodified paths:
should a shelf also include unmodified paths?
a path can appear in multiple shelves, but only in one changelist:
what to do when unshelving if a path clashes?
any extensions should apply uniformly to both shelving and changelists
if a shelf supports a log msg, a changelist should support a log msg too; this would be a good enhancement for changelists anyway even without shelving
Role models
changelists in IntelliJ IDEA (are unified across svn, git, hg, etc.)
changelists in Perforce (are new or pending or shelved or submitted)
Extensions / Not Supported Initially
When committing changeset X, Subversion could (offer to) delete the shelf X.
[with roll forward, as in undo/redo] Automatically save a new checkpoint before rollback.
GUI Considerations
GUI priorities tend to be a little different from a CLI.
APIs that do the job simply and reliably
progress indication and cancel, for any long-running ops
hopefully we won't have any long-running ops
ability to undo
we don't have to implement undo, just provide enough hooks so caller can
Complete WC State Shelving
A key concern will be to have APIs that do the job simply and reliably. Such as: ability to shelve a WC state whatever state it's in (all the odd states like unresolved conflicts, etc.) and restore to exactly the same state.
We should have two complementary APIs:
a "shelve" API should visit the WC reporting absolutely everything about the state, in such a way that the caller can capture that state;
a "diff" API should report the subset of this that is committable;
an "unshelve" API should be able to recreate any such state, using input entirely provided by the caller
a "patch" API should accept the subset of this that is committable.
To what extent do such APIs exist?
We should write automatic round-trip testing for diff-&-patch and for status-&-modify.
For states that we can't shelve, a GUI wants to know in advance that this is the case (to grey-out the button), rather than the operation to error out part way through. So we want:
an API to efficiently tell whether the WC state can be shelved (or why not)
Existing Issues to Overcome
... are tracked in https://issues.apache.org/jira/browse/SVN-3625 "commit shelving".
Glossary of Terms
shelf: a place of storage, within WC metadata, for: a time-ordered series of change-sets; and metadata that applies to the shelf as a whole, modifiable at any time, including a "log message"
save: to copy a change-set from WC working state to a shelf (reverting it from the working state)
shelve: to move a change-set from WC working state to a shelf (reverting it from the working state)
unshelve: to copy a change-set from a shelf into the WC working state; implies choosing the newest version of the change-set on that shelf but can take any version
restore: to copy a change-set from a shelf into the WC working state; implies choosing an older version of the change-set on that shelf but can take any version
version: one of the change-sets in a shelf; versions are assumed to be different (e.g. improving) implementations with the same intended purpose
change-set: a set of changes that Subversion could potentially record in a single revision (changes to files, directories, properties); sometimes refers also to associated revision metadata (log message, other revision properties).
A change-set may be shelved, unshelved, presented as a diff (patch file), etc.
patch: similar to change-set; implies formatted as a patch file
change: loose term, referring to a change-set or any change that a change-set could contain
changelist: similar to change-set; implies the existing 'svn changelist' feature
Subversion's existing 'changelist' should grow into the full 'change-set' concept.
checkpoint, savepoint: non-preferred term for (noun) "version", (verb) "save"
'savepoint' may be a little better than 'checkpoint' -- better associated with the command verb 'save' and does not share the prefix 'check' of 'checkout' nor the abbreviation 'cp' of 'copy'.
Julian Foad, Assembla, 2017