Local History Operations for Subversion

References:

1. Shelving and Checkpointing Dev 

2. Shelving Command-Line UI Design 

3. issue SVN-3625 "Commit shelving"

4. issue SVN-3626 "Commit checkpointing"

5. SavePoints wiki page 

The Problem

This is the problem we are looking to solve. We will use this to determine the scope of design and implementation.

Shelving

While developing one change, you need to stop and work on an urgent fix or you want to commit a quick and easy change or try a different approach in the same code. You need to store the unfinished work and return to it later. Existing options: server-based branches, patch files, extra Working Copies.

Priorities: fast, offline.

Checkpointing

While developing a change, you reach a state which you want to save. A test passes or a subtask is complete, and the next thing you do might break it. If so, you will want to roll back to this state. When the work is finished, you will commit it as usual, or if it did not work out you may discard it all.

Priorities: fast, offline.

Future extension possibilities:

• roll back and forth (undo/redo) along the series, e.g. to "bisect" for a bug

• checkpoint automatically, e.g. before 'update' (like auto-versioning, auto-save, undo systems)

• view your working state as a diff against the last checkpoint, instead of against the original base version

The Solution

Shelving

One-click patch management on top of 'svn diff' and 'svn patch'.

• similar to "git stash"

• backward-compatible extension to WC format

• (old clients still work, just not seeing the shelves)

• supports exactly the same kinds of change as 'svn patch' and 'svn diff'

• we will extend these later to cover currently missing kinds of change

Functions:

• shelve: save a WC diff as a patch and revert it from WC

• unshelve: apply specified patch to WC and delete the patch

• list shelves and the files that they affect

• delete a shelf

Compared to manual patch management:

• one-click simplicity

• don't need to choose a filename or remember where it is

• still can't save & restore WC base state (in v1)

Future upgrades:

• checkpointing, by storing multiple versions of each shelf

• supporting the initially unsupported kinds of change (e.g. binary content, copies, moves, directories)

• integration with the "changelist" feature

A note about "reliability". 'svn diff' and 'svn patch' have historically been incomplete with well known omissions (for example: binary content, copies, moves, directories) and incompatibilities. Developing a 'shelving' feature based on diff and patch will force us to adopt the mindset that diff and patch must interoperate reliably, and so fix those deficiencies. The result will be that using 'svn diff' and 'svn patch' manually will then be a reliable solution on its own for those who have reason to continue using it. The additional 'shelving' user interface will remain an added convenience.

Checkpointing

Simple management of a series of versions of a patch. Each patch version captures the WC state relative to the original WC base revision. The kind of checkpointing initially described in issue SVN-3626 "Commit checkpointing". Accessed by new 'svn' commands.

Aim: to streamline the manual method, which I use myself, of using 'svn diff' and 'svn patch' to develop successive versions of a patch, named 'feature-v1.patch', 'feature-v2.patch', etc.

Main functionality:

• save the WC modifications as a new version of a shelf (don't revert);

• roll back to an intermediate state and re-work from there;

• discard all versions of the patch.

To roll back to an intermediate state, the user will first 'revert' the modifications currently in the WC and then use a 'restore' command to apply a chosen version of the patch.

The user can use the regular 'commit' functionality to commit the final version of the work, or can restore an earlier version and commit that.

Existing features (such as diff) will not directly act on checkpoints. All existing commands operate against the original base revision. Especially:

• 'commit' commits current WC local mods as usual

• ('commit' offers to discard all the checkpoints?)

• 'update' updates the WC as usual

• 'update' does not rebase or do anything special to checkpoints

• 'revert' reverts to the original base

Key points:

• key benefit: roll back to a checkpoint

• key drawback: cannot (initially) view current work against last checkpoint

• backward-compatible extension to WC format

• (old clients still work, just not seeing the checkpoints)

Feature/Benefit Matrix; Comparison with Existing Options

* Note: "incremental re-build" means it touches only the relevant source files in the working copy, so in a typical compiled language build system only the modified files would be re-compiled.

* Note: "Automatic checkpoint" is called "Destructive command rollback" in "Design: SavePoints" [5].

Role Models

With Shelving, we are catching up with a feature that most current VCSs already provide. We should make it easy for users to apply what they have learnt in those systems, and avoid needless differences. We should also support Subversion's unique characteristics such as directories being versioned objects.

Headings are linked to corresponding documentation.

Stashing can also be achieved in a DVCS, in general, by creating a local branch and committing the local change there and then switching back to the previous branch; later being able to merge the change or switch back to it and (depending on the system) throw away that commit.

The Design: Shelving & Checkpointing (for Subversion 1.11?)

(For earlier versions, see Shelving-v1 in Svn-1.10.)

A shelf is like a set of changes moved out of the WC into a special "shelf" storage area.

Saving a checkpoint is like creating a new shelf that is related to a previous one, having the same name and a different version number.

Supports working on multiple independent change-sets at the same time, by specifying the WC path(s) to be saved, as long as each WC path only belongs to one change-set at a time.

Functionality

Shelving operations:

• shelve

  • move selected changes from WC to a named shelf

  • reverts the successfully shelves from the WC

• unshelve

  • copy changes from specified (or newest) shelf to WC

  • can apply to any revision or branch

• list shelves

• delete a shelf

Checkpointing operations:

• save a new checkpoint

• svn shelf-save NAME [PATH...]

• similar to 'shelve' except doesn't revert the working copy

• restore an older checkpoint, discarding all newer versions

• svn shelf-restore NAME [VERSION]

• assumes the working copy has already been reverted

• similar to 'unshelve' except can specify an older version

• list the versions of a shelf

• svn shelf-log NAME

• output a shelf version as a patch

• svn shelf-diff NAME [VERSION]

Other:

• optional log message (and other revprops) per shelf
• shelves stored in WC metadata dir '.svn/shelves/'

(The command-line syntax shown here is just for illustration.)

Implementation

• in libsvn_client
• supported in 'svn' command-line UI
• supported in TortoiseSVN (for Windows), Cornerstone (for Mac)

Kinds of change that can be shelved

Shelving and Checkpointing Commands

Command-Line UI Design:

• see Shelving Command-Line UI Design [2]

Shelving and Checkpointing together have the following generic interface. X is a shelf name.

Commit Log Message

A log message, and any other revision properties, can be attached to a shelf (not to each version).

Discussion

Ability to include a short description, or a longer commit log message, could be helpful both per shelf and per checkpoint.

In some situations a user will want to attach a carefully written log message to a shelf. In other cases, a user will want to quickly shelve or checkpoint their current working state without thinking about a description. For checkpointing, this would be rather like auto-save in a word processor -- and in a GUI could be performed automatically based on time intervals or before/after certain events, even as far as checkpointing before/after every svn mkdir/delete/move/copy, etc.

We need to be prepared to support both styles of working. A shelf may very well contain both some carefully constructed checkpoints (with log messages) and many of the auto-save kind (without, or with simple auto-generated messages). Therefore we should allow a log message but also make it easy for the user to distinguish checkpoints where no message was given. To help with the latter, we can display metadata such as the date (or age) of the checkpoint and a short summary of what changes it contained. (The command-line client prototype currently does that.)

If a user stores a big, carefully written log message in a shelf, then they will want an easy way to transfer this to the final commit log message. Until that is made easy, the UI should perhaps encourage the user to write only a short message.

Path spec [PATH...]

• applies only to 'save' commands (shelve, checkpoint save)
• defaults to "." like in most svn commands
• is restrictive (restricts operation to PATH...)
• is not tracked (the shelf doesn't remember what top-level path(s) you specified, or other associated inputs, it only remembers which individual paths had changes)
• clashes (same path in more than one applied patch) are not managed

Extensions to Consider

These possible extensions are not supported initially.

• allow showing a shelf's content -- like 'svn log [-v] [--diff]', 'svn status', 'svn diff [--summarize]'

• shelve: if no name given, automatically generate a name (so more like 'git stash')

• use shelf's log message when committing

  • see the "Commit Log Message" discussion section above

• consider allowing restricting paths on 'apply' commands

• when checkpointing, warn if PATH... excludes any paths that were in the previous version

• once the base state is recorded: warn or otherwise try to avoid accidentally unshelving to a different branch or unrelated path

Roll Forward, as in Undo/Redo

As a starting point, rollback is destructive, deleting versions newer than the target version. As an enhancement, it could be made to keep the newer versions and allow rolling forward to them. It could operate like the 'undo stack' model commonly found in editing applications, where roll-forward (often named 'redo') is possible up until a different change is saved to the stack, at which time the possibility is lost.

Rebasing Checkpoints

There are cases where it is potentially desirable to be able to rebase a series of checkpoints when updating (pulling in new changes from the repository). One case where this would be useful is if the user updates, goes offline, then wishes to roll back to a checkpoint saved before the update. Another is if the user wishes to make separate commits from several checkpoints in the series, some of which were saved before the last update; but that scenario is not a supported use case in this design.

In trunk-based development usually other people's changes have to be pulled before we can commit. In branch-based work flows that is uncommon.

Whenever we try to restore a checkpoint that was made against an older base, there is a chance that the patch application will run into conflicts. If and when we upgrade from traditional patch application to 3-way merge, conflicts would be reduced, but still possible. Rebasing a series of checkpoints would involve merging once into each checkpoint, with the possibility of conflicts at each step.

As conflicts require tedious manual recovery, these scenarios are best avoided as far as possible.

Rebasing a single checkpoint needs no special support. It is achieved by applying the checkpoint to the WC and then running 'svn update'. The result can then be saved as a new checkpoint or committed.

Therefore this potential extension is low priority.

Integrate Shelving with Changelists

I hate to add complexity without simplifying something at the same time. Considering the relationship between shelves and changelists:

• a changelist is a named set of file-paths in the WC

• a shelf is a named set of changes that is not currently applied to the WC

Integrate them like this:

• shelves and changelists share the same namespace

• shelving means converting a changelist to a shelf of the same name

  • or, if the desired changes are not already managed in a changelist, specify them explicitly

• unshelving means converting a shelf to a changelist of the same name

Benefits

• managing shelves is easier:
  • after unshelve, even if there are other changes elsewhere in the WC, a changelist will remember which changes belonged to that shelf, so you can later shelve or "checkpoint" it without having to specify which files to include

• clear and simple relationship with no overlap

• changelists remain backwards-compatible

Main issues

• a changelist can include unmodified paths:

• should a shelf also include unmodified paths?

• a path can appear in multiple shelves, but only in one changelist:

• what to do when unshelving if a path clashes?

• any extensions should apply uniformly to both shelving and changelists

• if a shelf supports a log msg, a changelist should support a log msg too; this would be a good enhancement for changelists anyway even without shelving

Role models

• changelists in IntelliJ IDEA (are unified across svn, git, hg, etc.)

• changelists in Perforce (are new or pending or shelved or submitted)

Extensions / Not Supported Initially

• When committing changeset X, Subversion could (offer to) delete the shelf X.

• [with roll forward, as in undo/redo] Automatically save a new checkpoint before rollback.

GUI Considerations

GUI priorities tend to be a little different from a CLI.

• APIs that do the job simply and reliably

• progress indication and cancel, for any long-running ops

• hopefully we won't have any long-running ops

• ability to undo

• we don't have to implement undo, just provide enough hooks so caller can

Complete WC State Shelving

A key concern will be to have APIs that do the job simply and reliably. Such as: ability to shelve a WC state whatever state it's in (all the odd states like unresolved conflicts, etc.) and restore to exactly the same state.

We should have two complementary APIs:

• a "shelve" API should visit the WC reporting absolutely everything about the state, in such a way that the caller can capture that state;

• a "diff" API should report the subset of this that is committable;

• an "unshelve" API should be able to recreate any such state, using input entirely provided by the caller

• a "patch" API should accept the subset of this that is committable.

To what extent do such APIs exist?

We should write automatic round-trip testing for diff-&-patch and for status-&-modify.

For states that we can't shelve, a GUI wants to know in advance that this is the case (to grey-out the button), rather than the operation to error out part way through. So we want:

• an API to efficiently tell whether the WC state can be shelved (or why not)

Existing Issues to Overcome

... are tracked in https://issues.apache.org/jira/browse/SVN-3625 "commit shelving".

Glossary of Terms

shelf: a place of storage, within WC metadata, for: a time-ordered series of change-sets; and metadata that applies to the shelf as a whole, modifiable at any time, including a "log message"

save: to copy a change-set from WC working state to a shelf (reverting it from the working state)

shelve: to move a change-set from WC working state to a shelf (reverting it from the working state)

unshelve: to copy a change-set from a shelf into the WC working state; implies choosing the newest version of the change-set on that shelf but can take any version

restore: to copy a change-set from a shelf into the WC working state; implies choosing an older version of the change-set on that shelf but can take any version

version: one of the change-sets in a shelf; versions are assumed to be different (e.g. improving) implementations with the same intended purpose

change-set: a set of changes that Subversion could potentially record in a single revision (changes to files, directories, properties); sometimes refers also to associated revision metadata (log message, other revision properties).

• A change-set may be shelved, unshelved, presented as a diff (patch file), etc.

patch: similar to change-set; implies formatted as a patch file

change: loose term, referring to a change-set or any change that a change-set could contain

changelist: similar to change-set; implies the existing 'svn changelist' feature

• Subversion's existing 'changelist' should grow into the full 'change-set' concept.

checkpoint, savepoint: non-preferred term for (noun) "version", (verb) "save"

• 'savepoint' may be a little better than 'checkpoint' -- better associated with the command verb 'save' and does not share the prefix 'check' of 'checkout' nor the abbreviation 'cp' of 'copy'.

Julian Foad, Assembla, 2017