Incremental Development

Wikipedia link to Iterative and incremental development.

Applying to Derby patches

A Good Thing!

Breaking a large change or implementation of a feature into a sequence of small independent and/or related patches is a good thing because such patches:

  • are easier for the contributor to spot bugs in
  • are easier for the reviewers to understand and spot issues with
  • may lead to reduced amount testing before submission
  • will lead to faster commits
  • easier to identify as a cause of a regression after the fact
  • allows early testing of exposed functionality

Ahead of Time

For a feature it's normal to list a set of steps and possibly sub-steps needed to complete it, each of these steps or sub-steps can correspond to a patch.

As an example, suppose one was planning to implement stored procedures written in SQL for Derby, the steps might be:

  1. Implement simple functions
    1. Add a RETURN statement to the parser (not used)
    2. Add support for RoutineAliasInfo storing additional information about the routine (e.g. language SQL, etc. etc.)
    3. Modify the parser to allow SQL function definition using the RETURN statement, not worrying about parameters
    4. Add code to generate the Java class corresponding to the function
    5. Add code to store the generated class and link it to the routine
    6. Add a mechanism to load the generated class for the function
    7. Add more tests to ensure simple SQL functions work.
  2. Implement functions with parameters
    1. Add named parameter support in the compiler context obtaining the parameter names from the routine defintion
    2. Add code to resolve paramters to the current set in scope (defined by the CREATE FUNCTION statement)
  3. Add support for multi-statement procedures
    1. add parser changes to allow single statement SQL procedures
    2. enhance the code generation to support statements that don't return result sets
    3. add parser changes to allow multiple statements
    4. enhance the code generation to support multiple statements
    5. add savepoints to generated code
  4. Add support for returning result sets in procedures from SQL statements
    1. Add parser changes
    2. etc. etc.
  5. Add control flow to the SQL supported
    1. Add parser changes
    2. etc. etc.

As development proceeds, then patches are submitted corresponding to the steps and sub-steps as needed. After each patch, the code is left is a useful working form but executing the actions may result in a not implemented exception, as an example after sub-step 1c) Derby would accept a valid CREATE FUNCTION statement with LANGUAGE SQL but would throw a not implemented exception at bind time. Most likely each page would contain tests that test out any added functionality, but that may not be a requirement for every patch. This approach leads to a much easier set of reviews than an entire patch that implements all of the above.

Another way to split up a patch to submit tests first in one (or more) patches and then the code. This works very well when one is replacing an existing implementation with a different (better) one. Submitting the tests against the old code helps ensure that your changes do not change the functional behaviour. Such changes could be masked in a single patch.

After the Fact

Sometimes one can be working on a feature and realise too late that the set of changes is large, complicated and will be hard for reviewers to understand. The trick is to spot this as soon as possible, if you are describig a patch and you have to describe more than two or three separate items then consider splitting the patch after the fact.

First one must look at the changes and determine if a set of independent patches or steps exist, e.g. if adding some new SQL feature can the parsing be separate, can the changes to the system table be independent etc? Once you have determined some logical way to break up the large patch then create a separate clean code line in parallel to your modified working code. Create the large patch from your modified code line and apply the portion of it corresponding to the first step to the clean code line. Sometimes this may be just applying a sub-set of the changes, sometimes it may require hand editing. The Eclipse IDE allows fine grained control when applying patches. Build & test the smaller patch in the clean code line and then submit as normal. Repeat until your large patch has been successfully applied through a number of smaller patches.

  • No labels