Overview
Build Creator
The BuildCreator tool was devised to enable the easy generation of binary packages that include more than one source. The purpose of the build creator is to simplify the building of projects that require the combining of multiple sources. To aid discussion of the tool we shall look at the configuration file combining the Apache Qpid release with that of the BerkeleyDB Store plugin form JBoss. However, the tool is a general binary build tool written in Python.
Command Line arguments
There are various command line arguments, mainly to change the amount of logging the tool generates.
subprocess is required for this tool and is not present in versions prior to 2.4.0
usage: buildCreator.py [options]
options:
-h, --help show this help message and exit
-c, --config=CONFIG
set configuration file : default = build.config
-v, --verbose enable verbose output
-d, --debug enable debug output
-q, --quiet Enable quiet ouptut
-i, --ignore-errors Ignore errors
There are a number of available targets:
Available Targets:
distclean [source] - Remove all or specified retrieved source
clean [source] - Remove all or specified source build directory
retrieve [source] - Retrieve all or specified source
prepare [source] - Prepare all or specified source : i.e. extract archives
patch [source] - Patch all or specified source
showbuilds - List all builds
build [build] - Perform the build scripts for all or specified build
release [build] - Perform the release scripts for all or specified source
full - Perform clean, retrieve, prepare, patch, build, release for all builds (DEFAULT)
Targets
The default target is full which as listed above will perform all the required steps to create all the releases for the builds listed in the configuration. Most of the targets do exactly what they say above however a few have additional points of interest:
retrieve
The retrieve target pulls together all the source and patches and places this pristine copy in the builder/src directory. This is done so that the build can be repeatable performed with the untouched source.
prepare
The prepare target creates the builder/build tree and copies the source and patches for the later stages. If the source was not retrieved svn then the files are inspected and if an archive format is found this is expanded into the build directory.
release
The release scripts that are specified in the build files can contain a version keyword substitution $writeVersions(file). This will append details about the sources and patches used to generate this release artefact. If svn targets are used then the revision information will be added to the file.
Configuration Details
Two files are currently required by the tool the main configuration file and the build file.
Config file
The main configuration file (build.config by default) specifies the various sources, patches and builds that should be utilised.
The configuration file is used to define any common environment variables that may be used for substitution in the build scripts. The <sources> section contains a list of all source locations that the build will utilise. The <patch> section allows a number of patches to be specified to apply to the pristinely downloaded source to be modified.
NOTE: Limitiations
The build section can only contain <include> values not <build> elements directly
<builder>
<environment>
<[variable]>[value]</[variable]>
</environment>
<sources>
<source>
<name>[source-name]</name>
<type>[source-type:svn|file|http|ftp]</type>
<url>[value]</url>
<path>[root offset, useful if to point the root of the source in to an archive output]</path>
</source>
</sources>
<patches>
<patch>
<name>[patch-name]</name>
<type>[source-type:svn|file|http|ftp]</type>
<url>[value]</url>
<source>[source name this patches]</source>
<prefix>[patch prefix -p value]</prefix>
<path>[root offset, useful if the base of the patch is not the root of the source]</path>
</patch>
</patches>
<builds>
<include>[string which is sent to ls to retrieve build include file so 'builds/*.build' works]</include>
</builds>
</builder>
Source Section
The <source> section contains a number of required values.
Entry |
Description |
<name> |
The name you wish to give to this build. This value should not contain spaces as it can be used in the build and release scripts as a variable to refer to the build location. |
<type> |
There are four types of source svn, file, http, ftp. These are explained further below |
<url> |
This is the URL or file path to this source |
<path> |
Optional path if you wish to move the root of the source. i.e. in to an archive file |
Source Types
The <type> of the determines how the creator will post process the file.
Type |
Description |
svn |
Uses svn to retrieve the given <url>, An optional <revision> element can be added to source to be used by the svn checkout |
file |
Uses cp to take a copy of the specified file. |
http |
Uses wget to retrieve the specified file or directory, it currently does not recurse. |
ftp |
Uses wget to retrieve the specified file or directory, it currently does not recurse. |
Patch Section
The patch section contains all of the same entries as the Source Section. It also has two additional entries that are used in applying the patches.
Entry |
Description |
<source> |
Source name that this patch entry should be applied to. |
<prefix> |
This is the patch prefix (-p) value used on the command line. |
The <url> value here can either be a file or a directory. In the case that it is a directory all files contained in that directory will be treated as patch files and applied to the specified <source> with the given options.
Builds section
The final section in the config file is the <builds> section. Currently this can only contain a single <include> value. This specifies the build scripts definitions that should be included in this configuration. Any value that is a valid argument to ls can be used here. This allows multiple files to be included. Future work includes the direct embedding of the build configuration.
Build file
The build file contains scripts to perform the build and release of a desired release. There is no restriction on what can be performed during either section however only one <build> may be contained in each file.
<builds>
<build>
<name>[build-name]</name>
<dependency>
<source>[source-name]</source>
</dependency>
<targets>
<build>
<script><![CDATA[
<shell script>
]]>
</script>
</build>
<release>
<script><![CDATA[
<shell script>
]]>
</script>
</release>
</targets>
</build>
</builds>
Build Section
The build sction contains three elements all must be present to be valid.
Element |
Desctiption |
<name> |
The name of this build. This is used to name directories on disk and to refer to this build directly on the command line. |
<dependency> |
This is a list of <source> entries that this build requires. |
<targets> |
This contains the two scripts one for <build> and one for <release>. The text in these scripts are executed in a shell rooted in the current directory. |
Writing release scripts
To easy the writing of release scripts there are a number of variables that have been predefined for your use. The source <name> values can be used as variables to refer to the root of that source's build location, in the examples below that would mean that $qpid would be converted in to builder/build/qpid. As mentioned above you can also define variables in the <environment> section of the configuration. Release scripts can contain a version keyword substitution $writeVersions(file). This will append details about the sources and patches used to generate this release artefact. If svn targets are used then the revision information will be added to the file.
Example Files
<builder>
<environment>
<version>M3.0-beta</version>
</environment>
<sources>
<source>
<name>qpid</name>
<type>file</type>
<url>http://people.apache.org/~aidan/qpid/M3-beta/qpid-incubating-M3-beta.tar.gz</url>
<path>qpid-incubating-M3</path>
</source>
<source>
<name>bdb</name>
<type>svn</type>
<url>https://svn.jboss.org/repos/rhmessaging/store/branches/java/broker-queue-refactor/java/bdbstore</url>
</source>
</sources>
<patches>
<patch>
<name>BDB-Classpath</name>
<type>file</type>
<url>/local/patches/bdb-qpid-run-classpath.diff</url>
<source>qpid</source>
<prefix>2</prefix>
<path>qpid-incubating-M3/qpid/java/<path>
</patch>
</patches>
<builds>
<include>builds/*.config</include>
</builds>
</builder>
<builds>
<build>
<name>qpid-broker</name>
<dependency>
<source>[source-name]</source>
<source>bdb</source>
</dependency>
<targets>
<build>
<script><![CDATA[
pushd $qpid/java
ant -Dproject.version=$version build
popd
cp $qpid/java/build/lib/qpid-broker-$version.jar $bdb/lib
cp $qpid/java/build/lib/qpid-broker-test-$version.jar $bdb/lib
cp $qpid/java/build/lib/qpid-common-$version.jar $bdb/lib
cp $qpid/java/build/lib/qpid-systests-$version.jar $bdb/lib
cp $qpid/java/build/lib/qpid-perftests-$version.jar $bdb/lib
cp $qpid/java/build/lib/qpid-junit-toolkit-$version.jar $bdb/lib
cd $bdb
ant build
]]>
</script>
</build>
<release>
<script><![CDATA[
# Create build package
mkdir -p $release/$build-$version
cp -r $qpid/java/build/* $release/$build-$version
cp $bdb/build/qpid-bdbstore.jar $bdb/lib/je-3.3.62.jar $release/$build-$version/lib
# Build release artifact
cd $release/$build-version
# Create release revisions
echo "Qpid Broker Release : $version" > REVISIONS.txt
echo -n "Built:" >> REVISIONS.txt
date +%Y-%m-%d-%H%M >> REVISIONS.txt
$writeVersions(REVISIONS.txt)
cd ..
tar cvzf $build-$version.tgz $build-$version
]]>
</script>
</release>
</targets>
</build>
</builds>