This farming system uses plugins directly. Therefore, it is decoupled from the deployment system and allows running the plugin packaging (or "deployment") and farm administration on different servers or machines. The current implementation uses multicast for node discovery. Farming information is stored in a database using jps. The data model allows a single administration server to manage:
- multiple farms (currently sometimes termed clusters)
- multiple lists of plugins per farm (in an mXn relationship)
- multiple plugins per plugin list
When a node starts up, it broadcasts a service advertisement including an optional node name (defaulting to host:jndi-port) and a farm name. When the administration server detects this as new, it instructs the new node to install all the plugin lists associated with that farm. If the plugins are already installed, this is a no-op; otherwise the plugins are downloaded and installed from the plugin repository specified in the plugin list.
Administrative operations include:
- Add a plugin list to a farm
- Add a plugin to a plugin list
- Add a plugin to a plugin list and the list to a farm
- Remove a plugin list from a farm
- Remove a plugin from a plugin list
(I haven't been able to think of a plausible meaning for "remove a plugin from a plugin list from a farm")
These operations are available through the GShell cluster/deploy command.
Missing features
- Admin console support for the administrative operations
- Better support for setting up and starting multiple servers on one geronimo installation.
Architectural Considerations
Two separate sets of functionality are required for plugin based farms:
- A plugin farm controller that tracks the farms node, plugin lists, and plugins and issues instructions to the nodes. The minimum configuration for this is the framework server with the o.a.g.configs/plugin-farm//car plugin installed.
- A geronimo plugin repository accessible to all farm nodes. This can be a geronimo server running the geronimo-as-maven servlet, the local file system maven repository, a file based repository served by httpd, or a maven repository manager such as nexus. If a geronimo server is used it can be the same server as the plugin farm controller or a separate server.
Possible plugin-oriented production workflow using maven
This scenario assumes use of maven as a build tool and separate development, quality assurance (QA), and production groups. It needs two maven repositories, for development and production artifacts. These can be simple file/httpd based repositories or something like nexus. Code transfer between groups is in the form of geronimo plugins packaged by the geronimo car-maven-plugin.
- Development group deploys plugins into the development repository.
- QA group tests these candidate plugins and upon acceptance deploys them to the production repository.
- Production group distributes plugins to appropriate farms.
Supporting plugins and assemblies
- o.a.g.configs/plugin-farm-member//car (plugins/clustering/plugin-farm-member) plugin turns a server into a plugin farm node. You can configure the multicast address and port, cluster name, and node name in config-substitutions.properties. It is probably most convenient to not configure node name and rely on the default host:jndi-port.
- o.a.g.assemblies/geronimo-plugin-farm-node (plugins/clustering/geronimo-plugin-farm-node) extends the framework assembly with the plugin-farm-member plugin. This is all you need to start up a farm node.
- o.a.g.configs/plugin-farm//car (plugins/clustering/plugin-farm) is the administration plugin for plugin-based farms. Currently it's only accessible through the gshell cluster/deploy command.
- o.a.g.configs/plugin-farm-datasource//car (plugins/clustering/plugin-farm-datasource) is a derby datasource used by the plugin farm. It is intended to be replaced in production with a remote datasource. As always, the replacement plugin should use an <artifact-alias> element to redirect dependencies on plugin-farm-datasource to itself.
In my sandbox until we find a better location or solution, as examples
- o.a.g.configs/controller-as-repo//car (sandbox/djencks/assemblies/controller-as-repo) is a non-classloader plugin that sets the default plugin repository to be the admin server. This currently only works if both the admin server and all the farm nodes are running on the same machine – obviously only a toy example scenario.
- o.a.g.assemblies/geronimo-tomcat-farm-controller//tar.gz|zip (sandbox/assemblies/geronimo-tomcat-farm-controller) is a server assembly that extends the tomcat javaee5 server with the plugin farm controller
GShell cluster/deploy command syntax
The GShell cluster/deploy command takes one argument, the action, which must be add or remove, and options for the farm, plugin list, and plugin.
-f (--farm) farm name -l (--pluginlist) plugin list name -a (--pluginartifact) plugin artifact name
In addition there are some inherited options for connecting to the server:
-s (--hostname, --server) server name or IP address -p (--port) server port -u (--user) user name -w (--password) password --secure (use secure communication to server)
Example walkthrough
This example demonstrates the plugin farm on a single server, using nodes that all share the same geronimo installation, and using the admin server as farm controller, application deployment server, and plugin repository. We'll install one of the sample applications on the admin server and deploy it to the farm. I've only tested this with all components built locally on one machine, but some such as the geronimo-plugin-farm-node should be available from the apache snapshot repository.
Farm node setup
- Obtain the geronimo-plugin-farm-node assembly, such as by building plugins/clustering.
- Unpack it and copy the var directory to multiple servers, say server1 through server5. On a unix like system this can be done with a script like
Here's a fancier script that can both start and stop servers but still illustrates my bash illiteracy:
#!/bin/bash tar xzf geronimo-plugin-farm-node-2.2-SNAPSHOT-bin.tar.gz cd geronimo-plugin-farm-node-2.2-SNAPSHOT for s in {1..5} do sn="server${s}" mkdir ${sn} cp -r var ${sn}/ #uncomment to actually start the servers in the background # ./bin/gsh geronimo/start-server -G server.name=${sn} -G config.substitution.PortOffset=${s}0 >${sn}.log & done cd ..
#!/bin/bash BASE=geronimo-plugin-farm-node-2.2-SNAPSHOT if [ "$1" = "stop" ] ; then ( cd $BASE for server in server*; do pid=$(ps ax | grep "gshell-bootstrap.jar geronimo/start-server -s $server" | grep -v 'grep' | cut -c 1-6) if [ $pid ] ; then echo "Killing gshell for $server at pid $pid" kill -9 $pid fi pid=$(ps ax | grep "javaagent.*$server" | grep -v 'grep' | cut -c 1-6) if [ $pid ] ; then echo "Killing $server at pid $pid" kill -9 $pid fi done ) fi rm -rf $BASE tar xzf $BASE-bin.tar.gz ( cd $BASE for s in {1..5} do sn="server${s}" mkdir ${sn} cp -r var ${sn}/ if [ "$1" = "start" ] ; then ./bin/gsh geronimo/start-server -s ${sn} -G config.substitution.PortOffset=${s}0 >${sn}.log & fi done )
- Start the servers with command lines like this:
substituting appropriately for server1 and 10
./bin/gsh geronimo/start-server -G server.name=server1 -G config.substitution.PortOffset=10
Farm controller setup
- Check out bits of my sandbox
svn co https://svn.apache.org/repos/asf/geronimo/sandbox/djencks/assemblies
- Build the plugin and assembly
cd assemblies/controller-as-repo mvn clean install cd ../geronimo-tomcat-farm-controller mvn clean install cd target tar xzf geronimo-tomcat-farm-controller-2.2-SNAPSHOT-bin.tar.gz ./geronimo-tomcat-farm-controller-2.2-SNAPSHOT/bin/gsh geronimo/start-server
Deploying a sample to the farm
- Check out and build samples (to get the plan for deployment)
svn co https://svn.apache.org/repos/asf/geronimo/samples/trunk cd trunk mvn clean install
- In a separate terminal in geronimo-tomcat-farm-controller-2.2-SNAPSHOT start GShell(bin/gsh) and run these commands to deploy a dependency (sample-datasource), the sample application, and then distribute it to the farm nodes:
deploy/list-plugins --username system --password manager org.apache.geronimo.samples/sample-datasource/2.2-SNAPSHOT/car # Choose the file repo based on your local maven repository deploy/deploy /Users/david/geronimo/svn/geronimo/samples/trunk/samples/bank/bank-ear/target/bank-ear-2.2-SNAPSHOT.ear /Users/david/geronimo/svn/geronimo/samples/trunk/samples/bank/bank-tomcat/target/resources/META-INF/plan.xml # Substitute actual paths to the ear and plan from your samples checkout cluster/deploy add -f cluster1 -l pluginList1 -a org.apache.geronimo.samples/bank-tomcat/2.2-SNAPSHOT/car
In the server logs for the admin server and the farm nodes you should see that the application is installed along with all dependencies. You should be able to see the bank sample on the nodes at for example,
http://localhost:8120/bank/
Removing plugins and plugin lists
In the gsh console above try
cluster/deploy remove -l pluginList1 -a org.apache.geronimo.samples/bank-tomcat/2.2-SNAPSHOT/car # add back the plugin... cluster/deploy add -f cluster1 -l pluginList1 -a org.apache.geronimo.samples/bank-tomcat/2.2-SNAPSHOT/car # remove it again cluster/deploy remove -f cluster1 -l pluginList1
Configuration of plugin repositories.
The example above is set up, for convenience and portability, to use the farm controller server as the plugin repository, and to use localhost as the server name. This means no custom configuration is needed to run everything on one machine but is obviously incomplete for any serious use.
The accessible plugin repositories are configured in each node in the var/config/plugin-repositories.properties
file. This needs to include the appropriate plugin repository with any needed credentials. One way to do this is to write a non-classloader plugin to install a new var/config/plugin-repositories.properties
to replace the default. You'd assemble a plugin node server including this additional plugin and distribute that to the nodes.
In addition, the controller also needs to have this plugin repository configured as the default in var/config/config-substitutions.properties
in this entry:
DefaultPluginRepository=http\://localhost\:8080/plugin/maven-repo
Future work
- enhancing the monitoring console to find nodes using multicast discovery
- enhancing the car-maven-plugin to assemble servers with multiple instance support in the assembly.