Farming using Deployment

Clustering and farming

Tomcat Native Clustering

This farming system uses plugins directly and is thus decoupled from the deployment system and allows running the plugin packaging (or "deployment") and farm administration on different servers or machines. The current implementation uses multicast for node discovery. Farming information is stored in a database using jps. The data model allows a single administration server to manage:

• multiple farms (currently sometimes termed clusters)
• multiple lists of plugins per farm (in an mXn relationship)
• multiple plugins per plugin list

When a node starts up it broadcasts a service advertisement including an optional node name (defaulting to host:jndi-port) and a farm name. When the administration server detects this as new it instructs the new node to install all the plugin lists associated with that farm. If the plugins are already installed, this is a no-op; otherwise the plugins are downloaded and installed from the plugin repository specified in the plugin list.

Administrative operations include:

• Add a plugin list to a farm
• Add a plugin to a plugin list
• Add a plugin to a plugin list and the list to a farm
• Remove a plugin list from a farm
• Remove a plugin from a plugin list

(I haven't been able to think of a plausible meaning for "remove a plugin from a plugin list from a farm")

These operations are available through the GShell deploy/farm command.

Missing features

• Admin console support for the administrative operations
• Better support for setting up and starting multiple servers on one geronimo installation.

Architectural Considerations

Two separate sets of functionality are required for plugin based farms:

1. A plugin farm controller that tracks the farms node, plugin lists, and plugins and issues instructions to the nodes. The minimum configuration for this is the framework server with the o.a.g.configs/plugin-farm//car plugin installed.
2. A geronimo plugin repository accessible to all farm nodes. This can be a geronimo server running the geronimo-as-maven servlet, the local file system maven repository, a file based repository served by httpd, or a maven repository manager such as nexus. If a geronimo server is used it can be the same server as the plugin farm controller or a separate server.

Possible plugin-oriented production workflow using maven

This scenario assumes use of maven as a build tool and separate development, quality assurance (QA), and production groups. It needs two maven repositories, for development and production artifacts. These can be simple file/httpd based repositories or something like nexus. Code transfer between groups is in the form of geronimo plugins packaged by the geronimo car-maven-plugin.

1. Development group deploys plugins into the development repository.
2. QA group tests these candidate plugins and upon acceptance deploys them to the production repository.
3. Production group distributes plugins to appropriate farms.

Supporting plugins and assemblies

• o.a.g.configs/plugin-farm-member//car (plugins/clustering/plugin-farm-member) plugin turns a server into a plugin farm node. You can configure the multicast address and port, cluster name, and node name in config-substitutions.properties. It is probably most convenient to not configure node name and rely on the default host:jndi-port.
• o.a.g.assemblies/geronimo-plugin-farm-node (plugins/clustering/geronimo-plugin-farm-node) extends the framework assembly with the plugin-farm-member plugin. This is all you need to start up a farm node.
• o.a.g.configs/plugin-farm//car (plugins/clustering/plugin-farm) is the administration plugin for plugin-based farms. Currently it's only accessible through the gshell deploy/farm command.
• o.a.g.configs/plugin-farm-datasource//car (plugins/clustering/plugin-farm-datasource) is a derby datasource used by the plugin farm. It is intended to be replaced in production with a remote datasource. As always, the replacement plugin should use an <artifact-alias> element to redirect dependencies on plugin-farm-datasource to itself.

In my sandbox until we find a better location or solution, as examples

• o.a.g.configs/controller-as-repo//car (sandbox/djencks/assemblies/controller-as-repo) is a non-classloader plugin that sets the default plugin repository to be the admin server. This currently only works if both the admin server and all the farm nodes are running on the same machine – obviously only a toy example scenario.
• o.a.g.assemblies/geronimo-tomcat-farm-controller//tar.gz|zip (sandbox/assemblies/geronimo-tomcat-farm-controller) is a server assembly that extends the tomcat javaee5 server with the plugin farm controller

GShell deploy/farm command syntax

The GShell deploy/farm command takes one argument, the action, which must be add or remove, and options for the farm, plugin list, and plugin.

-f (--farm) farm name
-l (--pluginlist) plugin list name
-a (--pluginartifact) plugin artifact name

In addition there are some inherited options for connecting to the server:

-s (--hostname, --server) server name or IP address
-p (--port) server port
-u (--user) user name
-w (--password) password
--secure (use secure communication to server)

Example walkthrough

This example demonstrates the plugin farm on a single server, using nodes that all share the same geronimo installation, and using the admin server as farm controller, application deployment server, and plugin repository. We'll install one of the sample applications on the admin server and deploy it to the farm. I've only tested this with all components built locally on one machine, but some such as the geronimo-plugin-farm-node should be available from the apache snapshot repository.

Farm node setup

1. Obtain the geronimo-plugin-farm-node assembly, such as by building plugins/clustering.
2. Unpack it and copy the var directory to multiple servers, say server1 through server5. On a unix like system this can be done with a script like

   #!/bin/bash
   
   tar xzf geronimo-plugin-farm-node-2.2-SNAPSHOT-bin.tar.gz
   cd geronimo-plugin-farm-node-2.2-SNAPSHOT
   
   for s in {1..5}
   do 
     sn="server${s}"
     mkdir ${sn}
     cp -r var ${sn}/
   #uncomment to actually start the servers in the background
   # ./bin/gsh geronimo/start-server -G server.name=${sn} -G config.substitution.PortOffset=${s}0 >${sn}.log &
   done
   
   cd ..
   

   Here's a fancier script that can both start and stop servers but still illustrates my bash illiteracy:

   #!/bin/bash
   
   BASE=geronimo-plugin-farm-node-2.2-SNAPSHOT
   if  [ "$1" = "stop" ] ; then
   (
     cd $BASE
     for server in server*; do
       pid=$(ps ax | grep "gshell-bootstrap.jar geronimo/start-server -s $server" | grep -v 'grep' | cut -c 1-6)
   
       if [ $pid ] ; then   
         echo "Killing gshell for $server at pid $pid"
         kill -9 $pid
       fi
       pid=$(ps ax |  grep "javaagent.*$server" | grep -v 'grep' | cut -c 1-6)
   
       if [ $pid ] ; then   
         echo "Killing $server at pid $pid"
         kill -9 $pid
       fi
     done
   )
   fi
   
   rm -rf $BASE
   tar xzf $BASE-bin.tar.gz
   (
   cd $BASE
   
   for s in {1..5}
   do 
     sn="server${s}"
     mkdir ${sn}
     cp -r var ${sn}/
     if  [ "$1" = "start" ] ; then
       ./bin/gsh geronimo/start-server -s ${sn} -G config.substitution.PortOffset=${s}0 >${sn}.log &
     fi
   done
   )
   
   

3. Start the servers with command lines like this:

   ./bin/gsh geronimo/start-server -G server.name=server1 -G config.substitution.PortOffset=10 
   

   substituting appropriately for server1 and 10

Farm controller setup

1. Check out bits of my sandbox

   svn co https://svn.apache.org/repos/asf/geronimo/sandbox/djencks/assemblies
   

2. Build the plugin and assembly

   cd assemblies/controller-as-repo
   mvn clean install
   cd ../geronimo-tomcat-farm-controller
   mvn clean install
   cd target
   tar xzf geronimo-tomcat-farm-controller-2.2-SNAPSHOT-bin.tar.gz 
   ./geronimo-tomcat-farm-controller-2.2-SNAPSHOT/bin/gsh geronimo/start-server
   

Deploying a sample to the farm

1. Check out and build samples (to get the plan for deployment)

   svn co https://svn.apache.org/repos/asf/geronimo/samples/trunk
   cd trunk
   mvn clean install
   

2. In a separate terminal in geronimo-tomcat-farm-controller-2.2-SNAPSHOT start GShell(bin/gsh) and run these commands to deploy a dependency (sample-datasource), the sample application, and then distribute it to the farm nodes:

   deploy/list-plugins --username system --password manager org.apache.geronimo.samples/sample-datasource/2.2-SNAPSHOT/car
   # Choose the file repo based on your local maven repository
   deploy/deploy /Users/david/geronimo/svn/geronimo/samples/trunk/samples/bank/bank-ear/target/bank-ear-2.2-SNAPSHOT.ear 
    /Users/david/geronimo/svn/geronimo/samples/trunk/samples/bank/bank-tomcat/target/resources/META-INF/plan.xml 
   # Substitute actual paths to the ear and plan from your samples checkout
   deploy/farm add -f cluster1 -l pluginList1 -a org.apache.geronimo.samples/bank-tomcat/2.2-SNAPSHOT/car  
   

In the server logs for the admin server and the farm nodes you should see that the application is installed along with all dependencies. You should be able to see the bank sample on the nodes at for example, http://localhost:8120/bank/

Removing plugins and plugin lists

In the gsh console above try

deploy/farm remove -l pluginList1 -a org.apache.geronimo.samples/bank-tomcat/2.2-SNAPSHOT/car  
# add back the plugin...
deploy/farm add -f cluster1 -l pluginList1 -a org.apache.geronimo.samples/bank-tomcat/2.2-SNAPSHOT/car 
# remove it again
deploy/farm remove -f cluster1 -l pluginList1   

Configuration of plugin repositories.

The example above is set up, for convenience and portability, to use the farm controller server as the plugin repository, and to use localhost as the server name. This means no custom configuration is needed to run everything on one machine but is obviously incomplete for any serious use.

The accessible plugin repositories are configured in each node in the var/config/plugin-repositories.properties file. This needs to include the appropriate plugin repository with any needed credentials. One way to do this is to write a non-classloader plugin to install a new var/config/plugin-repositories.properties to replace the default. You'd assemble a plugin node server including this additional plugin and distribute that to the nodes.

In addition, the controller also needs to have this plugin repository configured as the default in var/config/config-substitutions.properties in this entry:

DefaultPluginRepository=http\://localhost\:8080/plugin/maven-repo

Future work

• enhancing the monitoring console to find nodes using multicast discovery
• enhancing the car-maven-plugin to assemble servers with multiple instance support in the assembly.