You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Farming using Plugins

This farming system uses plugins directly and is thus decoupled from the deployment system and allows running the plugin packaging (or "deployment") and farm administration on different servers or machines. The current implementation uses multicast for node discovery. Farming information is stored in a database using jps. The data model allows a single administration server to manage:

  • multiple farms (currently sometimes termed "clusters")
  • multiple lists of plugins per farm (in an mXn relationship)
  • multiple plugins per plugin list

When a node starts up it broadcasts a service advertisement including an optional node name (defaulting to host:jndi-port) and a farmName. When the administration server detects this as new it instructs the new node to install all the plugin lists associated with that farm. If the plugins are already installed, this is a no-op; otherwise the plugins are downloaded and installed from the plugin repository specified in the plugin list.

Administrative operations include:

  • Add a plugin list to a farm
  • Add a plugin to a plugin list
  • Add a plugin to a plugin list and the list to a farm
  • Remove a plugin list from a farm
  • Remove a plugin from a plugin list

I haven't been able to think of a plausible meaning for "remove a plugin from a plugin list from a farm"

These operations are available through the gshell deploy/farm command.

Missing features

  • Admin console support for the administrative operations
  • Better support for setting up and starting multiple servers on one geronimo installation.

Architectural Considerations

Two separate sets of functionality are required for plugin based farms:

  1. A plugin farm controller that tracks the farms node, plugin lists, and plugins and issues instructions to the nodes. The minimum configuration for this is the framework server with the o.a.g.configs/plugin-farm//car plugin installed.
  2. A geronimo plugin repository accessible to all farm nodes. This can be a geronimo server running the geronimo-as-maven servlet, the local file system maven repository, a file based repository served by httpd, or a maven repository manager such as nexus. If a geronimo server is used it can be the same server as the plugin farm controller or a separate server.

Possible plugin-oriented production workflow using maven

This scenario assumes use of maven as a build tool and separate development, qa, and production groups. It needs two maven repositories, for development and production artifacts. These can be simple file/httpd based repositories or something like nexus. Code transfer between groups is in the form of geronimo plugins packaged by the geronimo car-maven-plugin.

  1. development group deploys plugins into the development repository
  2. qa group tests these candidate plugins and upon acceptance deploys them to the production repository
  3. production group distributes plugins to appropriate farms.

Supporting plugins and assemblies

  • o.a.g.configs/plugin-farm-member//car (plugins/clustering/plugin-farm-member) plugin turns a server into a plugin farm node. You can configure the multicast address and port, cluster name, and node name in config-substitutions.properties. It is probably most convenient to not configure node name and rely on the default host:jndi-port.
  • o.a.g.assemblies/geronimo-plugin-farm-node (plugins/clustering/geronimo-plugin-farm-node) extends the framework assembly with the plugin-farm-member plugin. This is all you need to start up a farm node.
  • o.a.g.configs/plugin-farm//car (plugins/clustering/plugin-farm) is the administration plugin for plugin-based farms. Currently it's only accessible through the gshell deploy/farm command.
  • o.a.g.configs/plugin-farm-datasource//car (plugins/clustering/plugin-farm-datasource) is a derby datasource used by the plugin farm. It is intended to be replaced in production with a remote datasource. As always, the replacement plugin should use an <artifact-alias> element to redirect dependencies on plugin-farm-datasource to itself.

In my sandbox until we find a better location or solution, as examples

  • o.a.g.configs/controller-as-repo//car (sandbox/djencks/assemblies/controller-as-repo) is a non-classloader plugin that sets the default plugin repository to be the admin server. This currently only works if both the admin server and all the farm nodes are running on the same machine – obviously only a toy example scenario.
  • o.a.g.assemblies/geronimo-tomcat-farm-controller//tar.gz|zip (sandbox/assemblies/geronimo-tomcat-farm-controller) is a server assembly that extends the tomcat javaee5 server with the plugin farm controller

GShell deploy/farm command syntax

The GShell deploy/farm command takes one argument, the action, which must be "add" or "remove", and options for the farm, plugin list, and plugin.

-f (--farm) farm name
-l (--pluginlist) plugin list name
-a (--pluginartifact) plugin artifact name

In addition there are some inherited options for connecting to the server:

-s (--hostname, --server) server name or IP address
-p (--port) server port
-u (--user) user name
-w (--password) password
--secure (use secure communication to server)

Example walkthrough

This example demonstrates the plugin farm on a single server, using nodes that all share the same geronimo installation, and using the admin server as farm controller, application deployment server, and plugin repository. We'll install one of the sample applications on the admin server and deploy it to the farm. I've only tested this with all components built locally on one machine, but some such as the geronimo-plugin-farm-node should be available from the apache snapshot repository.

Farm node setup.

  • obtain the geronimo-plugin-farm-node assembly, such as by building plugins/clustering.
  • unpack it and copy the var directory to multiple servers, say server1 through server5. On a unix like system this can be done with a script like 
    #!/bin/bash

tar xzf geronimo-plugin-farm-node-2.2-SNAPSHOT-bin.tar.gz
cd geronimo-plugin-farm-node-2.2-SNAPSHOT

for s in {1..5}
do 
  sn="server${s}"
  mkdir ${sn}
  cp -r var ${sn}/
#uncomment to actually start the servers in the background
# ./bin/gsh geronimo/start-server -G server.name=${sn} -G config.substitution.PortOffset=${s}0 >${sn}.log &
done

cd ..

Here's a fancier script that can both start and stop servers but still illustrates my bash illiteracy:

#!/bin/bash

BASE=geronimo-plugin-farm-node-2.2-SNAPSHOT
if  [ "$1" = "stop" ] ; then
(
  cd $BASE
  for server in server*; do
    pid=$(ps ax | grep "gshell-bootstrap.jar geronimo/start-server -s $server" | grep -v 'grep' | cut -c 1-6)

    if [ $pid ] ; then   
      echo "Killing gshell for $server at pid $pid"
      kill -9 $pid
    fi
    pid=$(ps ax |  grep "javaagent.*$server" | grep -v 'grep' | cut -c 1-6)

    if [ $pid ] ; then   
      echo "Killing $server at pid $pid"
      kill -9 $pid
    fi
  done
)
fi

rm -rf $BASE
tar xzf $BASE-bin.tar.gz
(
cd $BASE

for s in {1..5}
do 
  sn="server${s}"
  mkdir ${sn}
  cp -r var ${sn}/
  if  [ "$1" = "start" ] ; then
    ./bin/gsh geronimo/start-server -s ${sn} -G config.substitution.PortOffset=${s}0 >${sn}.log &
  fi
done
)
  • Start the servers with command lines like this: 
    ./bin/gsh geronimo/start-server -G server.name=server1 -G config.substitution.PortOffset=10
    substituting appropriately for "server1" and "10"

Farm controller setup

  • check out bits of my sandbox 
    svn co https://svn.apache.org/repos/asf/geronimo/sandbox/djencks/assemblies
  • build the plugin and assembly 
    cd assemblies/controller-as-repo
mvn clean install
cd ../geronimo-tomcat-farm-controller
mvn clean install
cd target
tar xzf geronimo-tomcat-farm-controller-2.2-SNAPSHOT-bin.tar.gz 
./geronimo-tomcat-farm-controller-2.2-SNAPSHOT/bin/gsh geronimo/start-server

Deploying a sample to the farm

  • check out and build samples (to get the plan for deployment) 
    svn co https://svn.apache.org/repos/asf/geronimo/samples/trunk
cd trunk
mvn clean install
  • In a separate terminal in geronimo-tomcat-farm-controller-2.2-SNAPSHOT start gshell (bin/gsh) and run these commands to deploy a dependency (sample-datasource), the sample application, and then distribute it to the farm nodes: 
    deploy/list-plugins --username system --password manager org.apache.geronimo.samples/sample-datasource/2.2-SNAPSHOT/car
# choose the file repo based on your local maven repository
deploy/deploy /Users/david/geronimo/svn/geronimo/samples/trunk/samples/bank/bank-ear/target/bank-ear-2.2-SNAPSHOT.ear  /Users/david/geronimo/svn/geronimo/samples/trunk/samples/bank/bank-tomcat/target/resources/META-INF/plan.xml 
# substitute actual paths to the ear and plan from your samples checkout
deploy/farm add -f cluster1 -l pluginList1 -a org.apache.geronimo.samples/bank-tomcat/2.2-SNAPSHOT/car

In the server logs for the admin server and the farm nodes you should see that the application is installed along with all dependencies. You should be able to see the bank sample on the nodes at e.g. http://localhost:8120/bank/

Removing plugins and plugin lists

In the gsh console above try

deploy/farm remove -l pluginList1 -a org.apache.geronimo.samples/bank-tomcat/2.2-SNAPSHOT/car  
# add back the plugin...
deploy/farm add -f cluster1 -l pluginList1 -a org.apache.geronimo.samples/bank-tomcat/2.2-SNAPSHOT/car 
# remove it again
deploy/farm remove -f cluster1 -l pluginList1

Configuration of plugin repositories.

The example above is set up, for convenience and portability, to use the farm controller server as the plugin repository, and to use "localhost" as the server name. This means no custom configuration is needed to run everything on one machine but is obviously incomplete for any serious use.

The accessible plugin repositories are configured in each node in the var/config/plugin-repositories.properties file. This needs to include the appropriate plugin repository with any needed credentials. One way to do this is to write a non-classloader plugin to install a new var/config/plugin-repositories.properties to replace the default. You'd assemble a plugin node server including this additional plugin and distribute that to the nodes.

In addition, the controller also needs to have this plugin repository configured as the default in var/config/config-substitutions.properties in this entry:

DefaultPluginRepository=http\://localhost\:8080/plugin/maven-repo

Future work

  • enhancing the monitoring console to find nodes using multicast discovery
  • enhancing the car-maven-plugin to assemble servers with multiple instance support in the assembly.
  • No labels