...

The basic design assumption is that all servers in the same group have the same applications deployed and are capable of doing the same job. Servers can be brought online and offline while clients are running. As members join/leave this information is sent to the client as part of normal EJB request/response communication so active clients always have the most current information on servers that can process their request should communication with a particular server fail.

Failover

On each request to the server, the client will send the version number associated with the list of servers in the cluster it is aware of. Initially this version will be zero and the list will be empty. Only when the server sees the client has an old list will the server send the updated list. This is an important distinction as the list is not transmitted back and forth on every request, only on change. If the membership of the cluster is stable there is essentially no clustering overhead to the protocol – 8 byte overhead to each request and 1 byte on each response – so you will not see an exponential slowdown in response times the more members are added to the cluster. This new list takes affect for all proxies that share the same connection.

...

By default, the failover is ordered but random selection is supported. The multicast discovery aspect of the client adds a nice randomness to the selection of the first server.

Discovery

Each discoverable service has a URI which is broadcast as a heartbeat to other servers in the cluster. This URI advertises the service's type, its cluster group, and its location in the format of 'group:type:location'. Say for example "cluster1:ejb:ejbd://thehost:4201". The URI is sent out repeatedly in a pulse and its presence on the network indicates its availability and its absence indicates the service is no longer available.

...

The multicast functionality is not just for servers to find each other in a cluster, it can also be used for EJB clients to discover a server. A special "multicast://" URL can be used in the InitialContext properties to signify that multicast should be used to seed the connection process. Such as:

   Properties propertiesp = new Properties();
   properties.setPropertyp.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.RemoteInitialContextFactory");
   properties.setPropertyp.put(Context.PROVIDER_URL, "multicast://239.255.2.3:6142?group=default");
   InitialContext remoteContext = new InitialContext(propertiesp);

The URL has optional query parameters such as "schemes" and "group" and "timeout" which allow you to zero in on a particular type of service of a particular cluster group as well as set how long you are willing to wait in the discovery process till finally giving up. The first matching service that it sees "flowing" around on the UDP stream is the one it picks and sticks to for that and subsequent requests, ensuring UDP is only used when there are no other servers to talk to.

...

Note that clients do not need to use multicast to communicate with servers. Servers can use multicast to discover each other, but clients are still free to connect to servers in the network using the server's TCP address.

   Properties propertiesp = new Properties();
   properties.setPropertyp.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.RemoteInitialContextFactory");
   properties.setPropertyp.put(Context.PROVIDER_URL, "ejbd://192.168.1.30:4201");
   InitialContext remoteContext = new InitialContext(propertiesp);

When the client connects, the server will send the URLs of all the servers in the group and failover will take place normally.

...

As TCP has no real As TCP has no real broadcast functionality to speak of, communication of who is in the network is achieved by each server having a physical connection to each other server in the network.

...

To join the network, the server must be configured to know the address of at least one server in the network and connect to it. When it does both servers will exchange the full list of all the other servers each knows about. Each server will then connect to any new servers they've just learned about and repeat the processes with those new servers. The end result is that everyone has a direct connection to everyone 100% of the time, hence the made-up term "multipoint" to describe this situation of each server having multiple point-to-point connections which create a fully connected graph.

...

The entire process is essentially the art of using a statically maintained list to bootstrap getting the more valuable dynamically maintained list.

Server Configuration

In the server this list can be specified via the conf/multipoint.properties file like so:

...

Configuration in the client is similar, but note that EJB clients do not participate directly in multipoint communication and do not connect to the multipoint port. The server list is simply a list of the regular "ejbd://" urls that a client normally uses to connect to a server.

   Properties propertiesp = new Properties();
   properties.setPropertyp.put(Context.INITIAL_CONTEXT_FACTORY, "org.apache.openejb.client.RemoteInitialContextFactory");
   properties.setPropertyp.put(Context.PROVIDER_URL, "failover:ejbd://192.168.1.20:4201,ejbd://192.168.1.30:4201,ejbd://192.168.1.40:4201");
   InitialContext remoteContext = new InitialContext(propertiesp);

Considerations

Network size

...

It is possible in this process that two servers learn of each other at the same time and each attempts to connect to the other simultaneously, resulting in two connections between the same two servers. When this happens both servers will detect the extra connection and one of the connections will be dropped and one will be kept. In practice this race condition rarely happens and can be avoided almost entirely by fanning out server startup by as little as 100 milliseconds.

Multipoint

...

Configuration Recommendations

When setting up As mentioned above the initialServers property of the conf/multipoint.properties file, it is important to keep in mind that the connection attempt is only made once when the server starts, so if that server is not running they will never be linked – at least not initially is only used for bootstrapping the multipoint network. Once running, all servers will dynamically establish direct connections with each other and there is no single point of failure.

However to ensure that the bootstrapping process can occur successfully, the initialServers property of the conf/multipoint.properties file must be set carefully and with a specific server start order in mind. Each server consults its initialServers list exactly once in the bootstrapping phase at startup, after that time connections are made dynamically.

This means that at least one of the servers listed in initialServers must already be running when the server starts or the server might never become introduced and connected to all the other servers in the network.

Failed scenario (background)

As an example of a failed scenario, imagine there are three servers; server1, server2, server3. They are setup only to point to the server in front of them making a chain:

...

The above setup is easily fixable via better configuration. If server3 listed both server1 and server2 in its initialServers list, rather than listing nothing at all, then all servers would fully discover each other regardless of startup order; assuming all three servers did eventually start.

...

Bootstrapping Three Servers or Less

In a three sever scenario, we recommend simply having all three servers list all three servers.

...

There's no harm to a server listing itself. It gives you one clean list to maintain and it will work even if you decide not to start one of the three servers.

...

Bootstrapping Four Servers or More

In a scenario of four or more, we recommend picking at least to servers and focus on always keeping at least one of them running. Lets refer to them as "root" servers for simplicity sake.

...

Of course all servers once running and connected will have a full list of all other servers in the network, so if at any time the "root" servers weren't around to make initial introductions to new servers it would be no trouble. It's possible to reconfigure new servers to point at any other server in the network as all servers will have the full list. So these "root" servers are no real point of failure in function, but only of convenience.

Command line overrides

Always remember that any property in a conf/<server-service>.properties file can be overridden on the command line or via system properties. So it is possible easily set the initialServers list in startup scripts.

...