This Confluence has been LDAP enabled, if you are an ASF Committer, please use your LDAP Credentials to login. Any problems file an INFRA jira ticket please.

Child pages
  • Networks of Brokers
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 33 Next »

To provide massive scalability of a large messaging fabric you typically want to allow many brokers to be connected together into a network so that you can have as many clients as you wish all logically connected together - and running as many message brokers as you need based on your number of clients and network topology.

If you are using client/server or hub/spoke style topology then the broker you connect to becomes a single point of failure which is another reason for wanting a network (or cluster) of brokers so that you can survive failure of any particular broker, machine or subnet.

From 1.1 onwards of ActiveMQ supports networks of brokers which allows us to support distributed queues and topics across a network of brokers.

This allows a client to connect to any broker in the network - and fail over to another broker if there is a failure - providing from the clients perspective a HA cluster of brokers.

N.B. By default a network connection is one way only - the broker that establishes the connection passes messages to the broker(s) its connected to. From version 5.x of ActiveMQ, a network connection can be optionally enabled to be duplex, which can be useful for hub and spoke architectures, where the hub is behind a firewall etc.

Configuring a network of brokers

The easiest way to configure a network of brokers is via the Xml Configuration. There are two main ways to create a network of brokers

  • use a hard coded list of URIs (the networkConnector / networkChannel elements in the activemq.xsd.html ).
  • use Discovery to detect brokers (multicast or rendezvous).

Example with a fixed list of URIs

Here is an example of using the fixed list of URIs

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://activemq.org/config/1.0">

  <broker brokerName="receiver" persistent="false" useJmx="false">
    <transportConnectors>
      <transportConnector uri="tcp://localhost:62002"/>
    </transportConnectors>

    <networkConnectors>
      <networkConnector uri="static:(tcp://localhost:62001)"/>
    </networkConnectors>

    <persistenceAdapter>
      <memoryPersistenceAdapter/>
    </persistenceAdapter>
  </broker>

</beans>

Example using multicast discovery

This example uses multicast discovery

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://activemq.org/config/1.0">

  <broker name="sender" persistent="false" useJmx="false">
    <transportConnectors>
      <transportConnector uri="tcp://localhost:0" discoveryUri="multicast://default"/>
    </transportConnectors>

    <networkConnectors>
      <networkConnector uri="multicast://default"/>
    </networkConnectors>

    <persistenceAdapter>
      <memoryPersistenceAdapter/>
    </persistenceAdapter>
  </broker>

</beans>

Connecting to a network of brokers

if you are not using discovery, you can just use a hard coded list of URLs and you will connect randomly to one of the available brokers.

uri="static:(tcp://host1:61616,tcp://host2:61616,tcp://..)"

There are some useful properties you can set on a static network connector for retries:

property

default

description

initialReconnectDelay

1000

time(ms) to wait before attempting a reconnect (if useExponentialBackOff is false)

maxReconnectDelay

30000

time(ms) to wait before attempting to re-connect

useExponentialBackOff

true

increases time between reconnect for every failure in a reconnect sequence

backOffMultiplier

2

multipler used to increase the wait time if using exponential back off

e.g.

uri="static:(tcp://host1:61616,tcp://host2:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"

NetworkConnector Properties

property

default

description

name

bridge

name of the network - for more than one network connector between the same two brokers - use different names

dynamicOnly

false

if true, only forward messages if a consumer is active on the connected broker

decreaseNetworkConsumerPriority

false

decrease the priority for dispatching to a Queue consumer the further away it is (in network hops) from the producer

networkTTL

1

the number of brokers in the network that messages and subscriptions can pass through

conduitSubscriptions

true

multiple consumers subscribing to the same destination are treated as one consumer by the network

excludedDestinations

empty

destinations matching this list won't be forwarded across the network

dynamicallyIncludedDestinations

empty

destinations that match this list will be forwarded across the network n.b. an empty list means all destinations not in the exluded list will be forwarded

staticallyIncludedDestinations

empty

destinations that match will always be passed across the network - even if no consumers have ever registered an interest

duplex

false

if true, a network connection will be used to both produce AND Consume messages. This is useful for hub and spoke scenarios when the hub is behind a firewall etc.

When to use and not use Conduit subscriptions

ActiveMQ relies on information about active consumers (subscriptions) to pass messages around the network. A broker interprets a subscription from a remote (networked) broker in the same way as it would a subscription from a local client connection and routes a copy of any relevant message to each subscription. With Topic subscriptions and with more than one remote subscription, a remote broker would interpret each message copy as valid, so when it in turns routes the messages to its own local connections, duplicates would occur. Hence default conduit behavior consolidates all matching subscription information to prevent duplicates flowing around the network. With this default behaviour, N subscriptions on a remote broker look like a single subscription to the networked broker.

However - duplicate subscriptions is a useful feature to exploit if you are only using Queues. As the load balancing algorithm will attempt to share message load evenly, consumers across a network will equally share the message load only if the flag conduitSubscriptions=false. Here's an example. Suppose you have two brokers, A and B, that are connected to one another via a forwarding bridge. Connected to broker A, you have a consumer that subscribes to a queue called Q.TEST. Connected to broker B, you have two consumers that also subscribe to Q.TEST. All consumers have equal priority. Then you start a producer on broker A that writes 30 messages to Q.TEST. By default, (conduitSubscriptions=true), 15 messages will be sent to the consumer on broker A and the resulting 15 messages will be sent to the two consumers on broker B. The message load has not been equally spread across all three consumers because, by default, broker A views the two subscriptions on broker B as one. If you had set conduitSubscriptions to "false", then each of the three consumers would have been given 10 messages.

Conduit subscriptions and consumer selectors

Conduit subscriptions ignore consumer selectors on the local broker and send all messages to the remote one. Selectors are then parsed on the remote brokers before messages are dispatched to consumers. This concept could create some problems with consuming on queues using selectors in a multi-broker network. Imagine a situation when you have a producing broker forwarding messages to two receiving brokers and each of these two brokers have a consumer with different selector. Since no selectors are evaluated on the producer broker side, you can end up with all messages going to only one of the brokers, so messages with certain property will not be consumed. If you need to support this use case, please turn off conduitSubscription feature.

Configuration Pitfalls

Networks do not work if the advisorySupport broker property is disabled

If dynamicOnly is set to false and the networkTTL to a low number it's possible that all the messages will end up at a certain broker with no networkTTL left. Therefore you will not be able to receive this messages from a queue if your client isn't connected to this specific broker. To avoid this behavior be sure to set the TTL to a high enough value (at very least the number of brokers the message will pass though in the worst case) and, more important, to set dynamicOnly to true.
This misconfiguration may also result in a delivery of only every second message to the actual client and the other half of the messages getting 'lost' within the network of broker.

Example Configuration using NetworkConnector properties

This part of an example configuration for a Broker

<networkConnectors>
      <networkConnector uri="static://(tcp://localhost:61617)"
         name="bridge"
         dynamicOnly="false"
         conduitSubscriptions="true"
         decreaseNetworkConsumerPriority="false">
      	<excludedDestinations>
      		<queue physicalName="exclude.test.foo"/>
      		<topic physicalName="exclude.test.bar"/>
      	</excludedDestinations>
      	<dynamicallyIncludedDestinations>
      		<queue physicalName="include.test.foo"/>
      		<topic physicalName="include.test.bar"/>
      	</dynamicallyIncludedDestinations>
        <staticallyIncludedDestinations>
      		<queue physicalName="always.include.queue"/>
      		<topic physicalName="always.include.topic"/>
      	</staticallyIncludedDestinations>
      </networkConnector>
    </networkConnectors>

It is possible to have more than one network connector between two brokers. Each network connector uses one underlying transport connection, so you may wish to do this to increase throughput, or have a more flexible configuration.
For example, if using distributed queues, you may wish to have equivalent weighting to queue receivers across the network, but only when the receivers are active - e.g.

<networkConnectors>
      <networkConnector uri="static://(tcp://localhost:61617)"
         name="queues_only"
         dynamicOnly="true"
         conduitSubscriptions="false"
         decreaseNetworkConsumerPriority="false">
      	<excludedDestinations>
      		<topic physicalName=">"/>
      	</excludedDestinations>
      </networkConnector>
    </networkConnectors>

N.B. You can use wildcards in inclusive , exclusive destination properties
N.B. Do not change the name of the bridge or the name of the Broker if you are using durable topic subscribers across the network. Internally ActiceMQ uses the network name and broker name to build a unique but repeatable durable subscriber name for the network.

Trying out using a network of brokers

If you run the following commands in separate shells you'll have 2 brokers auto-discovering themselves and 2 clients using fixed-URLs

maven -o server -Dconfig=xbean:file:src/test/resources/org/apache/activemq/usecases/receiver.xml
maven -o server -Dconfig=xbean:file:src/test/resources/org/apache/activemq/usecases/sender.xml
maven -o consumer -Durl=tcp://localhost:62002
maven -o producer -Durl=tcp://localhost:62001

Or to try the same thing again using Zeroconf discovery you could try this

maven -o server -Dconfig=src/test/org/activemq/usecases/receiver-zeroconf.xml
maven -o server -Dconfig=src/test/org/activemq/usecases/sender-zeroconf.xml
maven -o consumer -Durl=tcp://localhost:62002
maven -o producer -Durl=tcp://localhost:62001
  • No labels