Qpid has a substantial server performance test suite based around the junit-toolkit library.

The test cases fall into the following categories:

Throughput - straight line 0-600,000 messages/second in how long?
Latency - how quickly can I get a few messages?
Reliability - burn in, stress and soak tests

Building and running the tests

The performance tests live in qpid/java/perftests. When they are built the scripts are generated and placed into qpid/java/build/bin/perftests. These are detailed in the relevant list of test cases above. There are 3 convinience scripts which correspond to the categories to run all the throughput, latency or reliability tests.

Test case implementation

All test cases utilise one of a couple of test classes, with a different set of command line parameters.

The following parameters may be passed to the test runner to control its operation:

Parameter

Meaning

-c pattern

The number of tests to run concurrently.

-r num

The number of times to repeat each test.

-d duration

The length of time to run the tests for.

-t name

The name of the test case to execute.

-s pattern

The size parameter to run tests with.

-o dir

The name of the directory to output test timings to.

--csv

Output test results in CSV format.

--xml

Output test results in XML format.

-v

Verbose mode.

Here are some examples:

-c [10:20:30:40:50]

Runs the test with 10,20,...,50 threads.

-s [1:100]:samples=10

Runs the test with ten different size parameters evenly spaced between 1 and 100.

-s [1:1000000]:samples=10:exp

Runs the test with ten different size parameters exponentially spaced between 1 and 1000000.

-r 10

Runs each test ten times.

-d 10H

Runs the test repeatedly for 10 hours.

-d 1M, -r 10

Runs the test repeatedly for 1 minute but only takes a timing sample every 10 test runs.

-r 10, -c [1:5:10:50], -s [100:1000:10000]

Runs 12 test cycles (4 concurrency samples * 3 size sample), with 10 repeats each. In total the test will be run 199 times (3 + 15 + 30 + 150)

The test runner also accepts name=value properties on the end of the command line, and these are passed to the test code as parameters to control the operation of the test. The following properties may be set:

Parameter

Default

Comments

messageSize

0

Message size in bytes. Not including any headers.

destinationName

ping

The root name to use to generate destination names to ping.

persistent

false

Determines whether peristent delivery is used.

transacted

false

Determines whether messages are sent/received in transactions.

broker

tcp://localhost:5672

Determines the broker to connect to.

virtualHost

test

Determines the virtual host to send all ping over.

rate

0

The maximum rate (in hertz) to send messages at. 0 means no limit.

verbose

false

The verbose flag for debugging. Prints to console on every message.

pubsub

false

Whether to ping topics or queues. Uses p2p by default.

failAfterCommit

false

Whether to prompt user to kill broker after a commit batch.

failBeforeCommit

false

Whether to prompt user to kill broker before a commit batch.

failAfterSend

false

Whether to prompt user to kill broker after a send.

failBeforeSend

false

Whether to prompt user to kill broker before a send.

failOnce

true

Whether to prompt for failover only once.

username

guest

The username to access the broker with.

password

guest

The password to access the broker with.

selector

null

Not used. Defines a message selector to filter pings with.

destinationCount

1

The number of destinations to send pings to.

numConsumers

1

The number of consumers on each destination.

timeout

30000

In milliseconds. The timeout to stop waiting for replies.

commitBatchSize

1

The number of messages per transaction in transactional mode.

uniqueDests

true

Whether each receivers only listens to one ping destination or all.

durableDests

false

Whether or not durable destinations are used.

ackMode

AUTO_ACK

The message acknowledgement mode. Possible values are:

      0 - SESSION_TRANSACTED
      1 - AUTO_ACKNOWLEDGE
      2 - CLIENT_ACKNOWLEDGE
      3 - DUPS_OK_ACKNOWLEDGE
      257 - NO_ACKNOWLEDGE
      258 - PRE_ACKNOWLEDGE

consTransacted

false

Whether or not consumers use transactions. Defaults to the same value
as the 'transacted' option if not seperately defined.

consAckMode

AUTO_ACK

The message acknowledgement mode for consumers. Defaults to the same
value as 'ackMode' if not seperately defined.

maxPending

0

The maximum size in bytes, of messages sent but not yet received.
Limits the volume of messages currently buffered on the client
or broker. Can help scale test clients by limiting amount of buffered
data to avoid out of memory errors.

Added for 0.7

Parameter

Default

Comments

numConsumers

<int>

Augmented Allow a value of 0, meaning no consumers.

consumeOnly

boolean

Disable all message sending. Message counts are used by consumers as validation.

preFill

int

Message count to preFill the destination with before the test start.

delayBeforeConsume

int

Delay in ms to wait after the preFill has occured before the test starts.

Test case output

Test cases output data in csv format. The extratThroughputResults.sh script can be used to interpret the data and output the average throughput rate.

  • No labels

2 Comments

  1. Jonathan Robie

    This page ignores the existence of the C++ perftest, latencytest, etc.

    I assume that these client-side tools are broadly similar, but written in different languages? If so, the documentation for each should point to the other, and we should think about how to organize the content so that users can find both.

    1. Aidan Skinner

      Yeah, I forget that cwiki would flatten this down - in my head this is sitting beneath java developer docs.

      I don't think either set of tooling is a strict subset of the other, but I'm not really sure yet since the qpid/java/perftests stuff isn't very well documented. :/

      It might make sense to have all the docs for this sort of thing in a seperate bit of their own, there's no real reason for the split that I can see.