Qpid has a substantial server performance test suite based around the junit-toolkit library.
The test cases fall into the following categories:
Throughput - straight line 0-600,000 messages/second in how long?
Latency - how quickly can I get a few messages?
Reliability - burn in, stress and soak tests
Building and running the tests
The performance tests live in qpid/java/perftests. When they are built the scripts are generated and placed into qpid/java/build/bin/perftests. These are detailed in the relevant list of test cases above. There are 3 convinience scripts which correspond to the categories to run all the throughput, latency or reliability tests.
Test case implementation
All test cases utilise one of a couple of test classes, with a different set of command line parameters.
The following parameters may be passed to the test runner to control its operation:
Parameter |
Meaning |
---|---|
-c pattern |
The number of tests to run concurrently. |
-r num |
The number of times to repeat each test. |
-d duration |
The length of time to run the tests for. |
-t name |
The name of the test case to execute. |
-s pattern |
The size parameter to run tests with. |
-o dir |
The name of the directory to output test timings to. |
--csv |
Output test results in CSV format. |
--xml |
Output test results in XML format. |
-v |
Verbose mode. |
Here are some examples:
-c [10:20:30:40:50] |
Runs the test with 10,20,...,50 threads. |
-s [1:100]:samples=10 |
Runs the test with ten different size parameters evenly spaced between 1 and 100. |
-s [1:1000000]:samples=10:exp |
Runs the test with ten different size parameters exponentially spaced between 1 and 1000000. |
-r 10 |
Runs each test ten times. |
-d 10H |
Runs the test repeatedly for 10 hours. |
-d 1M, -r 10 |
Runs the test repeatedly for 1 minute but only takes a timing sample every 10 test runs. |
-r 10, -c [1:5:10:50], -s [100:1000:10000] |
Runs 12 test cycles (4 concurrency samples * 3 size sample), with 10 repeats each. In total the test will be run 199 times (3 + 15 + 30 + 150) |
The test runner also accepts name=value properties on the end of the command line, and these are passed to the test code as parameters to control the operation of the test. The following properties may be set:
Parameter |
Default |
Comments |
---|---|---|
messageSize |
0 |
Message size in bytes. Not including any headers. |
destinationName |
ping |
The root name to use to generate destination names to ping. |
persistent |
false |
Determines whether peristent delivery is used. |
transacted |
false |
Determines whether messages are sent/received in transactions. |
broker |
tcp://localhost:5672 |
Determines the broker to connect to. |
virtualHost |
test |
Determines the virtual host to send all ping over. |
rate |
0 |
The maximum rate (in hertz) to send messages at. 0 means no limit. |
verbose |
false |
The verbose flag for debugging. Prints to console on every message. |
pubsub |
false |
Whether to ping topics or queues. Uses p2p by default. |
failAfterCommit |
false |
Whether to prompt user to kill broker after a commit batch. |
failBeforeCommit |
false |
Whether to prompt user to kill broker before a commit batch. |
failAfterSend |
false |
Whether to prompt user to kill broker after a send. |
failBeforeSend |
false |
Whether to prompt user to kill broker before a send. |
failOnce |
true |
Whether to prompt for failover only once. |
username |
guest |
The username to access the broker with. |
password |
guest |
The password to access the broker with. |
selector |
null |
Not used. Defines a message selector to filter pings with. |
destinationCount |
1 |
The number of destinations to send pings to. |
numConsumers |
1 |
The number of consumers on each destination. |
timeout |
30000 |
In milliseconds. The timeout to stop waiting for replies. |
commitBatchSize |
1 |
The number of messages per transaction in transactional mode. |
uniqueDests |
true |
Whether each receivers only listens to one ping destination or all. |
durableDests |
false |
Whether or not durable destinations are used. |
ackMode |
AUTO_ACK |
The message acknowledgement mode. Possible values are: 0 - SESSION_TRANSACTED 1 - AUTO_ACKNOWLEDGE 2 - CLIENT_ACKNOWLEDGE 3 - DUPS_OK_ACKNOWLEDGE 257 - NO_ACKNOWLEDGE 258 - PRE_ACKNOWLEDGE |
consTransacted |
false |
Whether or not consumers use transactions. Defaults to the same value |
consAckMode |
AUTO_ACK |
The message acknowledgement mode for consumers. Defaults to the same |
maxPending |
0 |
The maximum size in bytes, of messages sent but not yet received. |
Added for 0.7
Parameter |
Default |
Comments |
---|---|---|
numConsumers |
<int> |
Augmented Allow a value of 0, meaning no consumers. |
consumeOnly |
boolean |
Disable all message sending. Message counts are used by consumers as validation. |
preFill |
int |
Message count to preFill the destination with before the test start. |
delayBeforeConsume |
int |
Delay in ms to wait after the preFill has occured before the test starts. |
Test case output
Test cases output data in csv format. The extratThroughputResults.sh script can be used to interpret the data and output the average throughput rate.
2 Comments
Jonathan Robie
This page ignores the existence of the C++ perftest, latencytest, etc.
I assume that these client-side tools are broadly similar, but written in different languages? If so, the documentation for each should point to the other, and we should think about how to organize the content so that users can find both.
Aidan Skinner
Yeah, I forget that cwiki would flatten this down - in my head this is sitting beneath java developer docs.
I don't think either set of tooling is a strict subset of the other, but I'm not really sure yet since the qpid/java/perftests stuff isn't very well documented. :/
It might make sense to have all the docs for this sort of thing in a seperate bit of their own, there's no real reason for the split that I can see.