ZooKeeper service latencies under various loads & configurations

Author: http://people.apache.org/~phunt/ (follow me on http://twitter.com/phunt)

The intent of this document is to examine the performance of a typical ZooKeeper service while varying the load and available computing power. This is not an exhaustive test but meant to shed some light on the amount of load ZK can support for various (lower powered) hardware configurations. In particular I was interested in seeing how hosts running ZooKeeper servers on shared disks performed - while varying the number of cpus and clients.

Hardware

All systems were dual quad core Intel(R) Xeon(R) CPUs running at 2.50GHz. Eight cores were available however as noted below Linux's processor affinity feature was used to limit CPU availability to the JVM. In the tests below I use 1, 2, or 4 cores from the first CPU (taskset 0x01 or 0x03 or 0x0f respectively).

All systems had 16gig ECC memory available, however unless specifically noted the JVM's -Xmx option was used to limit the size of the JVM heap to 512m.

All systems had 7200RPM SATA drives

  • hdparm -tT reported:
    • Timing cached reads: 22980 MB in 1.99 seconds = 11532.40 MB/sec
    • Timing buffered disk reads: 266 MB in 3.01 seconds = 88.29 MB/sec
  • time dd if=/dev/urandom bs=512000 of=/tmp/memtest count=1050
    • 537600000 bytes (538 MB) copied, 73.9991 seconds, 7.3 MB/s
    • real 1m14.001s
    • user 0m0.000s
    • sys 1m13.995s
  • time md5sum /tmp/memtest; time md5sum /tmp/memtest; time md5sum /tmp/memtest
    • real 0m1.498s
    • user 0m1.284s
    • sys 0m0.214s

All systems are interconnected with 1gig ethernet, approximate ping time is steady at 0.09ms roundtrip.

  • scp host60:/tmp/test /tmp/test
    • test 100% 513MB 64.1MB/s 00:08

Operating System

Linux version 2.6.18-53.1.13.el5 compiled using gcc version 4.1.2 20070626 (Red Hat 4.1.2-14))

Software

  • ZooKeeper 3.2.1
  • java version "1.6.0_05" (build 1.6.0_05-b13)
  • Java ?HotSpot(TM) Server VM (build 10.0-b19, mixed mode)

ZooKeeper Configuration

The following basic configuration was used on all serving nodes:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=<data directory>
clientPort=2181

A dedicated log device was not used, the data directory is store on the same device as the operating system, the ZK snapshots and log4j logs.

The standard zkServer.sh script that ships with the release was used to start the service. No modification were made other than to control processor affinity and heap size as noted in this document.

Test Client

Result Summary

In the following tests we are varying the number of cores and clients workloads. The clients are homogeneous, each is performing 110,000 asynchronous operations against the service, where the workload is:

  • create 10k permanent znodes of size 100 bytes
  • set 10k znodes of size 100 bytes
  • get 10k znodes (100 bytes returned)
  • delete all 10k znodes
  • create 10k ephemeral znodes of size 100 bytes
  • watch each of the 10k znodes 5 times (50k watches)
  • delete 10k znodes and measure time for notifications to be received by the client (from the 50k watches)

Of the 110k operations 50k are write operations, the remaining 60k are read operations. Read operations being local to the connected server while writes must be propagated through the leader & atomic broadcast protocol (ie copied to all servers in an ensemble)

Standalone Result

In the following graph you can see the results of this experiment for a single Standalone serving instance. The X axis shows the number of operations per second that were processed by the server for 1, 2 & 4 cores, where 1, 10 & 20 clients were testing the cluster.

Standalone Server

From the graph and the data below we can see that moving from a workload of 1 to 10 clients shows an increase in overall operations performed by the server. However moving from 10 to 20 clients we see no further increase (rather some decrease, potentially due to the overhead of selector processing and connection management in the server networking code). Increasing the number of cores provides modest increase in the overall output of the server. This makes sense as there are a limited number of threads running in a standalone server, it is unable to take advantage of the additional cores.

Ensemble with 3 Serving Nodes Result

In the following graph you can see the results of this experiment for a 3 Server Ensemble. The X axis shows the number of operations per second that were processed by the server for 1, 2 & 4 cores, where 1, 10 & 20 clients were testing the cluster.

Ensemble with 3 Serving Nodes

Here we see distinct increase in the output of the cluster as we move the workload from 1 to 10 to 20 clients. There is some additional headroom that we could see if we were to test with higher workloads. Additionally as the core count increases we see output gains there as well. Ensemble servers are running additional threads to process the quorum communication, these threads are taking advantage of the additional parallelism.

Conclusion

I hope this sheds more light on the tradeoffs that can be made when setting up a ZK cluster. In particular we can see the performance of a small cluster (reliable even in the face of a single server going down) is quite solid even under a heavy (30k+ operations per second on dual core with a high write to read ratio, much higher than you should typically see in a production deployment) client workload. Additionally the session stability was rock solid - no timeouts or session expirations were seen during any of the tests.

Notes, Futures

I used the out of the box zkServer.sh for this test. I did look at using the incremental garbage collector however as far as I could tell there was no benefit seen for these tests. It might be interesting to look at this more, is there some benefit to be had?

I also did not use the --server flag, again, I did some limited testing with this and afaics there is no benefit to using this option with a ZK cluster.

It would be interesting to test other workloads. Here I've weighted things more to a balanced write/read workload. In production deployments we typically see a heavy read dominant workload. In this case the service performance should be even better than what we are seeing here.

Additionally it might be interesting to try tuning the ZK configuration parameters. For example to examine the effects of leaderServes and forceSync options (to name just two) on overall performance and latency.

During the tests the snapshot (on disk copy of the znode data) reached approximately 100meg in size (as mentioned previously the JVM heap is limited to 512 meg), by default transactional log files are pre-allocated to ~60meg. Additional snapshots and logs are written as needed. Depending on your workload, the frequency of your snap/log cleanup, and your cluster configuration you should allow sufficient storage space. For example running a 20 client test required approximately 5 gig of disk storage. (the tests were not cleaning up old snaps/logs, so this was accumulated over the test lifetime).

Result Details

the results are as follows for standalone and 3 node ensemble

Note: in each case below I give only a single client's results, regardless of the number of clients run in the tests. I do this to limit the length of this document, the results that I do show are for the last, longest running client, so in general these are the worst performing results of any client in the test. I also verify that all clients completed successfully and without warnings or errors in the client log.

Results for Standalone Server

The Standalone version of ZooKeeper was run on a single host ("host60").

The client(s) was run as follows unless otherwise noted:

zk-latencies.py --cluster "host60:2181" --root_znode=/{{hostname}} --znode_size=100 --znode_count=10000 --timeout=5000 --watch_multiple=5

This means each client:

  • has a 5 second timeout
  • creates 10k znodes of size 100 bytes
  • sets 5 watches on each of the znodes it creates, for 50k znodes by each client

As a result, each client will send ~110,000 requests to the server during each test.

1 Core - 1 Client

110k operations against the server with elapsed time of 18 seconds

host40: Testing latencies on server host60:2181 using asynchronous calls
host40: created   10000 permanent znodes  in   1219 ms (0.121911 ms/op 8202.684317/sec)
host40: set       10000           znodes  in   1600 ms (0.160094 ms/op 6246.310789/sec)
host40: get       10000           znodes  in   1744 ms (0.174442 ms/op 5732.564236/sec)
host40: deleted   10000 permanent znodes  in   1386 ms (0.138648 ms/op 7212.503931/sec)
host40: created   10000 ephemeral znodes  in   1608 ms (0.160836 ms/op 6217.525272/sec)
host40: watched   50000           znodes  in   7509 ms (0.150180 ms/op 6658.670068/sec)
host40: deleted   10000 ephemeral znodes  in   2139 ms (0.213949 ms/op 4674.011338/sec)
host40: notif     50000           watches in      0 ms (included in prior)
host40: Latency test complete
host40: 15.20user 7.06system 0:17.74elapsed 125%CPU (0avgtext+0avgdata 0maxresident)k
host40: 0inputs+0outputs (0major+71244minor)pagefaults 0swaps

1 Core - 10 Client

1.1 million operations against the server with elapsed time of 58 seconds

host46: Testing latencies on server host60:2181 using asynchronous calls
host46: created   10000 permanent znodes  in   6493 ms (0.649358 ms/op 1539.982995/sec)
host46: set       10000           znodes  in   4538 ms (0.453877 ms/op 2203.239641/sec)
host46: get       10000           znodes  in  13229 ms (1.322971 ms/op 755.874244/sec)
host46: deleted   10000 permanent znodes  in   4967 ms (0.496763 ms/op 2013.030379/sec)
host46: created   10000 ephemeral znodes  in   5196 ms (0.519624 ms/op 1924.468834/sec)
host46: watched   50000           znodes  in  17421 ms (0.348424 ms/op 2870.067528/sec)
host46: deleted   10000 ephemeral znodes  in   4901 ms (0.490112 ms/op 2040.351571/sec)
host46: notif     50000           watches in      0 ms (included in prior)
host46: Latency test complete
host46: 15.94user 6.40system 0:57.40elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k
host46: 0inputs+0outputs (2major+69981minor)pagefaults 0swaps

1 Core - 20 Client

2.2 million operations against the server with elapsed time of 150 seconds

host67: Testing latencies on server host60:2181 using asynchronous calls
host67: created   10000 permanent znodes  in  12881 ms (1.288103 ms/op 776.335485/sec)
host67: set       10000           znodes  in  10448 ms (1.044807 ms/op 957.114388/sec)
host67: get       10000           znodes  in  14859 ms (1.485911 ms/op 672.988006/sec)
host67: deleted   10000 permanent znodes  in  17420 ms (1.742049 ms/op 574.036736/sec)
host67: created   10000 ephemeral znodes  in  13450 ms (1.345097 ms/op 743.440587/sec)
host67: watched   50000           znodes  in  53507 ms (1.070143 ms/op 934.454518/sec)
host67: deleted   10000 ephemeral znodes  in  26196 ms (2.619615 ms/op 381.735542/sec)
host67: notif     50000           watches in      0 ms (included in prior)
host67: Latency test complete
host67: 16.16user 6.43system 2:29.38elapsed 15%CPU (0avgtext+0avgdata 0maxresident)k
host67: 0inputs+0outputs (2major+73149minor)pagefaults 0swaps

2 Core - 1 Client

110k operations against the server with elapsed time of 20 seconds

host40: Testing latencies on server host60:2181 using asynchronous calls
host40: created   10000 permanent znodes  in   2420 ms (0.242055 ms/op 4131.292291/sec)
host40: set       10000           znodes  in   1613 ms (0.161318 ms/op 6198.940227/sec)
host40: get       10000           znodes  in   1694 ms (0.169480 ms/op 5900.384144/sec)
host40: deleted   10000 permanent znodes  in   1440 ms (0.144013 ms/op 6943.822193/sec)
host40: created   10000 ephemeral znodes  in   1589 ms (0.158944 ms/op 6291.516084/sec)
host40: watched   50000           znodes  in   9442 ms (0.188850 ms/op 5295.203361/sec)
host40: deleted   10000 ephemeral znodes  in   2272 ms (0.227226 ms/op 4400.912310/sec)
host40: notif     50000           watches in      0 ms (included in prior)
host40: Latency test complete
host40: 16.97user 8.17system 0:20.96elapsed 119%CPU (0avgtext+0avgdata 0maxresident)k
host40: 0inputs+0outputs (2major+73972minor)pagefaults 0swaps

2 Core - 10 Client

1.1 million operations against the server with elapsed time of 53 seconds

host49: Testing latencies on server host60:2181 using asynchronous calls
host49: created   10000 permanent znodes  in   5316 ms (0.531661 ms/op 1880.897081/sec)
host49: set       10000           znodes  in   4549 ms (0.454980 ms/op 2197.897811/sec)
host49: get       10000           znodes  in   3273 ms (0.327330 ms/op 3055.021880/sec)
host49: deleted   10000 permanent znodes  in   9378 ms (0.937850 ms/op 1066.268703/sec)
host49: created   10000 ephemeral znodes  in   5195 ms (0.519526 ms/op 1924.832170/sec)
host49: watched   50000           znodes  in  17633 ms (0.352667 ms/op 2835.533635/sec)
host49: deleted   10000 ephemeral znodes  in   7606 ms (0.760661 ms/op 1314.646067/sec)
host49: notif     50000           watches in      0 ms (included in prior)
host49: Latency test complete
host49: 16.26user 6.74system 0:53.65elapsed 42%CPU (0avgtext+0avgdata 0maxresident)k
host49: 0inputs+0outputs (2major+71633minor)pagefaults 0swaps

2 Core - 20 Client

2.2 million operations against the server with elapsed time of 108 seconds

host46: Testing latencies on server host60:2181 using asynchronous calls
host46: created   10000 permanent znodes  in  10765 ms (1.076519 ms/op 928.920312/sec)
host46: set       10000           znodes  in  13262 ms (1.326232 ms/op 754.015764/sec)
host46: get       10000           znodes  in  17171 ms (1.717179 ms/op 582.350402/sec)
host46: deleted   10000 permanent znodes  in  10307 ms (1.030792 ms/op 970.127826/sec)
host46: created   10000 ephemeral znodes  in   9636 ms (0.963677 ms/op 1037.692397/sec)
host46: watched   50000           znodes  in  31042 ms (0.620850 ms/op 1610.695533/sec)
host46: deleted   10000 ephemeral znodes  in  13254 ms (1.325475 ms/op 754.446396/sec)
host46: notif     50000           watches in      0 ms (included in prior)
host46: Latency test complete
host46: 18.18user 6.09system 1:48.16elapsed 22%CPU (0avgtext+0avgdata 0maxresident)k
host46: 0inputs+0outputs (0major+73188minor)pagefaults 0swaps

4 Core - 1 Client

110k operations against the server with elapsed time of 20 seconds

host40: Testing latencies on server host60:2181 using asynchronous calls
host40: created   10000 permanent znodes  in   1628 ms (0.162839 ms/op 6141.034732/sec)
host40: set       10000           znodes  in   1583 ms (0.158326 ms/op 6316.066553/sec)
host40: get       10000           znodes  in   1655 ms (0.165526 ms/op 6041.344054/sec)
host40: deleted   10000 permanent znodes  in   1463 ms (0.146302 ms/op 6835.186174/sec)
host40: created   10000 ephemeral znodes  in   1591 ms (0.159110 ms/op 6284.971481/sec)
host40: watched   50000           znodes  in   9426 ms (0.188539 ms/op 5303.937335/sec)
host40: deleted   10000 ephemeral znodes  in   2254 ms (0.225496 ms/op 4434.673973/sec)
host40: notif     50000           watches in      0 ms (included in prior)
host40: Latency test complete
host40: 17.03user 8.25system 0:20.14elapsed 125%CPU (0avgtext+0avgdata 0maxresident)k
host40: 0inputs+0outputs (0major+72575minor)pagefaults 0swaps

4 Core - 10 Client

1.1 million operations against the server with elapsed time of 43 seconds

host43: Testing latencies on server host60:2181 using asynchronous calls
host43: created   10000 permanent znodes  in   5421 ms (0.542148 ms/op 1844.515451/sec)
host43: set       10000           znodes  in   5050 ms (0.505025 ms/op 1980.098852/sec)
host43: get       10000           znodes  in   2831 ms (0.283189 ms/op 3531.208094/sec)
host43: deleted   10000 permanent znodes  in   3682 ms (0.368277 ms/op 2715.345120/sec)
host43: created   10000 ephemeral znodes  in   6300 ms (0.630003 ms/op 1587.294271/sec)
host43: watched   50000           znodes  in  13783 ms (0.275669 ms/op 3627.543622/sec)
host43: deleted   10000 ephemeral znodes  in   5911 ms (0.591175 ms/op 1691.546474/sec)
host43: notif     50000           watches in      0 ms (included in prior)
host43: Latency test complete
host43: 16.40user 6.93system 0:43.63elapsed 53%CPU (0avgtext+0avgdata 0maxresident)k
host43: 0inputs+0outputs (2major+71052minor)pagefaults 0swaps

4 Core - 20 Client

2.2 million operations against the server with elapsed time of 95 seconds

host65: Testing latencies on server host60:2181 using asynchronous calls
host65: created   10000 permanent znodes  in  11458 ms (1.145814 ms/op 872.742079/sec)
host65: set       10000           znodes  in   7248 ms (0.724857 ms/op 1379.582825/sec)
host65: get       10000           znodes  in   8641 ms (0.864138 ms/op 1157.222181/sec)
host65: deleted   10000 permanent znodes  in  11239 ms (1.123901 ms/op 889.758416/sec)
host65: created   10000 ephemeral znodes  in  13059 ms (1.305911 ms/op 765.748972/sec)
host65: watched   50000           znodes  in  32384 ms (0.647690 ms/op 1543.949402/sec)
host65: deleted   10000 ephemeral znodes  in  10303 ms (1.030382 ms/op 970.514240/sec)
host65: notif     50000           watches in      0 ms (included in prior)
host65: Latency test complete
host65: 14.64user 5.45system 1:35.02elapsed 21%CPU (0avgtext+0avgdata 0maxresident)k
host65: 0inputs+0outputs (0major+75539minor)pagefaults 0swaps

Results for 3 Server Ensemble

A 3 Server Ensemble (3 distinct hosts) was used for this test ("host60", "host62" & "host63").

The client was run as follows unless otherwise noted:

zk-latencies.py --cluster "host60:2181,host61:2181,host63:2181" --root_znode=/{{hostname}} --znode_size=100 --znode_count=10000 --timeout=5000 --watch_multiple=5

This means each client:

  • has a 5 second timeout
  • creates 10k znodes of size 100 bytes
  • sets 5 watches on each of the znodes it creates, for 50k znodes by each client

As a result, each client will send ~110,000 requests to the server during each test.

1 Core - 1 Client

110k operations against the server with elapsed time of 22 seconds

host40: created   10000 permanent znodes  in   3705 ms (0.370585 ms/op 2698.435645/sec)
host40: set       10000           znodes  in   1581 ms (0.158110 ms/op 6324.698298/sec)
host40: get       10000           znodes  in   1719 ms (0.171902 ms/op 5817.264317/sec)
host40: deleted   10000 permanent znodes  in   1417 ms (0.141717 ms/op 7056.330469/sec)
host40: created   10000 ephemeral znodes  in   1602 ms (0.160281 ms/op 6239.035623/sec)
host40: watched   50000           znodes  in   9156 ms (0.183121 ms/op 5460.880962/sec)
host40: deleted   10000 ephemeral znodes  in   2210 ms (0.221096 ms/op 4522.914060/sec)
host40: notif     50000           watches in      0 ms (included in prior)
host40: Latency test complete
host40: 16.77user 7.79system 0:22.02elapsed 111%CPU (0avgtext+0avgdata 0maxresident)k
host40: 0inputs+0outputs (2major+71106minor)pagefaults 0swaps

1 Core - 10 Client

1.1 million operations against the server with elapsed time of 52 seconds

host43: created   10000 permanent znodes  in  15068 ms (1.506821 ms/op 663.648755/sec)
host43: set       10000           znodes  in   6400 ms (0.640035 ms/op 1562.414824/sec)
host43: get       10000           znodes  in   6858 ms (0.685819 ms/op 1458.111314/sec)
host43: deleted   10000 permanent znodes  in   5444 ms (0.544488 ms/op 1836.588407/sec)
host43: created   10000 ephemeral znodes  in   5467 ms (0.546737 ms/op 1829.034317/sec)
host43: watched   50000           znodes  in   9212 ms (0.184244 ms/op 5427.576914/sec)
host43: deleted   10000 ephemeral znodes  in   2204 ms (0.220440 ms/op 4536.389994/sec)
host43: notif     50000           watches in      0 ms (included in prior)
host43: Latency test complete
host43: 16.49user 7.50system 0:52.00elapsed 46%CPU (0avgtext+0avgdata 0maxresident)k
host43: 0inputs+0outputs (2major+73374minor)pagefaults 0swaps

1 Core - 20 Client

2.2 million operations against the server with elapsed time of 93 seconds

host47: created   10000 permanent znodes  in  23807 ms (2.380728 ms/op 420.039583/sec)
host47: set       10000           znodes  in  18242 ms (1.824297 ms/op 548.156301/sec)
host47: get       10000           znodes  in   8831 ms (0.883173 ms/op 1132.281006/sec)
host47: deleted   10000 permanent znodes  in  13448 ms (1.344885 ms/op 743.558057/sec)
host47: created   10000 ephemeral znodes  in  11384 ms (1.138425 ms/op 878.406797/sec)
host47: watched   50000           znodes  in  10753 ms (0.215074 ms/op 4649.572468/sec)
host47: deleted   10000 ephemeral znodes  in   5636 ms (0.563623 ms/op 1774.235993/sec)
host47: notif     50000           watches in      0 ms (included in prior)
host47: Latency test complete
host47: 16.20user 6.97system 1:33.16elapsed 24%CPU (0avgtext+0avgdata 0maxresident)k
host47: 0inputs+0outputs (0major+71849minor)pagefaults 0swaps

2 Core - 1 Client

110k operations against the server with elapsed time of 20 seconds

host40: created   10000 permanent znodes  in   1910 ms (0.191017 ms/op 5235.149647/sec)
host40: set       10000           znodes  in   1604 ms (0.160475 ms/op 6231.495923/sec)
host40: get       10000           znodes  in   1708 ms (0.170852 ms/op 5853.005323/sec)
host40: deleted   10000 permanent znodes  in   1512 ms (0.151266 ms/op 6610.874968/sec)
host40: created   10000 ephemeral znodes  in   1597 ms (0.159736 ms/op 6260.325280/sec)
host40: watched   50000           znodes  in   9361 ms (0.187236 ms/op 5340.855592/sec)
host40: deleted   10000 ephemeral znodes  in   2278 ms (0.227898 ms/op 4387.931573/sec)
host40: notif     50000           watches in      0 ms (included in prior)
host40: Latency test complete
host40: 17.01user 7.95system 0:20.52elapsed 121%CPU (0avgtext+0avgdata 0maxresident)k
host40: 0inputs+0outputs (0major+74021minor)pagefaults 0swaps

2 Core - 10 Client

1.1 million operations against the server with elapsed time of 35 seconds

host49: created   10000 permanent znodes  in   7788 ms (0.778842 ms/op 1283.957982/sec)
host49: set       10000           znodes  in   4339 ms (0.433999 ms/op 2304.153334/sec)
host49: get       10000           znodes  in   2371 ms (0.237105 ms/op 4217.544356/sec)
host49: deleted   10000 permanent znodes  in   4862 ms (0.486300 ms/op 2056.345873/sec)
host49: created   10000 ephemeral znodes  in   3569 ms (0.356916 ms/op 2801.775772/sec)
host49: watched   50000           znodes  in   9279 ms (0.185594 ms/op 5388.116824/sec)
host49: deleted   10000 ephemeral znodes  in   2004 ms (0.200473 ms/op 4988.202935/sec)
host49: notif     50000           watches in      0 ms (included in prior)
host49: Latency test complete
host49: 16.35user 7.39system 0:35.09elapsed 67%CPU (0avgtext+0avgdata 0maxresident)k
host49: 0inputs+0outputs (3major+72326minor)pagefaults 0swaps

2 Core - 20 Client

2.2 million operations against the server with elapsed time of 65 seconds

host53: created   10000 permanent znodes  in  14369 ms (1.436911 ms/op 695.937229/sec)
host53: set       10000           znodes  in   9219 ms (0.921955 ms/op 1084.652098/sec)
host53: get       10000           znodes  in   7673 ms (0.767396 ms/op 1303.108503/sec)
host53: deleted   10000 permanent znodes  in   7229 ms (0.722930 ms/op 1383.259375/sec)
host53: created   10000 ephemeral znodes  in   8599 ms (0.859978 ms/op 1162.819917/sec)
host53: watched   50000           znodes  in  12619 ms (0.252387 ms/op 3962.162335/sec)
host53: deleted   10000 ephemeral znodes  in   4642 ms (0.464251 ms/op 2154.009006/sec)
host53: notif     50000           watches in      0 ms (included in prior)
host53: Latency test complete
host53: 14.44user 5.08system 1:05.21elapsed 29%CPU (0avgtext+0avgdata 0maxresident)k
host53: 0inputs+0outputs (2major+69809minor)pagefaults 0swaps

4 Core - 1 Client

110k operations against the server with elapsed time of 20 seconds

host40: created   10000 permanent znodes  in   1589 ms (0.158944 ms/op 6291.516084/sec)
host40: set       10000           znodes  in   1572 ms (0.157295 ms/op 6357.472953/sec)
host40: get       10000           znodes  in   1713 ms (0.171362 ms/op 5835.606165/sec)
host40: deleted   10000 permanent znodes  in   1447 ms (0.144768 ms/op 6907.623247/sec)
host40: created   10000 ephemeral znodes  in   1609 ms (0.160907 ms/op 6214.762414/sec)
host40: watched   50000           znodes  in   9535 ms (0.190719 ms/op 5243.304577/sec)
host40: deleted   10000 ephemeral znodes  in   2254 ms (0.225470 ms/op 4435.170105/sec)
host40: notif     50000           watches in      0 ms (included in prior)
host40: Latency test complete
host40: 16.95user 8.53system 0:20.27elapsed 125%CPU (0avgtext+0avgdata 0maxresident)k
host40: 0inputs+0outputs (0major+73094minor)pagefaults 0swaps

4 Core - 10 Client

1.1 million operations against the server with elapsed time of 30 seconds

host49: created   10000 permanent znodes  in   5470 ms (0.547053 ms/op 1827.976119/sec)
host49: set       10000           znodes  in   4285 ms (0.428503 ms/op 2333.703802/sec)
host49: get       10000           znodes  in   1695 ms (0.169513 ms/op 5899.249693/sec)
host49: deleted   10000 permanent znodes  in   3801 ms (0.380188 ms/op 2630.275589/sec)
host49: created   10000 ephemeral znodes  in   2953 ms (0.295355 ms/op 3385.752729/sec)
host49: watched   50000           znodes  in   9185 ms (0.183702 ms/op 5443.598826/sec)
host49: deleted   10000 ephemeral znodes  in   2229 ms (0.222966 ms/op 4484.998745/sec)
host49: notif     50000           watches in      0 ms (included in prior)
host49: Latency test complete
host49: 16.55user 7.69system 0:30.31elapsed 79%CPU (0avgtext+0avgdata 0maxresident)k
host49: 0inputs+0outputs (0major+69011minor)pagefaults 0swaps

4 Core - 20 Client

2.2 million operations against the server with elapsed time of 53 seconds

host47: created   10000 permanent znodes  in  10967 ms (1.096777 ms/op 911.762226/sec)
host47: set       10000           znodes  in   7628 ms (0.762822 ms/op 1310.922660/sec)
host47: get       10000           znodes  in   4896 ms (0.489673 ms/op 2042.178396/sec)
host47: deleted   10000 permanent znodes  in   9035 ms (0.903586 ms/op 1106.701776/sec)
host47: created   10000 ephemeral znodes  in   4735 ms (0.473520 ms/op 2111.844099/sec)
host47: watched   50000           znodes  in  11316 ms (0.226338 ms/op 4418.168721/sec)
host47: deleted   10000 ephemeral znodes  in   4044 ms (0.404485 ms/op 2472.282121/sec)
host47: notif     50000           watches in      0 ms (included in prior)
host47: Latency test complete
host47: 16.13user 6.67system 0:53.31elapsed 42%CPU (0avgtext+0avgdata 0maxresident)k
host47: 0inputs+0outputs (0major+69545minor)pagefaults 0swaps
  • No labels