...
zk: biggest issues we typically see are; gc causing vm to stall (client or server, we have no control over the jvm unfort, in hadoop i've heard they can see gc pauses of over 60 seconds), server disk io causing stalls (sequential consistency means that reads have to wait for writes to complete), network connectivity problems. Perhaps use of JVM tools such as gc monitoring and jvisualvm may help to track down. Over-provising a host will obviously slow down processing (running high cpu/disk processes on the same box(es) as zookeeper).
also see: https://issues.apache.org/jira/browse/ZOOKEEPER-545
zk: Mahadev mentioned that zk team commonly sees the types of issues hbase experiences at least on first deploy but that after tuning and research, all settles down
...
zk: this is the easiest to see http://hadoop.apache.org/zookeeper/docs/current/zookeeperOver.html#Performance however this is for optimum conditions (dedicated server class host with sufficient memory and dedicated spindle for the log) and a large number of clients. With smaller number of clients using synchronous operations you are probably limited more by network latency than anything.
...