...
The gfsh show metrics command can be used to show the active CPU (cpuUsage) of a member. An example is:
| Code Block |
|---|
show metrics --member=server1 --categories=member |
...
Member Metrics |
...
Category | Metric | Value |
...
-------- | --------------- | ----------------- member |
...
| upTime | 14 | cpuUsage | 84.13760375976562 | currentHeapSize | 2200 | maximumHeapSize | 81250 |
...
Action
Determining that there is a CPU issue is one thing. Finding the source of the issue is another. One thing that can be done is to dump the JVM threads using the operating system kill -3 command as shown below. These dumps will show you how many threads there are and what each thread is doing. Often, application issues can be found by examining these thread dumps.
| Code Block |
|---|
kill -3 22523 Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode): ... "ServerConnection on port 64124 Thread 1" prio=5 tid=0x00007fef8e02b800 nid=0xcc03 runnable [0x0000000133438000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at com.gemstone.gemfire.internal.cache.tier.sockets.Message.fetchHeader(Message.java:637) at com.gemstone.gemfire.internal.cache.tier.sockets.Message.readHeaderAndPayload(Message.java:661) at com.gemstone.gemfire.internal.cache.tier.sockets.Message.read(Message.java:604) at com.gemstone.gemfire.internal.cache.tier.sockets.Message.recv(Message.java:1104) - locked <0x0000000109e365a0> (a java.nio.HeapByteBuffer) at com.gemstone.gemfire.internal.cache.tier.sockets.Message.recv(Message.java:1118) at com.gemstone.gemfire.internal.cache.tier.sockets.BaseCommand.readRequest(BaseCommand.java:1003) at com.gemstone.gemfire.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:760) at com.gemstone.gemfire.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:942) at com.gemstone.gemfire.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1192) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at com.gemstone.gemfire.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:572) at java.lang.Thread.run(Thread.java:744) "Pooled Message Processor 1" daemon prio=5 tid=0x00007fef8d046800 nid=0xa603 waiting on condition [0x0000000131d23000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315) at com.gemstone.java.util.concurrent.SynchronousQueueNoSpin$TransferStack.awaitFulfill(SynchronousQueueNoSpin.java:449) at com.gemstone.java.util.concurrent.SynchronousQueueNoSpin$TransferStack.transfer(SynchronousQueueNoSpin.java:350) at com.gemstone.java.util.concurrent.SynchronousQueueNoSpin.take(SynchronousQueueNoSpin.java:884) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at com.gemstone.gemfire.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:726) at com.gemstone.gemfire.distributed.internal.DistributionManager$4$1.run(DistributionManager.java:1012) at java.lang.Thread.run(Thread.java:744) |
...
... |
...
|
If the thread dumps show mostly active Geode threads, one action would be to reduce the number of processing threads by one or more of the following methods:
...