Troubleshooting CPU
Description
When an application becomes CPU-bound, it no longer processes interrupts efficiently. This will eventually lead Geode's failure detector (GMSHealthMonitor) to force-disconnect the offending member (ForcedDisconnectException.)
Determination
One way to determine whether a machine or process is CPU-bound is to use an operating system command such as vmstat
or top
while the application is running.
...
Code Block |
---|
vmstat 2 procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy id wa 2 0 95592 61288 975548 329124 1 2 23 24 178 623 3 1 93 3 0 0 95592 61288 975548 329124 0 0 0 0 177 577 1 0 98 0 0 0 95592 61288 975552 329124 0 0 0 16 172 583 1 0 98 0 0 0 95592 61284 975552 329124 0 0 0 0 173 585 0 1 99 0 0 0 95592 61280 975556 329124 0 0 0 6 173 569 0 0 99 0 |
Per this source https://access.redhat.com/solutions/1160343 the "r" column is:
The number of processes waiting for run time
Also note:
The first line of the report will contain the average values since the last time the computer was rebooted. All other lines in the report will represent their respective current values.
The "r" column is a measure of "queue depth": the depth of the (conceptual) queue where ready-to-run threads wait for a CPU. Queueing indicates CPU saturation.
top
The top
output shows, among other things, CPU usage percentage. The output below shows that the CPUs are mostly in use (idle=3.0%) and that Java processes using most of that CPU.
Code Block |
---|
top 12:49:24 up 113 days, 23:36, 35 users, load average: 10.40, 5.20, 2.30 615 processes: 587 sleeping, 27 running, 1 zombie, 0 stopped CPU states: cpu user nice system irq softirq iowait idle total 61.7% 0.0% 31.4% 0.5% 2.5% 0.4% 3.0% PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 22523 user1 15 0 1102M 1.1G 18068 R 3.6 14.1 0:24 1 java 22778 user1 15 0 1102M 1.1G 18068 R 2.1 14.1 0:02 1 java 22682 user1 15 0 1102M 1.1G 18068 R 1.4 14.1 0:07 1 java 22698 user1 15 0 1102M 1.1G 18068 R 1.4 14.1 0:10 0 java 19286 user1 15 0 1100M 1.1G 18080 R 0.5 14.1 0:25 0 java |
...
vsd
Another way to determine whether a machine or process is CPU-bound is to use vsd
to display active CPU and process CPU time values contained in a given Geode statistics archive.
...