Description
Java application VMs allocate objects on the heap. The heap includes both live and dead objects. Quantity of heap space is configured using the -Xms and -Xmx VM arguments. When the heap becomes close to full, GCs occur which can cause application pauses.
Determination
Running out of heap space is a heap memory issue that will manifest itself in the Geode log file as an OutOfMemoryError with the message Java heap space thrown either by a Geode thread or an application thread. The error must contain the Java heap space message and not the unable to create new native thread message (see Troubleshooting Native Memory for details on that issue). An example is shown below.
[severe 2009/02/23 19:38:48.293 UTC <Gateway Event Processor> tid=0xbd] Uncaught exception in thread <Gateway Processor> java.lang.OutOfMemoryError: Java heap space
vsd
Another way to determine whether there is a heap memory issue is to use vsd
to display free, used and maximum heap memory values contained in a given Geode statistics archive.
vmHeapMemoryStats
The chart below shows VMMemoryUsageStats-vmHeapMemoryStats maxMemory and usedMemory values. The VMMemoryUsageStats values show the overall heap memory usage.
CMS Old Gen-Heap memory
The chart below shows VMMemoryPoolStats-CMSOldGen-Heapmemory currentMaxMemory and currentUsedMemory values. The VMMemoryPoolStats values show the heap memory usage for specific memory pools (i.e. CMS).
VMStats
The chart below shows the VMStats freeMemory values. The VMStats freeMemory values show the JVM's free memory.
gfsh
The gfsh
show metrics command can be used to show the current (currentHeapSize) and maximum (maximumHeapSize) heap sizes of a member. An example is:
show metrics --member=server1 --categories=member Member Metrics Category | Metric | Value -------- | --------------- | ----------------- member | upTime | 1638 | cpuUsage | 89.60260009765625 | currentHeapSize | 6800 | maximumHeapSize | 81250
Action
There are several ways to eliminate heap issues, including:
- Increase the maximum amount of heap available to the VM using the -Xmx VM argument
- Configure one or more regions to overflow so that their values are not stored on the heap
- Scale the distributed system horizontally by adding one or more additional members