The available native memory on a machine is the difference between the machine's physical RAM and the memory used by the processes running on it. It is actually even less than that since the operating system also uses some of this memory.
JVMs allocate thread stacks in native memory not the JVM heap. An application can exhaust the native heap with thread allocations and still have plenty of JVM heap.
Native memory issues are generally caused by one of two things, namely
- allocating too many processes/threads for the available native memory
- configuring too low of a maximum for user processes
A native memory issue will manifest itself in the Geode log file as an OutOfMemoryError with the message 'unable to create new native thread' thrown either by a Geode thread or an application thread. The error must contain the 'unable to create new native thread' message and not the 'Java heap space' message (see Troubleshooting/Monitoring Heap Issues for details on that issue). An example of the error is shown below.
One way to determine whether there is a native memory issue is to use an operating system command such as
top to see the available free memory.
The free command shows the amount of free and used memory on the machine. The
free output below shows ~48GB total memory with ~11GB used and ~37GB free.
topcommand shows, among other things, the amount of free and used memory on the machine as well individual processes. The
topoutput below is a different view of the
freeoutput. It below shows the same 48GB total memory with ~11GB used and ~37GB free. It also shows the JVM using most of that memory.
If the free memory looks ok, then the issue might be caused by the configured maximum user processes being set too low. Use an operating system command like
ulimit to see the maximum user processes.
ulimit command shows the resource limits allowed to a user (like files and processes). The
ulimit output below shows the soft limits.
The max user processes value is the one of interest for native memory issues. In this case, the soft limit of 501408 is fine.
In addition, you can see the limits for a specific running process in linux by dumping the limits file for that process. The limits for the process with pid 7360 are shown below.
The Max processes value is the one of interest for native memory issues. In this case, soft limit of 1024 is too low.
Another way to check whether there is a native memory issue is to use vsd to display the free memory and number of threads contained in a given Geode statistics archive. TheVMStats threads statistic shows the number of threads in the JVM. The LinuxSystemStats freeMemory shows the available free memory in the OS.
The chart below shows potentially unhealthy VMStats threads values.
The chart below shows unhealthy LinuxSystemStats freeMemory values. It also shows that the JVM heap is the source of the memory usage.
gfsh show metrics command can be used to show the number of threads (jvmThreads) of a member. An example is:
There are several actions that can help prevent native memory issues.
If there is not enough available RAM:
- Reducing the JVM's thread stack size. The -Xss JVM argument is used to determine the thread stack size. A thread stack size like -Xss256m or -Xss384m should be sufficient.
- Reduce the number of threads. See Troubleshooting CPU for additional details.
- Reduce the max heap size of the JVM using -Xmx. This will provide a greater difference between the RAM and the heap and thus more native memory.
- Add RAM to the machine
If the maximum for user processes is too low:
- Increase the maximum number of user processes. Check the operating system for specifics on how to do this.