JVMs use File Descriptors (FDs) to access files and sockets. Often, the default (1024) is not high enough to support a running Geode JVM.
A native memory issue will manifest itself in the Geode log file as a SocketException with the message Too many open files thrown either by a Geode thread or an application thread. An example of the exception is shown below.
One way to determine whether there is an FD issue is to use the operating system
lsof command to list open files including sockets of any running process including JVMs. These include:
- Java jar files
- Geode stats and log files
- Established and listening TCP sockets
- UDP sockets
The example below shows a partial list of the open files for the JVM with pid 26896.
The example below counts established TCP socket connections.
Another way to determine whether there is an FD issue is to use
vsd to display the open and maximum FD values contained in a given Geode statistics archive.
The chart below shows VMStats fdLimit and fdsOpen values. In this case, the application ran out of FDs.
The JVM will also maintain references to FDs for files and (mainly) sockets that have been closed until a GC cleans them up. This condition is referred to as a soft FD leak. It results in a saw-tooth pattern in the VMStats.
gfsh show metrics command can be used to show the FD limit (fileDescriptorLimit) and number of open FDs (totalFileDescriptorOpen) of a member. An example is:
There are several actions that can help prevent FD issues, including:
- Increase the open files limit. Check your operating system for specifics on how to do this.
- Change the Geode settings that control the life of connected sockets. For additional details, see Managing Open File Descriptors on GemFire Data Node Hosts.