Skip to end of metadata
Go to start of metadata

Description

JVMs use File Descriptors (FDs) to access files and sockets. Often, the default (1024) is not high enough to support a running Geode JVM.

Determination

A native memory issue will manifest itself in the Geode log file as a SocketException with the message Too many open files thrown either by a Geode thread or an application thread. An example of the exception is shown below.

 

lsof

One way to determine whether there is an FD issue is to use the operating system lsof command to list open files including sockets of any running process including JVMs. These include:

  • Java jar files
  • Geode stats and log files
  • Established and listening TCP sockets
  • UDP sockets

The example below shows a partial list of the open files for the JVM with pid 26896.

The example below counts established TCP socket connections.

vsd

Another way to determine whether there is an FD issue is to use vsd to display the open and maximum FD values contained in a given Geode statistics archive.

VMStats

The chart below shows VMStats fdLimit and fdsOpen values. In this case, the application ran out of FDs. 

VMStats

The JVM will also maintain references to FDs for files and (mainly) sockets that have been closed until a GC cleans them up. This condition is referred to as a soft FD leak. It results in a saw-tooth pattern in the VMStats.

VMStats

gfsh

The gfsh show metrics command can be used to show the FD limit (fileDescriptorLimit) and number of open FDs (totalFileDescriptorOpen) of a member. An example is:

 

Action

There are several actions that can help prevent FD issues, including:

  • No labels