Permalink to this page: https://cwiki.apache.org/confluence/x/yColBg
Troubleshooting and Diagnostics techniques.
Table of Contents
Techniques & Reference
- How To: Capture a thread dump
- How To: Capture a heap dump
- How To: Examine a Stacktrace
- How To: Configure Tomcat for debugging
- FAQ: Developing
- FAQ: Memory
- Tomcat Memory Leak Protection
- Notes on using JMX clients
- jinfo - Prints JVM process info
- jstack - Prints thread stack traces
- jmap - Dumps heap and shows heap status
- jhat - Heap Analyzer Tool
- jcmd - Multitool intended to replace the above JDK tools
Profilers & Heap Analyzers
Notes on using JMX clients
When running a JMX client (JConsole, VisualVM) on the same machine as the target JVM process it is possible to connect without pre-configuring a JMX port, using the local connector stub. This method relies on being able to create a protected temporary file, accessible only to a user with administrator privileges. Java processes which are accessible via the local connector will automatically appear in the client.
NB(1) On Windows, this means that the temporary directory must be located on an NTFS formatted disk. See the following link for more details.
NB(2) On Windows, if Tomcat is started using a service wrapper, this will prevent JConsole & VisualVM from using the local JMX connector stub.
From Java 6 onward a process does not need to have the management agent enabled when it starts, as the Attach API permits the management agent to be activated on demand.
Common Troubleshooting Scenario
If you have already looked into Tomcat logs, there are no error messages, and you just want to find out what is going on, you may try the following
- Look into Tomcat access log (the log file generated by AccessLogValve).
- If your request is not listed there, then it has not been processed by Tomcat. You need to look elsewhere (e.g. at your firewall).
- You will see what IP address your client is using, and whether it is using an IPv4 (
127.0.0.1) or IPv6 address (
Modern operating systems can use IPv6 addresses for localhost / local network access, while external network is still using IPv4.
- Take a thread dump. This way you will find out what Tomcat is actually doing.
- If you are troubleshooting some process that takes noticeable time, take several (three) thread dumps with some interval between them. This way you will see if there are any changes, any progress.
- Try debugging.
- A good place for a breakpoint is
org.apache.catalina.connector.CoyoteAdapter.service()method. That is the entry point from Tomcat connectors and into the Servlet engine. At that place your request has already been received and its processing starts.
- A good place for a breakpoint is
Troubleshooting unexpected Response state problems
If you encounter problems that manifest themselves as accessing a request or response that is an inconsistent state.
The main suspect is your own web application keeping a reference to Request / Response objects outside of their life cycle.
The lifetime of the Response object is documented in the Servlet specification. Quoting from section "5.8 Lifetime of the Response Object" of Servlet 4.0 specification:
"Each response object is valid only within the scope of a servlet’s service method, or within the scope of a filter’s doFilter method, unless the associated request object has asynchronous processing enabled for the component. If asynchronous processing on the associated request is started, then the response object remains valid until complete method on AsyncContext is called."
In case of asynchronous processing, when an error occurs Tomcat notifies all registered
AsyncListeners and then calls
complete() automatically if none of the listeners have called it yet. (Reference: 61768)
Also see sections "18.104.22.168 Thread Safety" and "3.13 Lifetime of the Request Object" of the same specification.
To troubleshoot the issue:
- Set the following system property in Tomcat configuration:
When the above flag is set, Tomcat recycles facades to its internal objects when request processing completes. This makes it easier to spot illegal access when it happens, instead of waiting until side effects of such access become visible. This flag is also mentioned on the Security Considerations page. The flag is
truewhen Tomcat runs with enabled Java Security Manager.
You can also search the archives of the Tomcat users' mailing lists for previous discussions mentioning the
- Read about Java ImageIO issue.
Accessing response objects after their lifetime can lead to security issues in your application, such as sending responses to wrong clients, mixing up responses. If you can reproduce the issue and the above diagnostic does not show your own bug, but a bug in Apache Tomcat, if the problem manifests as a security issue, see how to report it.
Troubleshooting "Too many open file descriptors"
The code that opens the descriptors can be identified using a tool such as http://file-leak-detector.kohsuke.org/