Analyze Slow and Blocked Processes(jstack)

 

Symptoms

The CRX/CQ process uses 100% of the CPU, the system doesn't respond, or the system is very slow.

Cause

Such problems can have many causes.

Analysis, Resolution

CRX 2.x

A simple CPU profiling tool is included in CRX 2.0.x. To start it, open:

http://localhost:7402/crx/diagnostic/prof.jsp

CRX 1.x

To help analyzing the problem, create a few full thread dumps. Those thread dumps can then be analyzed (for example using samurai.jar).

Creating Full Thread Dumps

Get the Process ID

To get the process id of your Java process, use

jps -l

 

If this doesn't work (path not set, JDK not installed, or older Java version), use

ps -el | grep java

 

Full Thread Dumps

To analyze a performance problem or a blocked process, create about 10 full thread dumps with about 1 second delay. If the problem may be related to clustering, please create at least 10 full thread dumps on each cluster node. If possible, the thread dumps should be created at roughly the same time (it doesn't need to be exact).

If installed, use jstack. This will print the thread dumps to system out:

jstack <pid>

 

This will append a full thread dump to a file:

jstack <pid> >> threadDumpNode1.txt

 

 

If this doesn't work, use kill -QUIT. This will print the thread dumps to the log file:

kill -QUIT <pid>

 

NOTE: If the steps above steps for obtaining thread dumps do not work in your environment then please see this article.

 

Check CPU Usage

To analyze the problem, it is important to know if CRX /CQ is running in a endless loop, or if it's merely sleeping. To find it out, type

top

 

This will get you the list of processes, sorted by CPU usage. If the top process is a 'java' process, and if the PID matches CRX/CQ, then the process is running full speed.

If you are unsure how to interpret the results, run the following statement and then include the filetop.txt in your problem report:

top -l5 -s5 > top.txt

Check Session Count

In many many cases the problem is the number of open sessions it too large. At some point, this slows down processing. To find out if this is the case, run

jps -l (to get the process id of the Java process)

jmap -histo <pid> | grep CRXSession (to get the number of open session)

If this is in fact the problem (the number is higher than a few hundred sessions) then this needs to be analyzed. Possibly a session pool is used (depending on the version of CRX / CQ there might be a hotfix for the given problem), or an internal (possibly application level) cache references sessions. To analyze where those sessions are opened, see the 'Analyze Unclosed Sessions' page.

Do Not Kill the Process

The CRX process should never be killed, also not when stopping takes too long. If you need to kill a process that is not responding, please create a full thread dump first and log a bug.

If you do kill the CRX process, the next time you start it up the Tar PM may create backup_.tar files.