4.25.2012

This article will provide you with a tutorial on how you can quickly pinpoint the Java thread contributors to a high CPU problem on the Windows OS. Windows, like other OS such as Linux, Solaris & AIX allow you to monitor the CPU utilization at the process level but also for individual Thread executing a task within a process.Tutorials are also available from some other OS as per below:

The simple program below is simply looping and creating new String objects. It will allow us to perform this CPU per Thread analysis. I recommend that you import it in an IDE of your choice e.g. Eclipse and run it from there. You should observe an increase of CPU on your Windows machine as soon as you execute it.

The Process Explorer tool visually shows the CPU usage dynamically. It is good for live analysis. If you need historical data on CPU per Thread then you can also use Windows perfmon with % Processor Time & Thread Id data counters. You can download Process Explorer from the link below:http://technet.microsoft.com/en-us/sysinternals/bb896653

In our example, you can see that the Eclipse javaw.exe process is now using ~25% of total CPU utilization following the execution of our sample program.

Step #2 – Launch Process Explorer Threads view

The next step is to display the Threads view of the javaw.exe process. Simply right click on the javaw.exe process and select Properties. The Threads view will be opened as per below snapshot:

-The first column is the Thread Id (decimal format)

-The second column is the CPU utilization % used by each Thread

-The third column is also another counter indicating if Thread is running on the CPU

In our example, we can see our primary culprit is Thread Id #5996 using ~ 25% of CPU.

Step #3 – Generate a JVM Thread Dump

At this point, Process Explorer will no longer be useful. The goal was to pinpoint one or multiple Java Threads consuming most of the Java process CPU utilization which is what we achieved. In order to go the next level in your analysis you will need to capture a JVM Thread Dump. This will allow you to correlate the Thread Id with the Thread Stack Trace so you can pinpoint that type of processing is consuming such high CPU.

JVM Thread Dump generation can be done in a few manners. If you are using JRockit VM you can simply use the jrcmd tool as per below example:

Once you have the Thread Dump data, simply search for the Thread Id and locate the Thread Stack Trace that you are interested in.

For our example, the Thread “Main Thread” which was fired from Eclipse got exposed as the primary culprit which is exactly what we wanted to demonstrate.

At this point you should have everything that you need to move forward with the root cause analysis. You will need to review each Thread Stack Trace and determine what type of problem you are dealing with. That final step is typically where you will spend most of your time and problem can be simple such as infinite looping or complex such as garbage collection related problems.

In our example, the Thread Dump did reveal the high CPU originates from our sample Java program around line 31. As expected, it did reveal the looping condition that we engineered on purpose for this tutorial.

I hope this tutorial has helped you understand how you can analyze and help pinpoint root cause of Java CPU problems on Windows OS. Please stay tuned for more updates, the next article will provide you with a Java CPU troubleshooting guide including how to tackle that last analysis step along with common problem patterns.

4.09.2012

This article will provide you an overview and tutorial on how you can perform an initial analysis and problem isolation of a JRockit Java Heap problem using the jrcmd tool. A deeper analysis and tutorial using JRockit Mission Control and Heap Dump analysis (JRockit version R28+ only) will be covered in future articles.

For a quick overview of the JRockit Java Heap Space, please consult the article below:

For this article, we created a simple Java program leaking internally. We will use this program to demonstrate how you can leverage jrcmd to perform your initial analysis.

Sample Java memory leak program

This simple Java program is simply adding String data to a static HashMap and slowly leaking to the point of the JVM running out of Java Heap memory. This program will allow you to visualize a slowly growing Java Heap leak via JRockit jrcmd. Please note that a Java Heap size of 128 MB (-Xms128m –Xmx128m) was used for this example.

// Slowdown the Java program so we can monitor the leak before the OutOfMemoryError condition

Thread.sleep(1);

}

} catch (Throwable any) {

if (any instanceof java.lang.OutOfMemoryError) {

System.out.println("OutOfMemoryError triggered! "

+ any.getMessage() + " [" + any + "]");

} else {

System.out.println("Unexpected Exception! " + any.getMessage()

+ " [" + any + "]");

}

}

System.out.println("JavaHeapLeakSimulator done!");

}

}

JRCMD - initial execution

JRCMD can be executed from the local server hosting the JVM that you want to monitor or remotely via JRockit Mission Control. The executable is located within the JRockit JDK that you are using:

<JRockit_JDK_HOME>/bin/jrcmd

The default jrcmd execution will return the list of active JRockit Java process Id that you can monitor:

C:\Apps\Weblogic1035\jrockit_160_24_D1.1.2-4\bin>jrcmd

5360 org.ph.javaee.tool.oom.JavaHeapLeakSimulator

5952

6852 jrockit.tools.jrcmd.JrCmd

JRCMD - Java Heap monitoring

The next step is to start monitoring the Java Heap memory usage and histogram. A Java Heap histogram is a snapshot of the biggest pools of Java Class instances. This allow you to pinpoint the leaking data type. Ple

You can either chose between print_object_summary (quick summary) or heap_diagnostics (complete breakdown).

- The first column correponds to the Class object type contribution to the Java Heap footprint in %- The second column correponds to the Class object type memory footprint in K- The third column correponds to the # of Class instances of a particular type- The fourth column correponds to the delta - / + memory footprint of a particular type

As you can see from the above snapshot, the biggest data type is [C (char in our case) & java.lang.String. In order to see which data types are leaking, you will need to generate several snapshots. The frequency will depend of the leaking rate. In our example, find below another snapshot taken after 5 minutes:

# After 5 minutes

--------- Detailed Heap Statistics: ---------

93.9% 26169k 28746+12032k [C

2.4% 674k 28768+295k java/lang/String

2.3% 637k 27219+295k java/util/HashMap$Entry

0.9% 259k 21 +128k [Ljava/util/HashMap$Entry;

0.2% 50k 462 +0k java/lang/Class

0.1% 24k 7 +0k [B

# After 5 more minutes

--------- Detailed Heap Statistics: ---------

94.5% 46978k 50534+20809k [C

2.4% 1184k 50556+510k java/lang/String

2.3% 1148k 49007+510k java/util/HashMap$Entry

0.5% 259k 21 +0k [Ljava/util/HashMap$Entry;

0.1% 50k 462 +0k java/lang/Class

The third & fourth column are showing a constant increase. As you can see, the leaking data in our case are [C, java.lang.String and java.util.HashMap$Entry which all increased from ~4 MB to 28 MB, 50 MB and growing…

It is easy to pinpoint the leaking data type(s) with this approach but what about the source (root cause) of the leaking data type(s)? This is where jrcmd is no longer useful. Deeper memory leak analysis will require you to use either JRockit Mission Control or Heap Dump analysis (JRockit R28+ only).

A final point, before you can conclude on a true Java Heap leak, please ensure that jrcmd snapshots are taken after at least one Full GC in between the captures (what you are interested in is OldGen leak e.g. Java objects surviving major GC collections).

JRCMD Thread Dump generation

Thread Dump analysis is crucial for stuck Thread related problems but can also be useful to troubleshoot certain types of Java Heap problem. For example, it can pinpoint the source of a sudden Java Heap increase by exposing the culprit Thread(s) allocating a large amount of memory on the Java Heap in a short amount of time. Thread Dump can be generated using jrcmd print_threads option.

** Thread Dump captured from our sample Java program after removing the Thread.sleep() and increasing the Java Heap capacity **