11.22.2012

If you are
as much enthusiasm as I am on Java performance, heap dump analysis should not
be a mystery to you. If it is then the good news is that you have an
opportunity to increase your Java troubleshooting skills and JVM knowledge.

The JVM
has now evolve to a point that it is much easier today to generate and analyze
a JVM heap dump vs. the old JDK 1.0 – JDK 1.4 days.

That being said, JVM heap dump
analysis should not be seen as a replacement for profiling & JVM analysis tools
such as JProfiler or Plumbr but complementary. It is particularly useful when
troubleshooting Java heap memory leaks and java.lang.OutOfMemoryError problems.

This post
will provide you with an overview of a JVM heap dump and what to expect out of
it. It will also provide recommendations on how and when you should spend time
analyzing a heap dump. Future articles will include tutorials on the analysis
process itself.

Java Heap Dump overview

A JVM heap dump is basically a “snapshot” of the Java heap memory at a
given time. It is quite different than a JVM thread dump which is a snapshot of
the threads.

Such snapshot contains low level detail about the java objects and
classes allocated on the Java heap such as:

Please note that it is usually recommended to generate a heap dump
following a full GC in order to eliminate unnecessary “noise” from non-referenced
objects.

Analysis reserved for the Elite?

One common misperception I have noticed over the last 10 years working
with production support teams is the impression that deeper analysis tasks such
as profiling, heap dump or thread dump analysis are reserved for the “elite” or
the product vendor (Oracle, IBM…).

I could not disagree more.

As a Java developer, you write code potentially running in a highly
concurrent thread environment, managing hundreds and hundreds of objects on the
JVM. You do have to worry not only about concurrency issues but also on garbage
collection and the memory footprint of your application(s). You are in the best
position to perform this analysis since you are the expert of the application.

Find below typical questions you should be able to answer:

How much concurrent threads are needed to run my
application concurrently as per load forecast? How much memory each active
thread is consuming before they complete their tasks?

What is the dynamic memory footprint of my
application under load? (sessions footprint etc.)

Did you profile your application for any memory
leak?

Load testing, profiling your application and analyzing Java heap dumps (ex: captured
during a load test or production problem) will allow you to answer the above
questions. You will then be in position to achieve the following goals:

Reduce risk of performance problems post
production implementation

Add value to your work and your client by
providing extra guidance & facts to the production and capacity management
team; allowing them to take proper IT improvement actions

Analyze the root cause of memory leak(s) or
footprint problem(s) affecting your client IT production environment

The last thing you want to reach is a skill “plateau”. If you are not
comfortable with this type of analysis then my recommendations are as per
below:

Ask a more senior member of your team to perform
the heap dump analysis and shadow his work and approach

Once you are more comfortable, volunteer yourself
to perform the same analysis (from a different problem case) and this time
request a more experienced member to shadow your analysis work

Eventually the student (you) will become the
mentor

When to use

Analyzing JVM heap dumps should not be done every time you are facing a
Java heap problem such as OutOfMemoryError. Since this can be a time consuming
analysis process, I recommend this analysis for the scenarios below:

I recommend
that you review the MAT summary page on how to acquire JVM heap
dump via various JVM & OS combinations.

Heap dump analysis tools

My primary
recommended tool for opening and analyzing a JVM heap dump is Eclipse Memory Analyzer (MAT). This is by far the best tool out there
with contributors such as SAP & IBM. The tool provides a rich interface and
advanced heap dump analysis capabilities, including a “leak suspect” report. MAT
also supports both HPROF & PHD heap dump formats.

I
recommend my earlier post for a quick tutorial on how to use MAT and analyze
your first JVM heap dump. I have also a few heap dump analysis case studies useful for your learning process.

Final words

I really
hope that you will enjoy JVM heap dump analysis as much as I do. Future articles
will provide you with generic tutorials on how to analyze a JVM heap dump and
where to start. Please feel free to provide your comments.

nice post. but the issue i face is in our prod system , a single java jvm normally has 40-60G heap on unix box (yes we do hold lot of data in memory). sometime we have to do heap analysis on it. Doing a heap dump on this kind of size of jvm is extremely slow, although the server is most powerful one. and if you want to load the dump into your local tools like MAT,VisualVM) for analysis is mission impossible. Do you know any tools that could do the heap analysis on a running system without dump whole heap onto disc and works on unix system as well?

That is correct, binary heap dump generation can be a real challenge for a 64-bit JVM of that size. Can you please let me know which JVM vendor you are using? HotSpot, Jrockit or IBM VM? 2 recommendations for you:

1) Explore just generating just a histogram for now and determine at least your top Java object consumers, you can use jmap for that purpose2) You may want to explore and do POC of JVM memory analysis tools such as Plumbr (see link in the article intro) which are active agents keeping track of your JVM memory vs. memory dump on the server

- Run the JDK jmap command under your JDK_HOME >> jmap -histo JAVA_PID * make sure you run it outside your production traffic as it will affect the performance *

- jmap can take several minutes to run depending of the size and speed of your hardware

- Analyze the results, you can also send the output to me via email (http://javaeesupportpatterns.blogspot.com/p/java-ee-it-consulting.html)so I can have a look and pinpoint to your your top Java object consumers or post it to the Java EE forum (link at the top of the page)

For IBM J9, my main recommendation is manually trigger a Heap Dump using kill -3 (after overriding the JVM settings via Xdump). Since the generation of a Heap Dump is intrusive, ensure you get maximum value with minimal risk.

Monitor your JVM utilization, following a major collection, you can trigger manually a Heap Dump. It all depends of the goal of what you are trying to achieve e.g. leak analysis, memory footprint analysis etc.