Oracle Blog

kto

HPROF, HAT, and jhat

My Own Little HPROF Nightmare

If you do a google search on
"not found for JNI local ref"
you will find many hits, this is unfortunately a bug in the new implementation
of the hprof agent
in JDK 5 (the Tiger release)
brought out by using the heap analysis tool
HAT or it's improved version
in JDK 6 (the upcoming Mustang release) called
jhat.

The HAT/jhat tool is one of the few tools that uses the binary hprof format
as a binary heap snapshot mechanism.
The failure usually looks something like:

java.io.IOException: Thread 200000 not found for JNI local ref
at hat.parser.HprofReader.getThreadObjectFromSequence(HprofReader.java:577)
at hat.parser.HprofReader.readHeapDump(HprofReader.java:408)
at hat.parser.HprofReader.read(HprofReader.java:221)
at hat.parser.Reader.readFile(Reader.java:90)
at hat.Main.main(Main.java:149)

And pretty much stops HAT and jhat dead in the water.

Unfortunately my attempts to fix this bug in Tiger Update 4
(see 6231936)
didn't work out as I'd planned, and hopefully a better fix should
be coming with Tiger Update 7
(see 6325354)
and the fix is also available in the latest Mustang downloads at
http://mustang.dev.java.net/.
The basic problem lies in the difference between a java.lang.Thread object existing,
the java thread actually being alive and running, and the timing of dumping out
objects for a given java thread that are on there way to being garbage collected.
My first attempted fix was to try and work around the timing, and the second fix
was more along the lines of assuring the heap dump didn't include missing references,
mapping any unknown thread references to the catch all 'system thread'.
I sure hope this is the last fix to this particular problem.
I'm also not kidding myself that it is the last problem with hprof, so if any of you
run into issues, please create a forum posting at
http://forum.java.sun.com/forum.jspa?forumID=47&start=0, eventually I'll see it.

I could give excuses for this foopaw but I'd rather tell you what has been done and what
will be done to prevent this kind of regression in the future.
The ability to get heap dump snapshots and view them is a critical need that we
recognize and plan on leveraging in the future.
[The hprof and HAT/jhat combination is a hard one to test automatically, and if anyone
out there reading this has any good ideas, please let me know.]

We have added HAT to the JDK as "jhat" in Mustang. This was done for many reasons,
but a primary one was to enable us to start building up a set of tests with
hprof and jhat to verify the accuracy and correctness of the hprof binary
heap dumps, and also make sure that hprof and jhat work well together.
In addition, by having the jhat tool in the JDK we could make it more
readily available and integrate it better with the rest of the JDK.
This Mustang jhat can be run on ANY hprof binary format files, even those
created by JDK 1.4.2, so even if you don't want to try Mustang JDK 6, you
might want to download it just for the sake of having this jhat tool handy.

We started a project to investigate for Dolphin (JDK 7) a new binary heap dump
snapshot file format, with a more formal specification
(see https://heap-snapshot.dev.java.net/).
The traditional hprof binary format is close to what we want, but it has never
been fully specified and we would like to improve upon it and create a more
standard format.

The hprof agent is not the only way to get a hprof binary format file for HAT/jhat.
The Mustang release includes a utility called 'jmap' that has an option to
dump a heap snapshot file in the hprof binary dump format, one that jhat can read
(HAT may be able to read it too, but the stacks for where objects were created
will be missing from the data, and sometimes HAT doesn't like that).
This jmap & jhat technique is a very fast way to get a look into the heap of
a running VM without the overhead of starting the VM with hprof.

Improve the error recovery of HAT/jhat.
The HAT project has been retired but
jhat is being improved in the Mustang release
(see http://blogs.sun.com/roller/page/sundararajan).
The basic HAT program is quite old, but it seems to answer many problems, we would
like to see it evolve and improve in the future.
By using it more and making it more available, we will make this more stable.