Main Menu

Heap dump on Linux 64 bits

I am working for a customer, doing some performance review for their java application and tuning glassfish as well.

They use linux 64 bits (kernel 2.6.18 SMP), glassfish v2 u2 and JDK 5 u12. At some point I needed to generate a heap dump to better understand the object allocation and if possible some hints as to where to look for optimization for the application.

Now the big problem, I could not generate a heap dump, not with the current tools available at that time. I have tried

-XX:+HeapDumpOnCtrlBreak and kill -3

jmap -heap:format=b

gcore utility

All of them generated an sun.jvm.hotspot.debugger.UnmappedAddressException, except gcore who just killed the process. See the stacktrace

# /usr/local/jdk/jdk1.6.0_07/bin/jmap -J-d64 -F -dump:file=java_dump_10791.hprof 10791Attaching to process ID 10791, please wait...Debugger attached successfully.Server compiler detected.JVM version is 10.0-b23Dumping heap to java_dump_10791.hprof ...Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jmap.JMap.runTool(JMap.java:178) at sun.tools.jmap.JMap.main(JMap.java:110)Caused by: sun.jvm.hotspot.debugger.UnmappedAddressException at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208) at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63) at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:205) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:471) at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:442) at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readOopHandle(LinuxDebuggerLocal.java:431) at sun.jvm.hotspot.debugger.linux.LinuxAddress.getOopHandleAt(LinuxAddress.java:115) at sun.jvm.hotspot.oops.Oop.getKlassForOopHandle(Oop.java:222) at sun.jvm.hotspot.oops.ObjectHeap.newOop(ObjectHeap.java:348) at sun.jvm.hotspot.utilities.HashtableEntry.literal(HashtableEntry.java:53) at sun.jvm.hotspot.memory.SymbolTable.symbolsDo(SymbolTable.java:106) at sun.jvm.hotspot.utilities.HeapHprofBinWriter.writeSymbols(HeapHprofBinWriter.java:830) at sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:396) at sun.jvm.hotspot.tools.HeapDumper.run(HeapDumper.java:56) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.HeapDumper.main(HeapDumper.java:77)

UPDATE: The stacktrace displayed above, shows a dump request from a JDK 6u7 to a target VM 6 u7.UPDATE: Previously, I had issued jmap from JDK 5 u12 to a target VM 5 u12. The stacktrace is the same.UPDATE: The target VM is not using -Xrs option.UPDATE: The VM process owner is the same who issued the jmap command, root.

UPDATE: Alan Bateman clarified a bit about the -F option, "The -F options causes the tool to attach to the target VM in a
non-cooperative way and so may observe the heap in an inconsistent
state. In other words, there is no guarantee that you will get a good
heap dump when you using the -F option" read more on the comment section of my portuguese blog (the blog post is the same as posted here).

Comments

Hi, sorry for not answering (I didn't get any notifications of your response).
The problem persists, but it doesn't happen always. E.g. I've never experienced this problem when doing a dump from a JBoss with no application deployed. However after deploying our EAR, the dump fails.
The same problem has appeared to my colleague, who was trying to a dump on the same machine but with JDK5:
$ /opt/java/jdk5/bin/java -version
java version "1.5.0_19"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_19-b02)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_19-b02, mixed mode)
(of course the jmap and the JBoss were run from the very same JAVA_HOME)