Main Menu

Hot Threads

I'm setting up my new desktop development machine, and netbeans installation is atrociously slow, like several minutes just to display the splash screen. The task manager shows the process consuming 50% CPU (on a dual core). After stuffing around barking up several wrong trees I drag out a JMX based tool I wrote a while back to find hot threads in a running application.

I had previously encountered slow startup with Netbeans 5.5 (itself - not the installer) and based on A. Sundararajan's blog Using Mustang's Attach API I had written a tool to output the stack traces for the three busiest threads in a java process.

The program attaches to the local java process (specified by the PID on the command line), it grabs information about the processing time of all threads, twice 500ms apart, and uses that to find the three busiest threads. It then takes 10 stacktrace snapshots of those three threads at 10ms intervals, and looks for the common parts on those stack traces for each thread. If a thread is busy, normally most of the stack stays the same, and just the top part changes. The program then outputs to common parts of the stack traces. From there you can see which thread is running hot, and where it is.

So I enabled PID display in the Window's task Manager and ran the program. Here's what I saw...

C:\>java -jar C:\projects\experimental\HotThread\dist\HotThread.jar 3544java.lang.UnsatisfiedLinkError: no attach in java.library.pathException in thread "main" com.sun.tools.attach.AttachNotSupportedException: no at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:190) at hotthread.Main.main(Main.java:52)

That was due to java's installation running the JRE's copy of java.exe, but it doesn't have some of the attach API support files, so you need to run the JDK version of java.exe. Trying again...

Comments

Bruce, I am using your invaluable tool and it saved me many times already. Thank you so much!
Unfortunately I was chasing one elusive endless loop for a over a year now and your HotThread did not detect it. I seem to have hit a JVM bug - HotSpot's CompilerThread is looping. I was able to determine which thread was responsible using the following IBM article: http://publib.boulder.ibm.com/infocenter/javasdk/tools/index.jsp?topic=/...
Enlightened by IBM's article I used the fact that java thread on Linux is a process which has its own process id and shows up in top's process list. I noticed that top showed one Java process eating CPU and its pid was different from the one reported by jps, so I converted this pid it to hex and voila - it matched nid for CompilerThread reported by jstack. Btw, this HotSpot bug is sporadic and seems to occur only in Server VM.
To sum it up - it would have been much easier if HotThread detected this problem. Could it be that HotThread misses native threads?

For jconsole there is the JTop plugin:
http://weblogs.java.net/blog/mandychung/archive/2006/05/mustang_jconsol_...Thanks for the link, but JTop seems pretty crappy. First it has a really basic bug so only whole seconds are shown, then it shows the total accumulated CPU use (whole seconds with ".0000" appended) for each thread, whereas what people are interested in is how much it is currently using (but thats not possible so how much in the most recent sample period is an acceptable approximation), shown as a percentage of the total in such situations to make it easily understandable. - Bruce

VisualVM already has a threads window but I played with your code and did a very rough first draft plugin, based on porting your code to a VisualVM plugin which I then installed in VisualVM Beta. I will work on it some more to make it more graphical:
http://blogs.sun.com/geertjan/resource/hot-threads.png

Should this work on a linux system?
[joebob@xyz ~]$ /usr/java/jdk1.6.0_03/bin/java -jar /root/hotthread.jar 26954
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/tools/attach/VirtualMachine
at hotthread.Main.main(Main.java:52)
Is there something else I need to install?