About Roger Hughes

Investigating Deadlocks – Part 3

In my previous two blogs in this series, part 1 and part 2, I’ve demonstrated how to create a piece of bad code that deadlocks and then used this code to show three ways of taking a thread dump. In this blog I’m going to analyze the thread dump to figure out what when wrong.

The discussion below refers to both the Account and DeadlockDemo classes from part 1 of this series, which contains full code listings.

The first thing that that I need is a thread dump from the DeadlockDemo application, so as they used to say onBlue Peter “here’s one I prepared earlier”.

Scanning quickly through, you can see that this thread dump is divided into four parts. These are:

A complete list of all the applcation’s threads

A list of deadlocked threads

A small stack trace of deadlocked threads

The application’s heap summary

The Thread List

The thread list in point one above is a list of all the application’s threads and their current status. From this you can see that an application consists of a whole bunch of threads, which you can roughly divide in to two. Firstly there are the background threads. These are the ones that every application has, which get on with all the dirty jobs that we, as application programmers, don’t usually need to worry about. These have names such as: “DestroyJavaVM“, Low Memory Detector, Finalizer, Exception Catcher Thread and Concurrent Mark-Sweep GC Thread. Secondly, there are the threads that you or I may create as part of our code. These usually have names that consist of the word Thread followed by a number. For example: Thread-3, Thread-6 and Thread-20.

java.lang.Thread.State: BLOCKED (on object monitor) <td >This is the status of the thread; in this case it’s BLOCKED. Also included is a stack trace outlining where the thread is blocked.

Note that a thread can also be marked as a daemon. For example: “RMI TCP Accept-0″ daemon prio=5 tid=7f97128fd800 nid=0x117837000 runnable [117836000] java.lang.Thread.State: RUNNABLE Daemon threads are background task threads such as the RMI TCP Accept-0 thread listed above. A daemon thread is a thread that does not prevent the JVM from exiting. The JVM will exit, or close down, when only daemon threads remain.

However, the thread list doesn’t really help in tracing the cause of a deadlock, so moving swiftly along…

The Deadlock Thread List

This section of the thread dump contains a list of all threads that are involved in the deadlock.

Found one Java-level deadlock:
=============================
"Thread-21":
waiting to lock monitor 7f97118bd560 (object 7f3366f58, a threads.deadlock.Account),
which is held by "Thread-20"
"Thread-20":
waiting to lock monitor 7f97118bc108 (object 7f3366e98, a threads.deadlock.Account),
which is held by "Thread-4"
"Thread-4":
waiting to lock monitor 7f9711834360 (object 7f3366e80, a threads.deadlock.Account),
which is held by "Thread-7"
"Thread-7":
waiting to lock monitor 7f97118b9708 (object 7f3366eb0, a threads.deadlock.Account),
which is held by "Thread-11"
"Thread-11":
waiting to lock monitor 7f97118bd560 (object 7f3366f58, a threads.deadlock.Account),
which is held by "Thread-20"

From the segment above, you can see that there are five threads all blocking on instances the threads.deadlock.Account class

Leaving aside the monitor ids and Account instances, you can see that “Thread-21″ is waiting for “Thread-20″, which is waiting for “Thread-4″, which in turn is waiting for “Thread-7″. “Thread-7″ is waiting for “Thread-11″, which is waiting for “Thread-20″: a deadlock loop as shown in the diagram below:

The Deadlock Stack Traces

The final piece of the puzzle is the list of deadlocked thread stack traces as shown below:

From the previous section, we know that Thread-20 is waiting, via a circuitous route, for Thread-11 and Thread-11 is waiting for Thread-20. This is our deadlock.

The next step is to tie this deadlock up to lines of code using the thread stack trace above and I’ve simplified this in the diagram below.

In the above diagram I’ve removed the 7f3366 prefix from the object ids for clarity; hence, object 7f3366f58 is now f58. From this diagram, you can see that object f58 is locked by Thread-20 on line 59 and is waiting for a lock on object e98 on line 86. Following the arrows down, you can see that Thread-7 is waiting for a lock on eb0 on line 86, which in turn is locked by Thread-11 on line 59. Thread-11 is waiting for a lock on f58 on line 86, which, looping back up, is locked on line 58 by Thread-20.

So, where are these lines of code? The following shows line 59:

…and this is line 86:

Everyone gets surprises sometimes and the stack trace above surprised me. I was expecting the locks to be on lines 85 and 86; however, they were on 59 and 86. Since line 59 doesn’t contain a synchronized keyword, I’m guessing that the compiler has done some optimisation on the transfer(…) method’s first synchronized keyword.

The conclusion that can be drawn from this is that the code, which randomly picks two Account objects from a list, is locking them in the wrong order on lines 59 and 86. So what’s the fix? More on that next time; however, there’s one final point to note, which is that the make up of a deadlock may not be the same every time you generate a thread dump on a program. After running the DeadlockDemo program again and using kill -3 PID to get hold of another thread dump, I obtained these results:

Newsletter

Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

Email address:

Join Us

With 1,043,221 monthly unique visitors and over 500 authors we are placed among the top Java related sites around. Constantly being on the lookout for partners; we encourage you to join us. So If you have a blog with unique and interesting content then you should check out our JCG partners program. You can also be a guest writer for Java Code Geeks and hone your writing skills!

Disclaimer

All trademarks and registered trademarks appearing on Examples Java Code Geeks are the property of their respective owners. Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries. Examples Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.