1.25.2013

Most Java
programmers are familiar with the Java thread deadlock concept. It essentially
involves 2 threads waiting forever for each other. This condition is often the
result of flat (synchronized) or ReentrantLock (read or write) lock-ordering
problems.

The good
news is that the HotSpot JVM is always able to detect this condition for you…or
is it?

A recent
thread deadlock problem affecting an Oracle Service Bus production environment
has forced us to revisit this classic problem and identify the existence of “hidden”
deadlock situations.

This
article will demonstrate and replicate via a simple Java program a very special
lock-ordering deadlock condition which is not detected by the latest HotSpot JVM
1.7. You will also find a video at the end of the article explaining you the
Java sample program and the troubleshooting approach used.

The crime scene

I usually
like to compare major Java concurrency problems to a crime scene where you play
the lead investigator role. In this context, the “crime” is an actual
production outage of your client IT environment. Your job is to:

The next
step of your investigation is to analyze the collected information and
establish a potential list of one or many “suspects” along with clear proofs.
Eventually, you want to narrow it down to a primary suspect or root cause. Obviously
the law “innocent until proven guilty” does not apply here, exactly the opposite.

Lack of
evidence can prevent you to achieve the above goal. What you will see next is
that the lack of deadlock detection by the Hotspot JVM does not necessary prove
that you are not dealing with this problem.

The suspect

In this
troubleshooting context, the “suspect” is defined as the application or
middleware code with the following problematic execution pattern.

Usage of FLAT lock followed by the usage of ReentrantLock
WRITE lock (execution path #1)

Usage of ReentrantLock READ lock followed by the
usage of FLAT lock (execution path #2)

Concurrent execution performed by 2 Java threads but
via a reversed execution order

The above lock-ordering
deadlock criteria’s can be visualized as per below:

Now let’s replicate
this problem via our sample Java program and look at the JVM thread dump output.

Sample Java program

This above
deadlock conditions was first identified from our Oracle OSB problem case. We
then re-created it via a simple Java program. You can download the entire
source code of our program here.

The
program is simply creating and firing 2 worker threads. Each of them execute a
different execution path and attempt to acquire locks on shared objects but in
different orders. We also created a deadlock detector thread for monitoring and
logging purposes.

For now,
find below the Java class implementing the 2 different execution paths.

As soon ad
the deadlock situation was triggered, a JVM thread dump was generated using
JVisualVM.

As you can
see from the Java thread dump sample. The JVM did not detect this deadlock condition
(e.g. no presence of Found one Java-level deadlock) but it is clear these 2
threads are in deadlock state.

Root cause: ReetrantLock READ lock behavior

The main
explanation we found at this point is associated with the usage of the
ReetrantLock READ lock. The read locks are normally not designed to have a
notion of ownership. Since there is not a record of which thread holds a read
lock, this appears to prevent the HotSpot JVM deadlock detector logic to detect
deadlock involving read locks.

Some improvements were implemented since then but we can see that the
JVM still cannot detect this special deadlock scenario.

Now if we replace the read lock (execution pattern #1) in our program by
a write lock, the JVM will finally detect the deadlock condition but why?

The lack of read lock per-thread tracking appears to prevent deadlock detection
for this scenario and significantly increase the troubleshooting complexity.

I suggest that you read Doug Lea’s comments on this whole issue since
concerns were raised back in 2005 regarding the possibility to add per-thread read-hold tracking due to some
potential lock overhead.

1.12.2013

The
following question is quite common and is related to OutOfMemoryError: unable to create new native thread problems during
the JVM thread creation process and the JVM thread capacity. This is also a typical
interview question I ask to new technical candidates (senior role). I recommend that you
attempt to provide your own response before looking at the answer.

The Java
thread creation process requires native memory to be available for the JVM
process. Expanding the Java heap space via the –Xmx argument will actually reduce
your Java thread capacity since this memory will be “stolen” from the native
memory space.

For a 32-bit JVM, the Java heap space is in a race with
the native heap, including the thread capacity

1.09.2013

This is my
first post for 2013. I want to thank all my readers for their great feedback
and comments from the various articles I posted in 2012.

It is
sometimes quite difficult to truly show and explain some troubleshooting
techniques via articles only. For 2013, on top of my regular posting, I will be
introducing tutorial and troubleshooting videos. These training videos will be
available for free from YouTube. I really hope that you will appreciate this
addition. Watch for a new Videos
section from the top navigation bar.

I’m really
looking forward for your suggestions and recommendations on what type of
tutorials and videos you would like to watch and learn. Here are some examples: