8.29.2012

As you may have seen from my past performance related articles and HashMap case studies, Java thread safety problems can bring down your Java EE application and the Java EE container fairly easily. One of most common problems I have observed when troubleshooting Java EE performance problems is infinite looping triggered from the non-thread safe HashMap get() and put() operations. This problem is known since several years but recent production problems have forced me to revisit this issue one more time.

This article will revisit this classic thread safety problem and demonstrate, using a simple Java program, the risk associated with a wrong usage of the plain old java.util.HashMap data structure involved in a concurrent threads context.

This proof of concept exercise will attempt to achieve the following 3 goals:

The first goal is to compare the performance level of our program when using different thread safe Map implementations:

Plain old Hashtable (since JDK 1.0)

Fully synchronized HashMap (via Collections.synchronizedMap())

ConcurrentHashMap (since JDK 1.5)

Find below the graphical results of the execution of the Java program for each iteration along with a sample of the program console output.

# Output when using ConcurrentHashMap

Infinite Looping HashMap Simulator

Author: Pierre-Hugues Charbonneau

http://javaeesupportpatterns.blogspot.com

All threads completed in 0.984 seconds

All threads completed in 0.908 seconds

All threads completed in 0.706 seconds

All threads completed in 1.068 seconds

All threads completed in 0.621 seconds

All threads completed in 0.594 seconds

All threads completed in 0.569 seconds

All threads completed in 0.599 seconds

………………

As you can see, the ConcurrentHashMap is the clear winner here, taking in average only half a second (after an initial ramp-up) for all 3 worker threads to concurrently read and insert data within a 500K looping statement against the assigned shared Map. Please note that no problem was found with the program execution e.g. no hang situation.

The performance boost is definitely due to the improved ConcurrentHashMap performance such as the non-blocking get() operation.

The 2 other Map implementations performance level was fairly similar with a small advantage for the synchronized HashMap.

HashMap infinite looping problem replication

The next objective is to replicate the HashMap infinite looping problem observed so often from Java EE production environments. In order to do that, you simply need to assign the non-thread safe HashMap implementation as per code snippet below:

/*** Assign map at your convenience ****/

assignedMapForTest = nonThreadSafeMap;

Running the program as is using the non-thread safe HashMap should lead to:

No output other than the program header

Significant CPU increase observed from the system

At some point the Java program will hang and you will be forced to kill the Java process

What happened? In order to understand this situation and confirm the problem, we will perform a CPU per Thread analysis from the Windows OS using Process Explorer and JVM Thread Dump.

1 - Run the program again then quickly capture the thread per CPU data from Process Explorer as per below. Under explore.exe you will need to right click over the javaw.exe and select properties. The threads tab will be displayed. We can see overall 4 threads using almost all the CPU of our system.

2 – Now you have to quickly capture a JVM Thread Dump using the JDK 1.7 jstack utility. For our example, we can see our 3 worker threads which seems busy/stuck performing get() and put() operations.

at org.ph.javaee.training4.HashMapInfiniteLoopSimulator.main(HashMapInfiniteLoopSimulator.java:75)

As you can see, the above correlation and analysis is quite revealing. Our main Java program is in a hang state because our 3 worker threads are using lot of CPU and not going anywhere. They may appear "stuck" performing HashMap get() & put() but in fact they are all involved in an infinite loop condition. This is exactly what we wanted to replicate.

HashMap infinite looping deep dive

Now let’s push the analysis one step further to better understand this looping condition. For this purpose, we added tracing code within the JDK 1.7 HashMap Java class itself in order to understand what is happening. Similar logging was added for the put() operation and also a trace indicating that the internal & automatic rehash/resize got triggered.

The tracing added in get() and put() operations allows us to determine if the for() loop is dealing with circular dependency which would explain the infinite looping condition.

Again, the added logging was quite revealing. We can see that following a few internal HashMap.resize() the internal structure became affected, creating circular dependency conditions and triggering this infinite looping condition (#iterations increasing and increasing...) with no exit condition.

It is also showing that the resize() / rehash operation is the most at risk of internal corruption, especially when using the default HashMap size of 16. This means that the initial size of the HashMap appears to be a big factor in the risk & problem replication.

Finally, it is interesting to note that we were able to successfully run the test case with the non-thread safe HashMap by assigning an initial size setting at 1000000, preventing any resize at all. Find below the merged graph results:

The HashMap was our top performer but only when preventing an internal resize. Again, this is definitely not a solution to the thread safe risk but just a way to demonstrate that the resize operation is the most at risk given the entire manipulation of the HashMap performed at that time.

The ConcurrentHashMap, by far, is our overall winner by providing both fast performance and thread safety against that test case.

JBoss AS7 Map data structures usage

We will now conclude this article by looking at the different Map implementations within a modern Java EE container implementation such as JBoss AS 7.1.2. You can obtain the latest source code from the github master branch.

Find below the report:

Total JBoss AS7.1.2 Java files (August 28, 2012 snapshot): 7302

Total Java classes using java.util.Hashtable: 72

Total Java classes using java.util.HashMap: 512

Total Java classes using synchronized HashMap: 18

Total Java classes using ConcurrentHashMap: 46

Hashtable references were found mainly within the test suite components and from naming and JNDI related implementations. This low usage is not a surprise here.

References to the java.util.HashMap were found from 512 Java classes. Again not a surprise given how common this implementation is since the last several years. However, it is important to mention that a good ratio was found either from local variables (not shared across threads), synchronized HashMap or manual synchronization safeguard so “technically” thread safe and not exposed to the above infinite looping condition (pending/hidden bugs is still a reality given the complexity with Java concurrency programming…this case study involving Oracle Service Bus 11g is a perfect example).

A low usage of synchronized HashMap was found with only 18 Java classes from packages such as JMS, EJB3, RMI and clustering.

Finally, find below a breakdown of the ConcurrentHashMap usage which was our main interest here. As you will see below, this Map implementation is used by critical JBoss components layers such as the Web container, EJB3 implementation etc.

Used in the context of ClassLoader and concurrent static Map data structures involving concurrent Threads access.

Total: 3

## JBoss Test Suite

Used in some integration testing test cases such as an internal Data Store, ClassLoader testing etc.

Total: 3

Final words

I hope this article has helped you revisit this classic problem and understand one of the common problems and risks associated with a wrong usage of the non-thread safe HashMap implementation. My main recommendation to you is to be careful when using an HashMap in a concurrent threads context. Unless you are a Java concurrency expert, I recommend that you use ConcurrentHashMap instead which offers a very good balance between performance and thread safety.

As usual, extra due diligence is always recommended such as performing cycles of load and performance testing. This will allow you to detect thread safety and / or performance problems before you promote the solution to your client production environment.

Please provide any comments and share your experience with ConcurrentHashMap or HashMap implementations and troubleshooting.