443892008-02-11 02:03:09 +0000Cleanup of org.apache.naming.resources.CacheEntry resources (memory leak)2008-04-16 14:56:37 +0000111UnclassifiedTomcat 6Catalina6.0.14PCAllRESOLVEDFIXEDP2normaldefault1drodriguezdev0oldest_to_newest1136040drodriguez2008-02-11 02:03:09 +0000Hi,
I am memory profiling a webapp for my company because we suspect it has
memory leaks when redeploying. This webapp has to be redeployed several times a
week for security reasons (most of the time it is undeployed) and the memory
leaks forced us to shutdown tomcat once a week to avoid OOM exception.
I'm using jvm1.5 and tomcat 6.0.13 (also tried with 6.0.14 and today with
6.0.16), and Yourkit as the profiling tool. I have made the test in both linux
and windows.
I have created a JMeter test to deploy the app using the manager, make some
request to the app and then undeploy it using again the manager app in a loop.
After some redeployments, I got my OutOfMemoryException. Then I checked with my
profiling tool and found that my own classes were correctly destroyed, and the
classes retaining more memory were these ones (listed as a heirarchy):
org.apache.catalina.core.ApplicationContext$DispatchData
org.apache.catalina.core.ApplicationContext
mapper of org.apache.catalina.core.StandardContext
context of org.apache.tomcat.util.http.mapper.Mapper
resources of org.apache.tomcat.util.http.mapper.Mapper$Context cache of
org.apache.naming.resources.ProxyDirContext cache of
org.apache.naming.resources.ResourceCache
org.apache.naming.resources.CacheEntry[]
The number of objects of each class were exactly the number of times I had
redeployed the webapp. I have checked the contents of CacheEntry inside
CacheEntry[] and it has an attribute "name" and the contents are the jsps and
tld files of my app. There are as many CacheEntry with the same jsp attribute as
times the webapp has been reloaded, so I guess there is no cleaning of
CacheEntry resources at context shutdown.
To reproduce this, a simple webapp with 2 jsp and a taglibrary can be used.
Deploy and undeploy it many times so you can see how the count of these objects
grows in memory.1142461arto.huusko2008-03-04 05:26:16 +0000Caused by non-static inner class DispatchData in org.apache.cataline.core.ApplicationContext being held in a (non-static) ThreadLocal.
While seemingly dispatchData is GC'able, Sun JVM Thread class holds reference to the ThreadLocal object through Thread class. Example path (from jhat):
--> java.lang.ThreadGroup@0x2aa4b85608 (67 bytes) (field threads:)
--> [Ljava.lang.Thread;@0x2aa674dda0 (272 bytes) (Element 21 of [Ljava.lang.Thread;@0x2aa674dda0:)
--> java.lang.Thread@0x2aa6f4b418 (164 bytes) (field threadLocals:)
--> java.lang.ThreadLocal$ThreadLocalMap@0x2aa6f86d18 (32 bytes) (field table:)
--> [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;@0x2aa6f86d38 (528 bytes) (Element 16 of [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;@0x2aa6f86d38:)
--> java.lang.ThreadLocal$ThreadLocalMap$Entry@0x2aa6f87408 (56 bytes) (field value:)
--> org.apache.catalina.core.ApplicationContext$DispatchData@0x2aa6f87440 (40 bytes) (field this$0:)
--> org.apache.catalina.core.ApplicationContext@0x2aa6f87b88 (72 bytes)
Because dispatchData field is non-static, what ever it contains is leaked when a particular ApplicationContext is discarded: a new ThreadLocal object is created for each ApplicationContext instance in each Thread, and when the instance is discarded, an unaccessible but still not GC'able reference stays in the JVM's Thread instances that used it.
Because DispatchData inner class is not static, it also causes a reference to the ApplicationContext instance itself, which leads to catastrophic results, if the ClassLoader of the web app is referencable throught the ApplicationContext.1155632markt2008-04-11 15:36:54 +0000Testing with YourKit confirms that making the nested class static fixed the leak.
Thanks for the analysis.
The fix has been applied to trunk and proposed for 6.0.x.1156523markt2008-04-16 14:56:37 +0000This has been fixed in svn for 6.0.x and will be included in 6.0.17 onwards.