JBoss Developer: Message List - Memory consumption 1h after JBoss startMost recent forum messageshttps://developer.jboss.org/community/jbossas/performance?view=discussionsJive Engage2010-08-05T06:27:36Z2010-08-05T06:27:36ZenRe: Memory consumption 1h after JBoss startJozsef Attila/people/jozsef_attilado-not-reply@jboss.com2010-08-05T06:27:36Z2010-08-05T06:27:36Z<!-- [DocumentBodyStart:da4a8219-fcf3-402c-baa0-f3c3e9549b8b] --><div class="jive-rendered-content"><p>Hi,</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Actually it looks like after exactly 1 hour it starts leaking unnamed event handles. This is on Windows 2003 server, JBoss 5.1, Java 6 u20. The application does nothing special after exactly 1 hour from startup.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p><span>I wonder if this is related to </span><a class="jive-link-external-small" href="https://jira.jboss.org/browse/JBVFS-159" rel="nofollow" target="_blank">https://jira.jboss.org/browse/JBVFS-159</a></p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Regards,</p><p>Arthur</p></div><!-- [DocumentBodyEnd:da4a8219-fcf3-402c-baa0-f3c3e9549b8b] --><img src='/beacon?t=1438562635804' />2010-08-05T06:27:36Z4 years 12 months ago0Re: Memory consumption 1h after JBoss startRuchir Choudhry/people/ruchircdo-not-reply@jboss.com2010-08-05T05:27:17Z2010-08-05T05:27:17Z<!-- [DocumentBodyStart:923630f5-aeba-4490-8f40-6d3504b16ab2] --><div class="jive-rendered-content"><p>-Xms512m -Xmx1024m--Can you please try try to make Min and Max of Heap as same, and publish the results.-Xms512m -Xmx512m</p></div><!-- [DocumentBodyEnd:923630f5-aeba-4490-8f40-6d3504b16ab2] -->2010-08-05T05:27:17Z4 years 12 months ago10Re: Memory consumption 1h after JBoss startJozsef Attila/people/jozsef_attilado-not-reply@jboss.com2010-08-04T11:32:48Z2010-08-04T11:32:48Z<!-- [DocumentBodyStart:11bebdf6-7cdc-4cbe-9adf-b7c2148bf7fa] --><div class="jive-rendered-content"><p>Hi Giovanni,</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Did you find a solution to this memory consumption? I think we may be suffering from something similar.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Regards,</p><p>Jozsef</p></div><!-- [DocumentBodyEnd:11bebdf6-7cdc-4cbe-9adf-b7c2148bf7fa] -->2010-08-04T11:32:48Z4 years 12 months ago0Re: Memory consumption 1h after JBoss startPeter Johnson/people/peterjdo-not-reply@jboss.com2010-07-23T18:13:30Z2010-07-23T18:13:30Z<!-- [DocumentBodyStart:e0f54cc3-4c5d-4e57-875d-f7725402e81f] --><div class="jive-rendered-content"><p>I ageee that your explanation of setting MaxTenuringThreshold to 0 is the commonly accepted advice. But of course, I disagree <span alt="Grin" class="emoticon-inline emoticon_grin" style="height:16px;width:16px;"></span></p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>The problem with setting MaxTenuringThreshold to 0 is that at the time the minor GC takes place you will have a lot of temporary objects that are still in use being promoted to the tenured generation. Then the tenured generation fills up quicker and you end up doing major CMS collections more often. It is usually better to let objects bounce around a few times between the survivor spaces in order to sift out objects that survive only a few collections and are then discarded, thus ensuring that only object that plan to stay around for a while get tenured.Tenuring temporary object is much more expensive than not tenuring long-lived objects.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Test I have done show that the minor collection time, even with MaxTenuringThreshold not set to 0, is consistently less that the two pauses times caused by the CMS collectior for a major collection. In addtion, having MaxTenuringThreshold set to a non-zero value has, in my tests, always outperfomed setting MaxTenuringThreshold to 0 (higher throughput, lower response times). At least, that is what I have observed in the many load tests that I have performed.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Of course, all apps are different - the reader is urged to perform his or her own load tests to see how their apps respond.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>And Ruchir, I agree with you that discussions like this are benefitial to the community.<span alt="Happy" class="emoticon-inline emoticon_happy" style="height:16px;width:16px;"></span> Hopefully the readers can take our two views into consideration and find what works best for their apps.</p></div><!-- [DocumentBodyEnd:e0f54cc3-4c5d-4e57-875d-f7725402e81f] -->2010-07-23T18:13:30Z5 years 1 week ago0Re: Memory consumption 1h after JBoss startRuchir Choudhry/people/ruchircdo-not-reply@jboss.com2010-07-22T21:37:06Z2010-07-22T21:37:06Z<!-- [DocumentBodyStart:27aa4600-3597-4540-922a-13fc5b2b55e6] --><div class="jive-rendered-content"><p>Apprecite your suggestions,</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>It all depends how you want to do it, JVM parameters gives a lot of flexibility to you and there is no one size fits all.</p><p>Moreover my intent was to help the team get some idea, and&nbsp; not to cut copy the exact thought, Diffrent app requires different type of setup.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>We ran series of test "Load, noise, bursting, endurace" with multiple appications on single Jboss EAP4.3 with different veriosn of JDK. Results came best with the setup we have explained.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>We used <code>-XX:MaxTenuringThreshold=0</code> because it</p><p>promotes all objects&nbsp; allocated by the application in the young generation, which survive a&nbsp; single GC cycle, directly to the old generation without copying them&nbsp; around between the survivor spaces in the young generation. This&nbsp; setting, when used with CMS collector in the old generation helps in two&nbsp; ways.</p><ul><li> The young generation GC does not have to waste its time copying the&nbsp; 10-20% long lived objects multiple times between the survivor spaces,&nbsp; only&nbsp; to finally promote them to the old generation.</li><li> Additionally, most of the collection and cleaning work of these&nbsp; objects can be done "concurrently" in the old generation. This behavior&nbsp; leads to additional reduction in GC sequential overhead.</li></ul><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p><a class="jive-link-external-small" href="http://java.sun.com/developer/technicalArticles/Programming/turbo/" rel="nofollow" target="_blank">http://java.sun.com/developer/technicalArticles/Programming/turbo/</a></p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>This setup was also good based on our test scenarios.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>I guess you are bringing some good points, we will use this in next series of test.Keep giving new Ideas<span alt="Happy" class="emoticon-inline emoticon_happy" style="height:16px;width:16px;"></span></p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Respectfully,</p><p>Ruchir Choudhry</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"><span alt="Wink" class="emoticon-inline emoticon_wink" style="height:16px;width:16px;"></span>&nbsp;</p></div><!-- [DocumentBodyEnd:27aa4600-3597-4540-922a-13fc5b2b55e6] -->2010-07-22T21:37:06Z5 years 2 weeks ago10Re: Memory consumption 1h after JBoss startPeter Johnson/people/peterjdo-not-reply@jboss.com2010-07-22T17:02:07Z2010-07-22T17:02:07Z<!-- [DocumentBodyStart:f5f06011-0826-43e0-a8fa-3f341bf0099a] --><div class="jive-rendered-content"><p>Ruchir, your suggested newsize is too small for the heap size specified - the newsize should be 1/3 - 1/4 the size of the heap. By the way, based on my tests, if you do not specify a newsize, JDK 6 eventually adjusts it to about 1/3 - 1/4 the heap size (just realized that was with the standard collection, not the CMS collector, but I suspect that letting the ).</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Setting MaxTenuringThreshold to 0 is a bad idea - I puts a larger strain on the CMS collector and degrades overall applicaiton performance. It is much better ot let needed objects survive a few minor collections before being tenured, and thus let the minor collector gather all the garbage, than requiring the CMS collector to collect objects that die soon. Besides, setting the UseConcMarkSweepGC sets MaxTenuringThreshold to 0 by default (and also sets the survivor spaces to a really small value.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>It is not always advisable to set UseConcMarkSweepGC. You must test your application both with and without it to determine hte best performance. Also, with UseConcMarkSweepGC, during a major collection, one of your CPUs will be dedicated to garbage collection. If you have only 2 CPUs, then you will reduce the amount of load your app can handle by 50%.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>For suggestions on how to gather GC data and analyze it, see these white papers:</p><p>Java Garbage Collection Statistical Analysis 101</p><p>Java Garbage Collection Performance Analysis 201 (this white paper discusses the CMS collector and its gotchas)</p><p>at <a class="jive-link-external-small" href="http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&amp;x=30&amp;y=10" rel="nofollow">http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&amp;x=30&amp;y=10</a></p><h2></h2></div><!-- [DocumentBodyEnd:f5f06011-0826-43e0-a8fa-3f341bf0099a] -->2010-07-22T17:02:07Z5 years 2 weeks ago20Re: Memory consumption 1h after JBoss startRuchir Choudhry/people/ruchircdo-not-reply@jboss.com2010-07-22T07:07:19Z2010-07-22T07:07:19Z<!-- [DocumentBodyStart:75bcc5d2-dfb7-4fc3-9488-177b28fe4ef8] --><div class="jive-rendered-content"><p>Hello,</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Please look into the following things, it will surly help you.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>01&gt; Your JVM setting is not as per Jboss or Sun Standards. both says the Xms and Xms need to be same. Try the below setup change it based on your system needs.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>-Xmx2048m -Xms2048m -XX:MaxNewSize=256m -XX:NewSize=256m -XX:MaxPermSize=128m -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=60 -XX:LargePageSizeInBytes=5m -XX:ParallelGCThreads=20 -XX:-DisableExplicitGC&nbsp; -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:ThreadStackSize=1024 -Dsun.rmi.dgc.client.gcInterval=3600000</p><p>-Dsun.rmi.dgc.server.gcInterval=3600000</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>I have listed a series of test on various parameters, as below(link) try to see if you can grab some info from there.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p><a class="jive-link-external-small" href="http://community.jboss.org/docs/DOC-15623?uniqueTitle=false" rel="nofollow" target="_blank">http://community.jboss.org/docs/DOC-15623?uniqueTitle=false</a></p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>02&gt; you can also use Visual VM provided by Sun to see if its something induced due to profiling, I dont see this as a reason but no harm is checking.</p><p>03&gt; The the GC cycle when its happening and what is the object left in NEW Gen and Old Gen, dose it get cleared eventually or its holding up.</p><p>04&gt; If its holding up than the problem is with the code, profile it, you will get explicit details which class, inner class, which method is causing it</p><p>05&gt; Are you using a lot of static methords, caching in JVM or large session object</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Please let me know how it goes, Publish the machine details too.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Thanks,</p><p>Ruchir</p></div><!-- [DocumentBodyEnd:75bcc5d2-dfb7-4fc3-9488-177b28fe4ef8] -->2010-07-22T07:07:19Z5 years 2 weeks ago30Re: Memory consumption 1h after JBoss startPeter Johnson/people/peterjdo-not-reply@jboss.com2010-01-22T20:56:29Z2010-01-22T20:56:29Z<!-- [DocumentBodyStart:7370d747-b9e6-4ca0-bc3b-b7091cec7e78] --><div class="jive-rendered-content"><p>It almost looks like a new app is being deployed at that time. That would explain the huge jump in the number of Zip* instances. Is there anything in the log that might shed light on what is going on?</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Also, add the "-verbose:class" JVM option to JAVA_OPTS in the run script. This option causes the JVM to print to stdout every class laoded and the JAR from which it came. It would be interesting to see what the output is when the spike happens. Of course, redirect stdout/stderr to a file - you will get a ton of output.</p></div><!-- [DocumentBodyEnd:7370d747-b9e6-4ca0-bc3b-b7091cec7e78] -->2010-01-22T20:56:29Z5 years 6 months ago0Re: Memory consumption 1h after JBoss startGiovanni Formenti/people/formentido-not-reply@jboss.com2010-01-22T11:06:07Z2010-01-22T11:06:07Z<!-- [DocumentBodyStart:4b6c6b4f-b535-4c61-9cbd-901e31789199] --><div class="jive-rendered-content"><p>Hi Peter,</p><p>I used Eclipse MAT (a great tools IMHO!) and also JProfiler. This is the difference between the memory after 50min and 63min (the peak is around 60min).</p><p><a href="https://developer.jboss.org/servlet/JiveServlet/showImage/2-521686-1829/heap2.JPG"><img alt="heap2.JPG" class="jive-image jive-image-thumbnail" height="384" onclick="" src="https://developer.jboss.org/servlet/JiveServlet/downloadImage/2-521686-1829/805-384/heap2.JPG" width="805"/></a></p><p>I trace the GC also and I notice a strange behavior of Survivor Space when the GC begin its intensive activity (after 1h). It seems that it exceeded the maximum:</p><p><a href="https://developer.jboss.org/servlet/JiveServlet/showImage/2-521686-1830/heap3.JPG"><img alt="heap3.JPG" class="jive-image jive-image-thumbnail" height="391" onclick="" src="https://developer.jboss.org/servlet/JiveServlet/downloadImage/2-521686-1830/743-391/heap3.JPG" width="743"/></a></p><p>Exploring the memory dump, the Maps and ZipEntries seem to be related to MBean map and deploy/vfs structures. Their shortest path to GC root involve http-0.0.0.0-8080 threads.</p><p>This could be related to connection pools or aliveness of connections?</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Thank you for any tips about this</p><p>Regards</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Giovanni</p></div><!-- [DocumentBodyEnd:4b6c6b4f-b535-4c61-9cbd-901e31789199] -->2010-01-22T11:06:07Z5 years 6 months ago10Re: Memory consumption 1h after JBoss startGiovanni Formenti/people/formentido-not-reply@jboss.com2010-01-22T10:50:15Z2010-01-22T10:50:15Z<!-- [DocumentBodyStart:1fdcdaf5-371a-460d-b4a9-652c57a08402] --><div class="jive-rendered-content"><p>Hi luv,</p><p>you are right but before the test I disable all kind of logging <span alt="Happy" class="emoticon-inline emoticon_happy" style="height:16px;width:16px;"></span></p><p>Thank you for your support</p><p style="min-height: 8pt; height: 8pt; padding: 0px;">&nbsp;</p><p>Regards</p><p>Giovanni</p></div><!-- [DocumentBodyEnd:1fdcdaf5-371a-460d-b4a9-652c57a08402] -->2010-01-22T10:50:15Z5 years 6 months ago0