I'm trying to tune the JVM options for a JAVA app that requires low pauses. After playing around with several diferent combinations I still can not get a setup that will not give an occasional very long pause.

Anyway, I'm not sure which combination of options you've tried. Just let me scratch the surface.
- Would you try lower CMSInitiatingOccupancyFraction value? to 55 or something?
- Would you try CompactAtFullCollection option off?
- Would you try CMSIncrementalMode instead of CompactAtFullCollection?

I've played enough with the occupancy fraction and I don't believe this is the culprit although it will definitely cause problems if it's set too high.

I also already played with ICMS, perhaps not enough but I don't think this will help either.

I turned diagnostics on so I can create another log file in the format you requested. The test is running now, I can't say how long it will take to get a long pause but it will ... I will post the results then.

Here's another big pause: I have the whole logfile if anyone is interested but here's the region where the pause happens. THere's nothing in the log that gives me a clue of what's doing when the pause happens.

7741.587: [GC 7741.588: [ParNew
Desired survivor size 94371840 bytes, new threshold 5 (max 8)
- age 1: 24412472 bytes, 24412472 total
- age 2: 21038144 bytes, 45450616 total
- age 3: 20597840 bytes, 66048456 total
- age 4: 19746928 bytes, 85795384 total
- age 5: 19062040 bytes, 104857424 total
: 921600K->102400K(921600K), 0.3088080 secs] 8269812K->7471051K(13209600K), 0.3093120 secs]
Total time for which application threads were stopped: 0.3196070 seconds
7752.152: [GC 7752.152: [ParNew
Desired survivor size 94371840 bytes, new threshold 5 (max 8)
- age 1: 22719688 bytes, 22719688 total
- age 2: 22518840 bytes, 45238528 total
- age 3: 20254832 bytes, 65493360 total
- age 4: 19675160 bytes, 85168520 total
- age 5: 18701336 bytes, 103869856 total
: 921600K->102400K(921600K), 0.3118630 secs] 8290251K->7495915K(13209600K), 0.3124020 secs]
Total time for which application threads were stopped: 0.3217890 seconds
7762.887: [GC 7762.887: [ParNew
Desired survivor size 94371840 bytes, new threshold 5 (max 8)
- age 1: 22629096 bytes, 22629096 total
- age 2: 21318736 bytes, 43947832 total
- age 3: 21733472 bytes, 65681304 total
- age 4: 19664776 bytes, 85346080 total
- age 5: 19511384 bytes, 104857464 total
: 921600K->102400K(921600K), 0.3134700 secs] 8315115K->7518690K(13209600K), 0.3139990 secs]
Total time for which application threads were stopped: 0.3213620 seconds
Total time for which application threads were stopped: 0.0036340 seconds
Total time for which application threads were stopped: 0.0033670 seconds
7773.727: [GC 7773.727: [ParNew
Desired survivor size 94371840 bytes, new threshold 5 (max 8)
- age 1: 22024192 bytes, 22024192 total
- age 2: 20501928 bytes, 42526120 total
- age 3: 20502032 bytes, 63028152 total
- age 4: 21021552 bytes, 84049704 total
- age 5: 19313504 bytes, 103363208 total
: 921600K->102400K(921600K), 0.3165940 secs] 8337890K->7543482K(13209600K), 0.3171480 secs]
Total time for which application threads were stopped: 0.3241270 seconds
Total time for which application threads were stopped: 0.0067410 seconds
Total time for which application threads were stopped: 0.0038350 secondsTotal time for which application threads were stopped: 56.1708090 seconds
Total time for which application threads were stopped: 0.0034490 seconds
Total time for which application threads were stopped: 0.0039120 seconds
Total time for which application threads were stopped: 0.0072940 seconds
Total time for which application threads were stopped: 0.0070070 seconds
Total time for which application threads were stopped: 0.0089790 seconds
Total time for which application threads were stopped: 0.0164470 seconds
Total time for which application threads were stopped: 0.0034560 seconds
Total time for which application threads were stopped: 0.0033550 seconds
Total time for which application threads were stopped: 0.0034270 seconds
Total time for which application threads were stopped: 0.0034200 seconds
Total time for which application threads were stopped: 0.0032840 seconds

This has worked for me to reduce the pause times. Also could you tell me how often is the pause time of that order is happening. Does it happen after a really long delay? Does it happen after a really long delay or quite frequently.This is an application that processes a million transactions a day and needs very low pause times.

Thanks for your suggestion. This would not work for me as we really need 24G of heap.

If what you meant is to use the same options but with 24G of heap I have already tried that sometime ago and I get pauses much worse than 40->50 secs. This options would let the JVM ergonomics choose the young generation.

It would be interesting if anyone know a verbose option that would show me what the JVM was doing during the pause. It was not a Full CMS otherwise we would see it in the logs, it was not a a concurrent mode failure, knowing what it is can help me to try to avoid the situation.

I think you have the most plausible explanation for my problem. I will experiment with the -XX+UseMembar option (although it seems to penalize performance) but I guess there will be a new JVM release for Java a6 soon as well.

We still see unacceptable long pauses after updating our application to 1.5u14. This needs to be addressed since no server application can sit dead for multiple seconds, especially when there is tons of free heap and other memory spaces are looking fine.

We still see unacceptable long pauses after updating our application to 1.5u14. This needs to be addressed since no server application can sit dead for multiple seconds, especially when there is tons of free heap and other memory spaces are looking fine.

This is not a continuation of the current thread, but should be a new thread.
If you start a new thread, could you post a larger GC log snippet? Does the
occasional ParNew pause display this long pause anomaly, or do the longer
ParNew pauses occur in a stretch? Please file a bug with bugs.sun.com
if you can still reproduce this problem with the latest update releases of the
JVM.

I encounter same problem, no "promotion failed" tip but ParNew cost more than 20s, the following is the gc log's snapshot:
my jdk version is the latest JDK6u21 on RHEL5.1/IBM x3850 with 16core.
I try to use -X::+UseMemBar, but no effect.