The latest news directly from E-Business Suite Development

Do You Need to Bounce E-Business Suite Application Servers Regularly?

A customer recently reported an issue where they needed to bounce their E-Business Suite application tier (mid-tier) servers once a week to resolve stability issues. This shouldn't be necessary. We recommend against bouncing EBS application servers regularly. We've done a lot of work with the E-Business Suite to ensure that regular bounces are not required with the latest releases.

Bouncing your system can reduce performance

It seems counter-intuitive, but bouncing your EBS application tier servers can actually reduce your system's performance rather than improve it. Oracle E-Business Suite is explicitly designed to minimize round trips between the application tier and the database. We make extensive use of caches:

JDBC cache buffers

Java object cache

MSD cache for OA Framework pages

... and others

Every time you bounce your mid-tier, you clear these caches. The longer your mid-tier remains up and running, the better-populated these caches will be, and the better your overall system performance.

If you're experiencing problems that seem to go away when you bounce your mid-tier, that's a red flag. You should log a formal Service Request with Oracle Support so that we can help you identify the root cause.

I'm glad you wrote about this and I hope other customers give their feedback on this topic.

All the issues mention above are all real-world issues faced by customers regularly. We've been trying to get the java heap size right for years now, we still get the occasional java.lang.OutOfMemory errors. My experience working on these issues with Oracle support has been very very frustrating. I recently raised an SR on this, but had to close it after a few weeks as the analyst was kind of shooting in the dark.

If someone in the EBS Performance team in Oracle can clearly lay out some best practices document and settings to avoid these issues, it would tremendously help the Oracle Apps DBA community.

No single set of settings will meet everyone's needs. Your first step should be to review the presentation linked in the article above to isolate the issues as much as possible. Once you get to that stage, log another SR, send it to me, and I'll have one of our Applications Performance specialists in ATG Development work with your Support engineer on it.

The screen shot is from JConsole? If so, how do you get it to work? I never could make it work with either 11i, 12.0.x or 12.1.3. I found many doc on metalink and even opened SR and never could get my issue fix.

Whenever I start JConsole, the only process it finds is itself.
Any idea?

What version of java are you using? There can be some issues with JDK lower than 1.5.10. Additionally, ensure that the following system property is set in the server

com.sun.management.jmxremote

Additionally, if you want to connect remotely (let's say from your desktop to the server), you need additional parameters depending on your current configuration. If you have multiple JVMs per instance, you need to connect via process PID locally on the server given that multiple JVMs will compete for the same port in the server. Just execute $INST_TOP/admin/scripts/adopmnctl.sh status to find the OACORE instances PIDs.

Feel free to shoot me an email at gustavo.jimenez@oracle.com for further details.

In order to tune your JVM settings in case you run out of heap space in your JVM, ensure that these JVM options are present:

-XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/some_dir/

This will cause the JVM to create an HPROF file for later examination with any heap analysis tool (i.e.: Eclipse Memory Analyzer) to find the source of the leaks and fix or tune heap space accordingly.

Note that the above JVM options are available starting with JDK 1.5.10, so you definitely need to upgrade your JDK if you are below 1.5.10.

If the heap shows signs of memory leaks caused by e-Business Suite code or techstack components, please do let us know right away via a Service Request with Oracle Support.

Since MWA runs in its own dedicated JVM (outside of the J2EE container-namely OC4J or Apache Jserv), it will not affect the entire stability of the container. Your point is valid and should perhaps be addressed via a service request with the MWA team to ensure and/or to provide reasons for this requirement.

The objective of this blog entry was to clear confusion regarding the need of bouncing the main middle tier J2EE container at regular intervals, which is not affected by the MWA telnet server.

Having said the above, regarding the note you mention, I extract the following paragraph which caught my attention:

"3. It is always recommended to be on the latest version of Java for the JDK Caching ability and MWA performance is increased with each newer version. The following Documents can be used for upgrading the JDK. With JDK versions 5 and 6, MWA no longer requires a daily bounce. "

I'd agree with Naveen's initial comments. If you want to avoid doing the daily bounce and find the cause of the issue, then be prepared for weeks of investment in time to troubleshoot the problem. During this time be prepared to hit the problem (for which you have implemented the bounce) numerous times and see how the business like it.
My experience has led me to over allocate JVMs to mitigate against against memory leaks and perform a staggered bounce accross Apps tiers. The cost in doing this has been far better than the cost of the time invested in going backwards and forwards with Oracle.
Until we have an easier method to find the culprits (perhaps when weblogic is introduced) I'm sticking to this.

Thanks for sharing your own experiences. Everyone's mileage varies, since everyone's respective tolerance for bounces and investigating SRs is different. I'm glad that you've found an approach that works for you.

If you're finding that bounces are required, we still would prefer to help track down the root issues in EBS environments that cause JVM leaks. It's entirely possible that we've already fixed the issue in a released patch, so it never hurts to call us before resigning yourself to a rolling bounce schedule.

I would like to add one question, recently we migrated on inv_replenish_detail_pub package to production instance . next day PO receipt started giving error RVTII-060-Subroutine
INV_TXN_MANAGER_PUB.process_transaction() returned Error

we bounce application server and problem got resolved. is this mandatory to bounce the server or any configuration problem in server?