December 12, 2011

Recently, I was engaged in Exadata POV for benchmarking for Ebiz R12 on Exa* boxes. We had quarter rack running 11.2.0 db with 2 node RAC Cluster (24 CPU) and all other components/Middle Teir including Form, Web, CM were configured on Exalogic Server. Objective was to achieve best possible scalability numbers on Exa* boxes for few Ebiz modules.

Both boxes were connected with Infiniband network, Test scenarios were given for Order2Cash and Self-services modules. Idea was to utilized Load runner to generate the load using few hundred users. After struggling a while we could streamlined our testing scripts and now were ready to focus on performance.

We had several hiccups while running performance tests but issues were eroded by following generic guidelines on performance tuning, we were doing good progress until we stuck with “library cache: mutex X” issues which was kind of road block for us. Each time we touch 600 concurrent users for Order 2 Cash, we had high waits on mutex issue, CPU utilization were constantly touching 100%. Have a look to below screenshot, most of the sessions were waiting on concurrency.

One of my colleague immediately pointed out about known bug in Oracle 11g for mutex issues, We could find several hits for the bug but apply a patch was not an option due to conflict with

Exadata Bundle Patch. Latest BP did not solve the issue and we were left out to do something by own ( if we can ).

I was interested to *PINNED* column, it tells high number of calls for IGI_GEN_VERT and MO_GLOBAL packages.By default, Oracle creates multiple copies of object based on CPU Core. But still this could lead to contention if you have highly concurrent users using same package. I came to know about undocumented dbms_shared_pool.markhot() procedure Markhot procedure creates multiple copies of object in shared pool which can eliminate the contention.

Since, it was not an production instance, I decided to give it a try by marking both packages as hot.