4 node oracle Rack high cpu i/o wait

We are using 4 node RAC on redhat. weekly node1, node3 and node4 memory increase, than swap after that CPU i/o wait increase than (above %35) all application will kill. This incident happening weekly 2 or 3 times in either any node (node1, node3 and node4) . During the incident time we found KSWAPD Process use above 90% CPU Utilisation and more swapin and swapout activity happening. Even we logged call with hewlett packard as per them due to big database OS mis behaving and more swap out happening and recommended hugepage settings. hp recommending out of 72gb memory allocate 50gb to oracle (pga+sga) the rest 22gb to OS

Please read below questions and recommend me what to do ? how to avoid this problem?

In the past few days, We have been collecting the information on all the production nodes, and with the four incidents since Jul 31, We need your help to understand the below questions to clarify on system behavior.

1) what has triggered the OS to clear the memory:
?? Memory Free from 1GB to 20GB in one minutes
?? Is this solely a react to the demand increase, or just because the server is experiencing RAM stress, and OS has to periodically cleared the memory?
?? More obvious observed in node 2

3) 30GB freed, 20GB from cached. Where is the other 10GB from?
?? - 5GB from pagetable? What will happen if pagetable is swapped???

4) During normal run, 50GB cached, 12GB page table, total used 70GB => Oracle is only allowed with 8GB RAM? Where is Oracle memory?

5) If Oracle memory is part of 50GB cached memory, what happened if this was reduced from 50GB to 30GB, while incident happened?

6) Current 72gb memory is not enough to handle ? need to upgrade memory ?

7) ?We also verified four occurrence of the mem issue since Jul 30, and the event always happened in the below sequence. So, I will suggest need to study on what has triggered OS to acquire more free memory, and any way to set a limit on the memory it can clear?
Per verification from storage, db, server, so far we have not really seen any workload increase when the problem happened. If this is an OS behavior, or due to resource constraint, please verify and confirm
So the sequence of event should be:
Memfree increased at 7:46am

Popular White Paper On This Topic

Hello
1- as it happens only on some nodes and in a same order , it may be related to the way you balanced the load and services/modules on nodes. please monitor your cluster waits , (cache fusion ,,,, ) to find what is the database activity witch cause the problem. Some abnormal status may be seemed exactly before the memory trashing.
2- mean time, you need to prevent the downtime. I recommend to make your paging/swaping faster. Use fast swap partitions on your SAN (add new SAN swaps with higher priority). It will make your system to be responsive even if you have 100000 pagings/sec (depending on SAN perf) . and hopefully your service will remain available. (note that it is just firefighting)
3- I think using hugemem may reduce the swap problem (I am not sure RH 5.4 support it)
4- Also investigate about HupePage , it will increase TLB hit ratio and reduce page table size and page maintenance.
look at following link for hugepage : http://download.oracle.com/docs/cd/B28359_01/serve r.111/b32009/appi_vlm.htm
5- monitor your swaping/paging by a vmstat command. cron it and redirect the out_put to a file. you can run following command per hour/day :
vmstat 1 |sed -e 's/^/echo "$(date) : /g' -e 's/$/\"/g' | bash > LOG_FILE
be careful about log file size.
6- try to log the ps command frequently (i prefer every min) to check processes on the mem trashing time.
7- look for "out of mem" or any abnormal alert in /var/log/messages .