Oracle 11g internals part 1: Automatic Memory Management

This is my attempt for getting cheap popularity out of recent Oracle 11g release. This is not going to be another Oracle 11g new features list, I’ll be just posting any of my research findings here, in a semi-organized way.

The first post is is about Automatic Memory Management. AMM manages all SGA + PGA memory together, allowing it to shift memory from SGA to PGAs and vice versa. You only need to set a MEMORY_TARGET (and if you like, MEMORY_MAX_TARGET parameter).

You can read rest of the general details from documentation, I will talk about how this feature has been implemented on OSD / OS level (or at least how it looks to be implemented).

When I heard about MEMORY_TARGET , then the first question that came into my mind was that how can Oracle shift shared SGA memory to private PGA memory on Unix? This would mean somehow deallocating space from existing SGA shared memory segment and releasing it for PGA use. To my knowledge the traditional SysV SHM interface is not that flexible that it could downsize and release memory from a single shared memory segment. So I started checking out how Oracle had implemented this.

One option of course is not to implement it at all – just do not use the extra space in the extra SGA area and it will be soon paged out if there’s memory pressure (as long as you don’t keep your SGA pages locked – which can’t be used together with MEMORY_TARGET anyway). However should this “unneeded” memory be used again, all would have to be loaded back from swap area.

pmap output reveals that Oracle 11g likes to use /dev/shm for shared memory implementation instead. There are multiple 16MB “files” mapped to Oracle server processes address space.
This is the Linux’es POSIX-oriented SHM implementation, where everything, including shared memory segments, is a file.

Thanks to allocating SGA in many smaller chunks, Oracle is easily able to release some parts of SGA memory back to OS and server processes are allowed to increase their aggregate PGA size up to the amount of memory released.
(Btw, if your MEMORY_MAX_TARGET parameter is larger than 1024 MB then Oracle’s memory granule size is 16MB on Linux, otherwise it’s 4MB).

Note that the PGA memory is still completely independent memory, allocated just by mmap’ing /dev/zero, it doesn’t really have anything to do with shared memory segments ( unless you’re using some hidden parameters on Solaris, but that’s another story ).
PGA_AGGREGATE_TARGET itself is just a recommended number, leaving over from MEMORY_TARGET – SGA_TARGET (if it’s set). And Oracle uses that number to decide how big PGAs it will “recommend” for sessions that are using WORKAREA_SIZE_POLICY=AUTO.

So how does Oracle actually release the SGA memory when it’s downsized?

Compare these outputs:

/dev/shm before starting instance:

$ ls -l /dev/shm/
total 0

Obviously there’s nothing reported as no /dev/shm segments are in use.

/dev/shm after starting instance with fairly large SGA (note that some output is cut for brewity).
See how some of the memory “chunks” are zero in size. These chunks are the ones which have been chosen as victims for destruction (or for not-even-creation) when space was needed or PGA areas. If you look into pmap output for any server processes you will still see this memory mapped into the address space, but it’s not just used because Oracle knows this memory is really freed.

So, in Linux (tested on OEL5) Oracle 11g is using a new mechanism for managing shared memory. Well this mechanism itself isn’t that new, but it’s unconventional considering the long history of Unix SysV SHM segment use for Oracle SGAs.

Does this all matter? Yes
Why does it matter? There are few administrative differences compared to the conventional implementation.First of all, ipcs -m doesn’t show the full size of these segments anymore. You need to list /dev/shm contents for that.Also, pmap always reports that the memory is mapped (because it is) even though it doesn not have physical backing storage on tmpfs on /dev/shm device.

One more important note is that if you have not configured your tmpfs size on /dev/shm properly, then Oracle fails to allocate new POSIX-style shared memory and will not allow you to use MEMORY_TARGET parameters (startup without those parameters will however succeed).

The error message you likely get looks like that:

SQL> ORA-00845: MEMORY_TARGET not supported on this system

And is accompanied by following entry in alert.log:

Sat Aug 18 12:37:31 2007
Starting ORACLE instance (normal)
WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to be mounted for at least 847249408 bytes. /dev/shm is either not mounted or is mounted with available space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is 0 and used is 0 bytes.
memory_target needs larger /dev/shm

So you need to configure large enough tmpfs on /dev/shm device to fit all memory up to MEMORY_MAX_TARGET.

This one allows /dev/shm to grow roughly up to 1300MB, allowing you to use MEMORY_MAX_TARGET (or MEMORY_TARGET) set to 1300MB. The Linux-specific Oracle 11g documentation has more details how to configure this.

Note that after resetting various parameters I played with I realized that finally Oracle has implemented the human-friendly way for resetting parameters in SPFILE:

oops, the way I wrote that insinuated that Oracle uses Indirect Data Buffers to implement 11g AMM…I didn’t mean that…what I meant was Oracle has exercised this style of shared memory before…albeit for an entirely different reason.

Thanks for the great in depth explanation. I benifited alot from it. I have a question, I set my memory max target to 28G and memory target 24G, my system has by default max_sga=4G. Will the system ignore the max sga? or it will not give the sga_target more the 4GB? shall I set it to 0?
my physical memory 32G and oracle 11g is the only app on the system
Thanks again and keep up the good work

Hello there,
Thank you for all the info.
I am very new to oracle 11g install on linux, tried to recover a db (created using dbca) using rman, ran into error,when tried to do startup mount, got the error described here, I am trying to implement the solution given by you,
tried to umount /dev/shm- got the error device is too busy,
does this mean I have to shut down the db and then try to unmount?
Can you please help? Many thanks in anticipation.

Tanel, Thank you for your excellent analysis and summary of Oracle 11g memory management. I am installing 11g on a new Sun server and could not figure out why I received memory allocation errors when the SGA was increased to more than 4GB.

It looks like we are seeing 5.6gb of memory used, so I’m confused about why our sgas are defined to consume 8.75gb, but we only see 5.6gb used. read that the sga by default takes 60% of memory_target and pga gets the rest. Because there are not many connections, is this why we see a lower value?

Does the tmpfs at 16gb mean we can only use 16gb of ram for our databases?

We are just getting up and running, so there are not yet many connections. I
Any insights would be appreciated.

I have a little comment reqarding calculating free memory and posix shared memory segments on Linux.

After doing some tests, I see that SysV IPC shared memory segment allocation is “invisible” when issuing the “free” command, i.e the memory allocated is not accounted for as used. Allocated posix shared memory in /dev/shm seems to be counted for under “cached”.

It is not trivial hard to calculate how much memory which is available for applications on Linux, and here is what I think gives an *indication*:

Free memory according to “free” in the line “-/+ buffers/cache”
– sum(size of shared memory segments from ipcs -m)
– sum(size of shared memory segments in /dev/shm)
————————————————–
Memory available for applications
==================================================

Anonymous nmaps are not accounted for in this calculation, neither is hugepages and I guess there is other numbers I should take into account from /proc/meminfo.

But at least the numbers I get are better than using top and free without any consideration to shared memory segments.

Any comment or input regarding calculating free memory on linux would be greatly appreciated.

moro,
Just “a dummy one comment”.
There are still some “poor customers” who must use on same host more than one instance (per database on non-RAC).
Basing on this article and comments, on OS side there is one /dev/shm and SGA’s are using the very same /dev/shm (not possible to do other way?), so as result when calculating proper max size for /dev/shm (tmpfs) one should consider all SGAs (a’ka instances on host). Is’t possible to make more /dev/shm’s like /dev/shm1, /dev/shm2 and give the choosen one shm to choosen one instance – no?.

Very nice article. With AMM in 11g R2, if /dev/shm is not used for the PGA, how much space should we allocate for /dev/shm -> just the SGA_MAX_SIZE? For e.g.
In 10g R2, we had SGA_MAX_SIZE as 1.5GB and PGA_AGGREGATE_TARGET as 1GB, now when this database is upgraded to 11g, should MEMORY_MAX_TARGET be set to 1.5GB or 2.5GB?

Hi,
Does the value of MEMORY_TARGET depend on the number of CPUs in any way? We have a situation where if we increase the # of CPUs (enable them — OS is Solaris), then the cluster throws an <> error. Before the cpu count was increased, it was working fine.

Quick question:
Oracle says ” The use of AMM is absolutely incompatible with HugePages.” in of the notes of implementing HugePages.
In case of hugepage implementations, how is Oracle using memory structure. Does it use System-V way of using SGA or POSIX way.
We are using Hugepages. I see files getting created under /dev/shm. But number of files getting created directly corresponds to ‘nattch’ of ipcs.
In this case, is there way to configure /dev/shm using tmpfs.

Please reply with details if any on its usage in 11gR2 with respect to Hugepages.

When we set a value for PGA_AGGREGATE_TARGET parameter, oracle can use more OS memory for PGA than what we set in PGA_AGGREGATE_TARGET when it is required, with 11g AMM does this work same way? Can oracle allocate more than MEMORY_MAX_TARGET for PGA when it is required?

Great post, thanks.
I am dealing with many databases being upgraded to 11g from 10g. (all 3 node rac)
One of them actually have 100g sga per instance. In 10gR2 we didnt have amm and hugepages were on, in 10gR2 I used to see one line for ipcs -m for my each instance showing the sga size.

in 11gR2 I am not using the AMM and keeping both memory_target and memory_max_target 0.
Hugepages are still turned on(OEL 5). The alert log mentions Large Pages are being used(good new alert log feature of 11g). But ipcs -m actually shows 3 rows, one being the exact sga size, other two are of size 402653184 and 2097152, also despite the fact the we are not using AMM, /dev/shm has files from the unix user which owns the instance and the size of files amount to 500M (the sga in this case is 60g).

is it normal with 11g now to have 2 additional rows in ipcs -m and files in /dev/shm even though AMM is not used?