Linux Memory Management or 'Why is there no free RAM?'Revision 2.3
Copyright 2004 sapphirecat. The text of this post is licensed under a Creative Commons License.

Sections

Overview of memory management

The mysterious 880 MB limit on x86

The difference among VIRT, RES, and SHR in top output

The difference between buffers and cache

Swappiness (2.6 kernels)

1. Overview of memory management
Traditional Unix tools like 'top' often report a surprisingly small amount of free memory after a system has been running for a while. For instance, after about 3 hours of uptime, the machine I'm writing this on reports under 60 MB of free memory, even though I have 512 MB of RAM on the system. Where does it all go?

The biggest place it's being used is in the disk cache, which is currently over 290 MB. This is reported by top as "cached". Cached memory is essentially free, in that it can be replaced quickly if a running (or newly starting) program needs the memory.

The reason Linux uses so much memory for disk cache is because the RAM is wasted if it isn't used. Keeping the cache means that if something needs the same data again, there's a good chance it will still be in the cache in memory. Fetching the information from there is around 1,000 times quicker than getting it from the hard disk. If it's not found in the cache, the hard disk needs to be read anyway, but in that case nothing has been lost in time.

To see a better estimation of how much memory is really free for applications to use, run the command:

Code:

free -m

The -m option stands for megabytes, and the output will look something like this:

The -/+ buffers/cache line shows how much memory is used and free from the perspective of the applications. Generally speaking, if little swap is being used, memory usage isn't impacting performance at all.

Notice that I have 512 MB of memory in my machine, but only 503 is listed as available by free. This is mainly because the kernel can't be swapped out, so the memory it occupies could never be freed. There may also be regions of memory reserved for/by the hardware for other purposes as well, depending on the system architecture.

2. The mysterious 880 MB limit on x86
By default, the Linux kernel runs in and manages only low memory. This makes managing the page tables slightly easier, which in turn makes memory accesses slightly faster. The downside is that it can't use all of the memory once the amount of total RAM reaches the neighborhood of 880 MB. This has historically not been a problem, especially for desktop machines.

To be able to use all the RAM on a 1GB machine or better, the kernel needs recompiled. Go into 'make menuconfig' (or whichever config is preferred) and set the following option:

Code:

Processor Type and Features ---->
High Memory Support ---->
(X) 4GB

This applies both to 2.4 and 2.6 kernels. Turning on high memory support theoretically slows down accesses slightly, but according to Joseph_sys and log, there is no practical difference.

3. The difference among VIRT, RES, and SHR in top output
VIRT stands for the virtual size of a process, which is the sum of memory it is actually using, memory it has mapped into itself (for instance the video card's RAM for the X server), files on disk that have been mapped into it (most notably shared libraries), and memory shared with other processes. VIRT represents how much memory the program is able to access at the present moment.

RES stands for the resident size, which is an accurate representation of how much actual physical memory a process is consuming. (This also corresponds directly to the %MEM column.) This will virtually always be less than the VIRT size, since most programs depend on the C library.

SHR indicates how much of the VIRT size is actually sharable (memory or libraries). In the case of libraries, it does not necessarily mean that the entire library is resident. For example, if a program only uses a few functions in a library, the whole library is mapped and will be counted in VIRT and SHR, but only the parts of the library file containing the functions being used will actually be loaded in and be counted under RES.

4. The difference between buffers and cache
Buffers are associated with a specific block device, and cover caching of filesystem metadata as well as tracking in-flight pages. The cache only contains parked file data. That is, the buffers remember what's in directories, what file permissions are, and keep track of what memory is being written from or read to for a particular block device. The cache only contains the contents of the files themselves.

Corrections and additions to this section welcome; I've done a bit of guesswork based on tracing how /proc/meminfo is produced to arrive at these conclusions.

5. Swappiness (2.6 kernels)
Since 2.6, there has been a way to tune how much Linux favors swapping out to disk compared to shrinking the caches when memory gets full.

ghoti adds:
When an application needs memory and all the RAM is fully occupied, the kernel has two ways to free some memory at its disposal: it can either reduce the disk cache in the RAM by eliminating the oldest data or it may swap some less used portions (pages) of programs out to the swap partition on disk.
It is not easy to predict which method would be more efficient.
The kernel makes a choice by roughly guessing the effectiveness of the two methods at a given instant, based on the recent history of activity.

Before the 2.6 kernels, the user had no possible means to influence the calculations and there could happen situations where the kernel often made the wrong choice, leading to thrashing and slow performance. The addition of swappiness in 2.6 changes this.
Thanks, ghoti!

Swappiness takes a value between 0 and 100 to change the balance between swapping applications and freeing cache. At 100, the kernel will always prefer to find inactive pages and swap them out; in other cases, whether a swapout occurs depends on how much application memory is in use and how poorly the cache is doing at finding and releasing inactive items.

The default swappiness is 60. A value of 0 gives something close to the old behavior where applications that wanted memory could shrink the cache to a tiny fraction of RAM. For laptops which would prefer to let their disk spin down, a value of 20 or less is recommended.

As a sysctl, the swappiness can be set at runtime with either of the following commands:

Code:

# sysctl -w vm.swappiness=30
# echo 30 >/proc/sys/vm/swappiness

The default when Gentoo boots can also be set in /etc/sysctl.conf:

Code:

# Control how much the kernel should favor swapping out applications (0-100)
vm.swappiness = 30

Some patchsets allow the kernel to auto-tune the swappiness level as it sees fit; they may not keep a user-set value.

...........................................................................................................
I promised to write about this as soon as I figured out the last thing I was curious about myself. If there are other topics I should cover, please send me a PM._________________Former Gentoo user; switched to Kubuntu 7.04 when I got sick of waiting on gcc. Chance of thread necro if you reply now approaching 100%...

Last edited by sapphirecat on Mon Jun 28, 2004 5:06 pm; edited 4 times in total

Thanks, I'm sure this will help many people.
Why Linux seemed to use so much memory is one thing that I never understood when I first started using Linux, it took a long time for me to learn why, so I'm glad someone has taken the trouble to explain it for the people that might be newer to Linux.

So, by getting a 1 GB+ I can enjoy even some better locality performance?

Probably not; the cache has a quickly diminishing performance boost. The only reason Linux lets it get so big is because the memory is available and a marginal gain is still a gain.

Besides, if you look at the buffers/cache line, over half your memory is still truly available. _________________Former Gentoo user; switched to Kubuntu 7.04 when I got sick of waiting on gcc. Chance of thread necro if you reply now approaching 100%...

hm.. perhaps this can be made a sticky or something - since
many new users ask the question about why gentoo/linux/etc
'uses up' most of their ram._________________proud to be a scout and a chronic penguin hugger
Legion of Lore - site

Quite informative, thanks. I figured out how to read meminfo a while ago, but something that I didn't know (not that I really researched it, to be honest ) is what exactly buffers and cache are._________________moo

Excellent !
It is indeed a FAQ, even in french, so I decided to translate your text and post it on the french forum .
I only hope not to have betrayed your mind !

Great!

I can't read French, but that didn't stop me from trying. AFAICT, Leander256 expanded on the swappiness section? I would appreciate having that in English to merge in here. Feel free to PM me about that._________________Former Gentoo user; switched to Kubuntu 7.04 when I got sick of waiting on gcc. Chance of thread necro if you reply now approaching 100%...

AFAICT, Leander256 expanded on the swappiness section? I would appreciate having that in English to merge in here. Feel free to PM me about that.

In fact, Leander256 was afraid a french speaking n00b could not understand the english word "swappiness", nor my poor translation of the word. So I've added a paragraph to explain the concepts even more.

I've tried to translate it in english but please be lenient and feel free to adapt this to a less "frenchie english"

Quote:

When an application needs memory and all the RAM is fully occupied, the kernel has two "relief tanks" at disposal :
it can either reduce the disk cache in the RAM by eliminating the oldest data or it may swap some less used portions (pages) of programs out to the swap partition on disk.
It is not easy to predict which method would be more efficient.
Almost because of the Murphy's Law (AKA "law of the buttered slice of bread" ) which says that the last portion you've just eliminated will at once be required the next second after !
The kernel makes a choice by roughly guessing the effectiveness of the two methods at a given instant.
For instance, it can see that, in the last minutes, all the data in the cache where manipulated by a program but that another program was sleeping in the meantime.
One may rightfully suppose that this situation will not change in the next seconds. So, it would be better to swap out the "sleeping" program and to keep the "alive" datas in the faster RAM.
On the contrary, it can happen that all the programs are very active but that they manipulate almost no data. It would then be more efficient to keep the programs in memory and eliminate those old datas that seem not used anymore.

Before the 2.6 kernels, the user had no possible mean to influence the calculations and there could happen situations where the kernel often made the bad choice. But the consequence of a bad choice is a great activity from the disk which results in a considerable reduction of performances.

Since the 2.6, the user can assign a variable to force on every instant the degree of preference the kernel must use to make his choices. This is the "swappiness" variable (awkwardly translated in french as "permutabilité") that refers to the degree with which the kernel would tend to swap the pages to the disk rather than to reduce the disk cache in RAM.

I've tried to translate it in english but please be lenient and feel free to adapt this to a less "frenchie english"

Alright, revision 2.1 done. I hope you don't mind me taking the liberty of condensing it quite a bit._________________Former Gentoo user; switched to Kubuntu 7.04 when I got sick of waiting on gcc. Chance of thread necro if you reply now approaching 100%...

After a semester of "machine arch" course,
I remember my prof said about it's good to have big size L1, L2, and L3 caches, just so it can greatly redude cache misses (capacity misses in this case)...
The idea is that not needing to touch the swap space as much as possible (since it's on-disk, so the move to transfering data into ram is like... I think it's said 100x ~1000x slower), since kernel 2.6 will make fully use of the RAMs.

so in your case (maybe a lot of mem intensive programs hogging), if you are running out of physical mem, just buy another 512 MB RAM, and once again every bite is worth the money

to zero to prevent things to be swapped to disk but ever time i reboot it comes to 83.

Is it one of the kernels which auto-tunes swappiness? Does the value change after you run it a while (say, while emerging something)?_________________Former Gentoo user; switched to Kubuntu 7.04 when I got sick of waiting on gcc. Chance of thread necro if you reply now approaching 100%...

sory for delay, now my swap file is untouched!
my box stay long time whitout reboot, maybe i forgot it
thanks all for reply, i just noted that my emerge sync now take ages, dunno why.
my kernel is gentoo-dev-sources.

Last edited by revertex on Mon Jun 21, 2004 11:23 pm; edited 1 time in total

It's happen again!
I'm working with some svg files, not so big, then my machine start to use swap, after this swappiness goes to 94!

Code:

cat /proc/sys/vm/swappiness 94

i will take a closer look in my kernel config later, thanks about the auto-tunes swappiness info.
beastmaster, i'm running fluxbox with some "diet" apps, 512 isn't enough?
therefore another 512 is a matter of time (read money)