top free mem swap... and their relationships

I need a little explanation on the values that show up in top or in free.
Everyone keeps saying you can never have enough memory.
It would be helpful to support this and explain where am I 'loosing' by NOT
adding more RAM.

In the past couple of days, I've experienced some high CPU values running up to 18 at some times.
Normally,. the values are running at the 0.x and sometime inch up to 1.x
So 18 is obviously is not the norm.

What could affect the high not normal loads? What can explain this sudden jump in cpu load?

Well, I have several websites that have forums in them that are very active. Those forums are using no other than MySQL databases, and I imagine they have few unoptimized queries that are keeping the server busy.

On my server, I have P4 3.06Ghz (not a Xeon) with 1GB of RAM and RAID-5 SATA HD with lots of HD space available. I even have 256MB of RAM on the RAID card itself.

To make things run even faster,. we have the Zend Performance Suite installed (just the caching portion of the suite - highly recommended if you can afford it).

But, with all that, we are still running with 1GB of RAM.
I have about 130 websites/addons domains on the server and the MySQL database is always running busy.

The question that I'm asking is,. I have the option to add MORE RAM.
Should I?
The problem so far is that I don't know what value should I look at as the indicator that tells me: "hey... can't you see you are running low on RAM and we are starting to SWAP"?
So,.. if someone can explain me what exactly is the meaning and the relationships of the values available in top, such as: used, free, shrd, active, inactive, cached and how they are
relating to each other.. it would be very helpful.. and I'm sure not just to me.
I have one more question to this, It is obvious that using the SWAP is not a favorable practice if I have enough RAM to use, and that is why I'm very confused when I see the values in the SWAP that says.. both used, free, avg and cached.

If I'll add another 1GB of RAM, what should I see as far as the value changes in the top?

So, you have 1GB of physical RAM in the server (the Mem: total), your server is using most of that (av), and there's 13216KB free. Sounds bad. Well, no, because Linux will use any spare physical RAM to cache frequently used pages in memory, hence the 645260k cache, which is more than 50% of your available memory.

However, your swap file (which is 2GB in size), which is basically a file on a partition called swap, is being used to the tune of 363080k. This means that the OS has swapped out some pages from memory to the disk because at some point it had insufficient free+cache to cope with demand for pages in memory. It hasn't seen fit to swap them back in, because none of your processes have asked for those pages. Page swappng to disk is bad because it takes up disk I/O which is (relatively) slow and also consumes CPU cycles to perform.

On the face of it, the server appears to be relatively OK, but the high swap file usage is something to watch. What you really need is some MRTG graph data so that you can easily see how the cache and swap usage changes over the day. If the swap file usage remains high and the cache size fluctuates and becomes very low at times, then you might benefit from more RAM on the server.

If you were to install more RAM, then you should see your cache size go up, your swap usage drop to something very small, and your load average to improve, though between 1 and 2 for a 2 CPU server is not necessarily that bad, though obviously reasonably loaded.

So, you have 1GB of physical RAM in the server (the Mem: total), your server is using most of that (av), and there's 13216KB free. Sounds bad. Well, no, because Linux will use any spare physical RAM to cache frequently used pages in memory, hence the 645260k cache, which is more than 50% of your available memory.

However, your swap file (which is 2GB in size), which is basically a file on a partition called swap, is being used to the tune of 363080k. This means that the OS has swapped out some pages from memory to the disk because at some point it had insufficient free+cache to cope with demand for pages in memory. It hasn't seen fit to swap them back in, because none of your processes have asked for those pages. Page swappng to disk is bad because it takes up disk I/O which is (relatively) slow and also consumes CPU cycles to perform.

On the face of it, the server appears to be relatively OK, but the high swap file usage is something to watch. What you really need is some MRTG graph data so that you can easily see how the cache and swap usage changes over the day. If the swap file usage remains high and the cache size fluctuates and becomes very low at times, then you might benefit from more RAM on the server.

If you were to install more RAM, then you should see your cache size go up, your swap usage drop to something very small, and your load average to improve, though between 1 and 2 for a 2 CPU server is not necessarily that bad, though obviously reasonably loaded.

That's m take on it

Click to expand...

Hi chirpy,

Ok.. I've studied your input and could we go over few points again?
First,.. when we set the server up, we decided not to use a /swap partition on its own.
I have a RAID-5 for my /home and we decided to just stick it on it if I remember correctly.
At the time it seem like a working solution as we had zero load (well we had this zero load for almost a year, so apparently this was indeed a working solution till now).

Second, looking at this quote:
"your swap file (which is 2GB in size), is being used to the tune of 363080k....It hasn't seen fit to swap them back in,..... Page swappng to disk is bad because it takes up disk I/O which is (relatively) slow and also consumes CPU cycles to perform."

Does the 363080k matter? meaning,.. do they need to be flushed every so often?
Do we care for them or not really as they just take HD space (that is my least of worries as I have over capacity in my RAID-5)?
If there is a way to flush the 363080k and bring it to zero,. I'll be able to track the swapping better and see if there is any swapping done on regular basis to begin with.
If I just look at the value of 363080k, that by itself means that I was 'short' of 365MB or physical RAM for a better performance load. Is that correct? If so,. then this should definatly
be of added value that is realized immediatly.
That is of course true if the swap is indeed running at the 363MB on daily avg usage. If this has been a single instance in some old time processing,.. I wouldn't need to care for this except if I want to cover seriously extreme cases.
Am I correct with my assumptions here?
One more question about the SWAP thingy: If the server is using Swap to the HD, does it overwrite previous data or does it add it to the swap file and only after I 'max' the swap file definition, does it start erasing from it?
I know you referenced it previously in regards to "It hasn't seen fit to swap them back in, because none of your processes have asked for those pages. "
So with regards to that quote, if another swap action is taking place, does it add to the 363080k and now it will become (assuming 40MB of added swap) 400000k? or will it
utilize the 363080k area first, and then do new writes?

AH.. Magic:
"On the face of it, the server appears to be relatively OK, but the high swap file usage is something to watch. What you really need is some MRTG graph data so that you can easily see how the cache and swap usage changes over the day. If the swap file usage remains high and the cache size fluctuates and becomes very low at times, then you might benefit from more RAM on the server."

So,. to better understand the actual utilization of the server's memory, cache and swap, I need teh MRTG graph (a new tool for me to learn and will probably bring more questions, but with better understanding).

So,.. sorry for the shift in the this thread,.. but when I type:
mrtg
I get the following:

mrtg
Usage: mrtg <config-file>

mrtg-2.9.17 is the Multi Router Traffic Grapher.

If you want to know more about this tool, you might want
to read the docs. They came together with mrtg!

I would imagine that having the swapfile on the same partition as the system would likely only introduce a small degradation in performance and nothing significant (could be wrong)

The 363080k doesn't matter. It only really matters if you've run out of RAM and start using the swap file continuously (memory thrashing). The pages in the swapfile will be released either when the process that created them asks for access to them, or the process dies. Otherwise, they will stay in there and any further paging to the swapfile will be cumulative.

The reason that you have pages swapped out to the swap file, is that at some point, a process asked for some memory and there was insufficient available in physical RAM, so the OS paged some old pages (that hadn't been accessed for a long time) out to the swap file and gave the physical RAM over to the process that requested it.

As you say, it may be nothing to worry about. This could typically happen when upcp runs overnight and the backups kick in.

Although you have MRTG installed, it isn't configured. You need to do a search for some predefined graph configurations to use.

I would imagine that having the swapfile on the same partition as the system would likely only introduce a small degradation in performance and nothing significant (could be wrong)

The 363080k doesn't matter. It only really matters if you've run out of RAM and start using the swap file continuously (memory thrashing). The pages in the swapfile will be released either when the process that created them asks for access to them, or the process dies. Otherwise, they will stay in there and any further paging to the swapfile will be cumulative.

The reason that you have pages swapped out to the swap file, is that at some point, a process asked for some memory and there was insufficient available in physical RAM, so the OS paged some old pages (that hadn't been accessed for a long time) out to the swap file and gave the physical RAM over to the process that requested it.

As you say, it may be nothing to worry about. This could typically happen when upcp runs overnight and the backups kick in.

Although you have MRTG installed, it isn't configured. You need to do a search for some predefined graph configurations to use.

Click to expand...

Well.. it seems like the 363080k has grown to 575764K, so aparently it does grow in size and obviously that means there is Usage of the Swap. And having that process (of using the swap) is something that is not very desirable, I'm forking the $$ for additional 2GB (1GB+1GB) of Kingston memory (If I'm lucky,. I'll have 3GB on the server,. but if for whatever reason the server doesn't like the mix of memory vendors (the other two banks have 2x512 of DDR 512 400MHz Elixir ), then I'll just have 2GB of the new Kingston.

The one thing that strikes me as an odd thing is that there are so many good utilities and configuration worth of settings that 'everone' should have.
Why is it that there are no predefined config files that we can use?

There is a finite set of available configurations of servers. While there are obviously various options to run and set them, I don't think it is too much to 'ask' to have an aggregation of:

"If you are running a single P4 with 1GB use this set of configs"
"If you are running a single P4 with 2GB use this set of configs"
"If you are running a single P4 with 3+GB use this set of configs"

"If you are running a single Xeon with 1GB use this set of configs"
"If you are running a single Xeon with 2GB use this set of configs"
"If you are running a single Xeon with 3+GB use this set of configs"

"If you are running a dual Xeon with 1GB use this set of configs"
"If you are running a dual Xeon with 2GB use this set of configs"
"If you are running a dual Xeon with 3+GB use this set of configs"

All that is needed to use for initial install, is only 9 configuarions or add the 1.5GB as
another set of optional settings.
That's it. With that,.. about 80% of the system initial configuration is already optiimized
for a busy load. How many procesess in the Apache.. the open tables in MySQL etc.

I've read many "suggested things to do" to optimize one piece or another, but there are missing pieces about the specifics of configurations especially when using cPanel.
There are some defaults that can be suggested to be used off the top and you shouldn't
need to 'shop around' for scientific papers trying to figure if you need to have
#
MinSpareServers 5
MaxSpareServers 10
StartServers 140
MaxClients 300

Or some other value. As mentioned, we already know what are the possible settings for the physical hardware,.. and yes.. every year and a half they may be edited or more likely add new pre-defined hardware configs and have them optimized.

BUT,. I would advise to add one more step to such "pre-defined" configuration files.
Knowing that if a value of memory attribute like cached or another exceed some value in relation to the physical hardware profile,. you can already suggest the users that if they monitor the server, and specifically point them to look for the specific attributes of the utility they should be using. If they go through a routine check of 10 key elements to monitor a server and they see a 'glitch' of running out of resources, they will have the insight to upgrade the particular element that is showing the High Load values. And.. as mentioned
it is very important to identify what is a "normal" and what is "start monitoring this" and of course "You better add more stuff here now" values.

Just my $0.02 rant.

Thanks for the explanation about the SWAP, and I'll try to get the config thing running with MRTG.