Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.

Notices

Welcome to LinuxQuestions.org, a friendly and active Linux Community.

You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!

Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.

If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.

Having a problem logging in? Please visit this page to clear all LQ-related cookies.

Introduction to Linux - A Hands on Guide

This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter.
For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. This book contains many real life examples derived from the author's experience as a Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own.

Well considering you have about 144 GB of ram on this server, and over 95% is used, as well as the swap, I would do a 'top' command and sort it by 'Mem' and see which application is using all your ram; I bet this same app is using your swap file like there is no tomorrow.

To see the actual swap, you can fire up "top", press "f" (to add a column) and then press "p" to activate the SWAP column. Press enter and it will take you back to the the normal view, and you can sort by swap usage

Looking at this is tedious to do manually. I think memory in smaps that is "r-x" (executable code) doesn't get swapped out to the swap space, just memory that is "rw-". I'm not certain how top handles something that is swapped out, but not swapped out to the swap space. I think top may total up everything so it may show more swapped out than what is in your swap space. I could be wrong, though.

And bear in mind that unless you run out of swap, pages written there have a tendency to linger even if the page has been swapped back in. My simplified understanding is that they are only cleared when the process ends, or you've filled swap and some other page needs to be swapped out. I'm not certain how a modified swapped in memory page is handled in respect to the page left out in swap. (Adding another swap space and swapping off the full one will show you what is truly needed. I wouldn't do this on a mission critical system, though.)

I'm not certain how top handles something that is swapped out, but not swapped out to the swap space.

Can't happen - since (at least) before the 2.4 kernel series was released. Only dirty (i.e. modified) memory get swapped - and then only if there is demand for memory.
The numbers for swap in smaps should be reliable - else it's a reportable kernel bug. "top" and your script should be o.k.
A while back I wondered if those numbers included swap-cache or not - never did chase that up. Could skew the numbers for (swap) cached pages that had never been physically swapped; especially with current kernels that also compress the swap-cache to reduce the need to physically swap at all. Must put it back on my to-do list ...

Is there some way out that I can figure what Process (or PIDs) are consuming SWAP ?

I have no explanation for top's output. This looks like a dive into the code is in order.

But I go back to my original statement (which may have not been too clear), the free command shows you how much swap has been used (by still active processes), not necessarily what is still swapped out.

And another point, the pages that have gone out to swap have no direct relationship to the process that may be causing the swapping to occur. The system has a terribly complicated "Page Frame Reclaiming Algorithm" that determines what pages will be stolen.

Also, from the size of your cache, your system is no longer stressed.

btw, the "Swap:" entry in /proc/<pid>/smaps is only available in more recent kernels. Sorry it didn't dawn on me to mention that.

Can't happen - since (at least) before the 2.4 kernel series was released. Only dirty (i.e. modified) memory get swapped - and then only if there is demand for memory.

Agreed.

I was using the term "swap" too loosely, as many do. I was referring to "Syncable" pages, which can be reclaimed by PFRA, but do not get written to the swap space. I'm not certain if these "page steals" are represented in any metric anywhere. And I'm really too old, and too lazy to look at code.

I think this might be the bug but I am unable to find all Swap entries in /proc/*/smaps of 0 KB only.

Does this mean you have swap entries in smaps, but they are all zero ?. If so you have the (kernel) support, but no processes have any swap slots allocated - as tommylovell says above, the numbers you are seeing are probably residual in that case. Easy enough to prove, swapoff followed by swapon.NOT that I'm recommending you do that on a prod box.

Quote:

I have 2753 MB of swap used here, but top then f and then p shows 22 GB and 4.2 GB

To see the actual swap, you can fire up "top", press "f" (to add a column) and then press "p" to activate the SWAP column. Press enter and it will take you back to the the normal view, and you can sort by swap usage

That suggestion is not useful. The SWAP column in top is pretty much useless and in any case does not represent use of swap space, nor anything with significant correlation with use of swap space.

I would expect the information in /proc/*/smaps to include complete and accurate detail on the actual use of swap space. I don't know why the OP's results from /proc/*/smaps don't seem to identify the swap use.

The SWAP column in top is pretty much useless and in any case does not represent use of swap space, nor anything with significant correlation with use of swap space.

Good point - we've been here before haven't we ? ....
Just ran strace over "top" - no indication of reference to smaps, so it (top) is still doing incantations with various fields in /proc/<pid>/stat and statm to work out it's idea of swap usage.

interesting enough this came up just the other day with some colleagues and someone pointed out there is a patch to top to fix this. I believe it is now looks at /proc/*/status which on newer kernels has added a VmSwap field. Check out this:

can't say for sure which was the first version to support it but the info above was from a 2.6.32 kernel. I'm trying to figure out if I should add it to collectl process memory display - I'm thinking I should

Thanks Mark, just happened to see this post before I went to bed.
The (kernel) patch appears to maintain swap_pte counters. Wonder how swap cache compression affects this. This is to remind me to go look sometime.