vmstat – virtual memory

When called, the grand-daddy of all memory and process related displays, vmstat, continuously runs and posts its information. It takes two arguments:

# vmstat is the interval in seconds between two runs. is the number of repetitions vmstat makes. Here is a sample when we want vmstat to run every five seconds and stop after the tenth run. Every line in the output comes after five seconds and shows the stats at that time.

The output shows a lot about the system resources. Let’s examine them in detail:

procs Shows the number of processes

r Processs waiting to be run. The more the load on the system, the more the number of processes waiting to get CPU cycles to run.

b Uninterruptible sleeping processes, also known as “blocked” processes. These processes are most likely waiting for I/O but could be for something else too.

Sometimes there is another column as well, under heading “w”, which shows the number of processes that can be run but have been swapped out to the swap area.

The numbers under “b” should be close to 0. If the number under “w” is high, you may need more memory.

The next block shows memory metrics:

swpd Amount of virtual memory or swapped memory (in KB)

free Amount of free physical memory (in KB)

buff Amount of memory used as buffers (in KB)

cache Kilobytes of physical memory used as cache

The buffer memory is used to store file metadata such as i-nodes and data from raw block devices. The cache memory is used for file data itself.

The next block shows swap activity:

si Rate at which the memory is swapped back from the disk to the physical RAM (in KB/sec)

so Rate at which the memory is swapped out to the disk from physical RAM (in KB/sec)

The next block slows I/O activity:

bi Rate at which the system sends data to the block devices (in blocks/sec)

bo Rate at which the system reads the data from block devices (in blocks/sec)

The next block shows system related activities:

in Number of interrupts received by the system per second

cs Rate of context switching in the process space (in number/sec)

The final block is probably the most used – the information on CPU load:

us Shows the percentage of CPU spent in user processes. The Oracle processes come in this category.

sy Percentage of CPU used by system processes, such as all root processes

id Percentage of free CPU

wa Percentage spent in “waiting for I/O”

Let’s see how to interpret these values. The first line of the output is an average of all the metrics since the system was restarted. So, ignore that line since it does not show the current status. The other lines show the metrics in real time.

Ideally, the number of processes waiting or blocking (under the “procs” heading) should be 0 or close to 0. If they are high, then the system either does not have enough resources like CPU, memory, or I/O. This information comes useful while diagnosing performance issues.

The data under “swap” indicates if excessive swapping is going on. If that is the case, then you may have inadequate physical memory. You should either reduce the memory demand or increase the physical RAM.

The data under “io” indicates the flow of data to and from the disks. This shows how much disk activity is going on, which does not necessarily indicate some problem. If you see some large number under “proc” and then “b” column (processes being blocked) and high I/O, the issue could be a severe I/O contention.

The most useful information comes under the “cpu” heading. The “id” column shows idle CPU. If you subtract that number from 100, you get how much percent the CPU is busy. Remember the top command described in another installment of this series? That also shows a CPU free% number. The difference is: top shows that free% for each CPU whereas vmstat shows the consolidated view for all CPUs.

The vmstat command also shows the breakdown of CPU usage: how much is used by the Linux system, how much by a user process, and how much on waiting for I/O. From this breakdown you can determine what is contributing to CPU consumption. If system CPU load is high, could there be some root process such as backup running?

The system load should be consistent over a period of time. If the system shows a high number, use the top command to identify the system process consuming CPU.

Usage for Oracle UsersOracle processes (the background processes and server processes) and the user processes (sqlplus, apache, etc.) come under “us”. If this number is high, use top to identify the processes. If the “wa” column shows a high number, it indicates the I/O system is unable to catch up with the amount of reading or writing. This could occasionally shoot up as a result of spikes in heavy updates in the database causing log switch and a subsequent spike in archiving processes. But if it consistently shows a large number, then you may have an I/O bottleneck.

I/O blockages in an Oracle database can cause serious problems. Apart from performance issues, the slow I/O could cause controlfile writes to be slow, which may cause a process to wait to acquire a controlfile enqueue. If the wait is more that 900 seconds, and the waiter is a critical process like LGWR, it brings down the database instance.

If you see a lot of swapping, perhaps the SGA is sized too large to fit in the physical memory. You should either reduce the SGA size or increase the physical memory.