sar – System Activity Recorder

From the earlier discussions, one common thread emerges: Getting real time metrics is not the only important thing; the historical trend is equally important.

Furthermore, consider this situation: how many times has someone reported a performance problem, but when you dive in to investigate, everything is back to normal? Performance issues that have occurred in the past are difficult to diagnose without any specific data as of that time. Finally, you will want to examine the performance data over the past few days to decide on some settings or to make adjustments.

The sar utility accomplishes that goal. sar stands for System Activity Recorder, which records the metrics of the key components of the Linux system—CPU, Memory, Disks, Network, etc.—in a special place: the directory /var/log/sa. The data is recorded for each day in a file named sa where is the two digit day of the month. For instance the file sa27 holds the data for the date 27th of that month. This data can be queried by the command sar.

The simplest way to use sar is to use it without any arguments or options. Here is an example:

CPU The CPU identifier; “all” means all the CPUs%user The percentage of CPU used for user processes. Oracle processes come under this category.%nice The %ge of CPU utilization while executing under nice priority%system The %age of CPU executing system processes%iowait The %age of CPU waiting for I/O%idle The %age of CPU idle waiting for work

From the above output, you can see that the system has been well balanced; actually severely under-utilized (as seen from the high degree of %age idle number). Going further through the output we see the following:

This tells a different story: the system was loaded by some user processes between 3:00 and 3:40. Perhaps an expensive query was executing; or perhaps an RMAN job was running, consuming all that CPU. This is where the sar command is useful–it replays the recorded data showing the data as of a certain time, not now. This is exactly what you wanted to accomplish the three objectives outlined in the beginning of this section: getting historical data, finding usage patterns and understanding trends.

If you want to see a specific day’s sar data, merely open sar with that file name, using the -f option as shown below (to open the data for 26th)

# sar -f /var/log/sa/sa26

It can also display data in real time, similar to vmstat or mpstat. To get the data every 5 seconds for 10 times, use:

Did you notice the “all” value under CPU? It means the stats were rolled up for all the CPUs. In a single processor system that is fine; but in multi-processor systems you may want to get the stats for individual CPUs as well as an aggregate one. The -P ALL option accomplishes that.

kbmemfree The free memory available in KB at that timekbmemused The memory used in KB at that time%memused %age of memory usedkbbuffers This %age of memory was used as bufferskbcached This %age of memory was used as cachekbswpfree The free swap space in KB at that timekbswpused The swap space used in KB at that time%swpused The %age of swap used at that timekbswpcad The cached swap in KB at that time

At the very end of the output, you will see the average figure for time period.

You can also get specific memory related stats. The -B option shows the paging related activity.

pgpgin/s The amount of paging into the memory from disk, per secondpgpgout/s The amount of paging out to the disk from memory, per secondfault/s Page faults per secondmajflt/s Major page faults per second

To get a similar output for swapping related activity, you can use the -W option.