The beginning portion of the output shows metrics such as CPU free and I/O waits as you have seen from the mpstat command.

The next part of the output shows very important metrics for each of the disk devices on the system. Let’s see what these columns mean:

Device The name of the device

tps Number of transfers per second, i.e. number of I/O operations per second. Note: this is just the number of I/O operations; each operation could be huge or small.

Blk_read/s Number of blocks read from this device per second. Blocks are usually of 512 bytes in size. This is a better value of the disk’s utilization.

Blk_wrtn/s Number of blocks written to this device per second

Blk_read Number of blocks read from this device so far. Be careful; this is not what is happening right now. These many blocks have already been read from the device. It’s possible that nothing is being read now. Watch this for some time to see if there is a change.

Blk_wrtn Number of blocks written to the device

In a system with many devices, the output might scroll through several screens—making things a little bit difficult to examine, especially if you are looking for a specific device. You can get the metrics for a specific device only by passing that device as a parameter.

While the above output can be helpful, there is lot of information not readily displayed. For instance, one of the key causes of disk issues is the disk service time, i.e. how fast the disk gets the data to the process that is asking for it. To get that level of metrics, we have to get the “extended” stats on the disk, using the -x option.

rrqm/s The number of read requests merged per second. The disk requests are queued. Whenever possible, the kernel tries to merge several requests to one. This metric measures the merge requests for read transfers.

wrqm/s Similar to reads, this is the number of write requests merged.

r/s The number of read requests per second issued to this device

w/s Likewise, the number of write requests per second

rsec/s The number of sectors read from this device per second

wsec/s The number of sectors written to the device per second

rkB/s Data read per second from this device, in kilobytes per second

wkB/s Data written to this device, in kb/s

avgrq-sz Average size of the read requests, in sectors

avgqu-sz Average length of the request queue for this device

await Average elapsed time (in milliseconds) for the device for I/O requests. This is a sum of service time + waiting time in the queue.

svctm Average service time (in milliseconds) of the device

%util Bandwidth utilization of the device. If this is close to 100 percent, the device is saturated.

Well, that’s a lot of information and may present a challenge as to how to use it effectively. The next section shows how to use the output.

How to Use ItYou can use a combination of the commands to get some meaning information from the output. Remember, disks could be slow in getting the request from the processes. The amount of time the disk takes to get the data from it to the queue is called service time. If you want to find out the disks with the highest service times, you issue:

This shows that the disk sdat has the highest service time (64.05 ms). Why is it so high? There could be many possibilities but three are most likely:

The disk gets a lot of requests so the average service time is high. The disk is being utilized to the maximum possible bandwidth. The disk is inherently slow. Looking at the output we see that reads/sec and writes/sec are 0.00 (almost nothing is happening), so we can rule out #1. The utilization is also 0.00% (the last column), so we can rule out #2. That leaves #3. However, before we draw a conclusion that the disk is inherently slow, we need to observe that disk a little more closely. We can examine that disk alone every 5 seconds for 10 times.

# iostat -x sdat 5 10

If the output shows the same average service time, read rate and utilization, we can conclude that #3 is the most likely factor. If they change, then we can get further clues to understand why the service time is high for this device.

Similarly, you can sort on the read rate column to display the disk under constant read rates.

The information helps you to locate a disk that is “hot”—that is, subject to a lot of reads or writes. If the disk is indeed hot, you should identify the reason for that; perhaps a filesystem defined on the disk is subject to a lot of reading. If that is the case, you should consider striping the filesystem across many disks to distribute the load, minimizing the possibility that one specific disk will be hot.