Benchmark Network Speed

Benchmark Disk Speed

Trivial Disk Speed Testing

Often I want to run simple read and write tests of a disk for sanity testing. Performance geeks will object to simple tests as being meaningless, but these are often good enough for quick comparison testing.

The /dev/urandom device is based on SHA1 which has a small, but significant computational expense. If you need random data then generate a source data file in shared memory (the special device mounted here: /dev/shm/). This is a virtual filesystem stored entirely in memory. You couldn't store the test data on a separate drive because then you would be including the read speed of that drive in the test. By putting the file on /dev/shm you test only copying from memory to a drive.

dd if=/dev/urandom of=/dev/shm/random-data.bin bs=104857600 count=1

You cannot use /dev/null as the input file is valid because `dd` looks at this the same as if it were trying to read from a closed file. It immediately quits the test. However, I have found that the test results are about the same when using an input file set to /dev/zero as compared to /dev/shm/random-data.bin. You might think that /dev/zero would short-circuit some part of the read logic in the kernel since it does not have to do any large memory copies, whereas reading from /dev/shm/random-data.bin must transfer a large chunk of memory to the drive. Also, you might wonder if the drive or kernel can somehow compress the data stream since it is all zeros, whereas random data cannot be compressed. No matter what may be happening under the hood, I have not found the results to be much faster than reading from /dev/shm/random-data.bin.

This does the same, but will make the math easier in future tests. This sets the blocksize to 1MB and count to 1 block. It is good to see that `dd` doesn't show any odd behavior here. The speed is about the same.

But the Kernel will cache output data and keep writing after the process is done and has close the file. How do we know the data is really all there? Add 'oflag=sync'. Note that this does slow down the total speed a little bit.

But that shouldn't make a big difference for large blobs of data. But the dataset would have to be larger than the page cache which is less than the physical RAM. Even a 100MB file is plenty small enough to fit. The kernel will cache the entire thing and then sync it to disk at its leisure.

So we still see a drop in speed when using sync even for large blocks. With smaller block sizes and larger block counts we begin to see a penalty -- from 4.0 MB/s down to 1.2 MB/s.

That speed seems close to using one block with a size of 10485760 bytes, so blocksize does not seem to effect the speed very much. But with sync turned on that changes and we can see that `dd` must be syncing the disk much more often. Ouch:

It's debatable whether one should use 'fsync' or 'fdatasync' options. The 'fsync' option also makes sure filesystem metadata is written to disk. That makes more sense if you are testing the whole filesystem speed. The 'fdatasync' option only ensure the file's contents is on the disk. That would be better if you care about testing raw disk performance... But this is all theoretical. In general you won't see a difference and these are primitive tests that you wouldn't want to take to a performance testing debate.

Uh oh...

This shows why primitive testing can be bad. You have to know what you are doing. Look at these terrible results. It turns out that `dd` reads one byte then writes one byte over and over until done.

Things get even worse if you add the 'sync' option because now `dd` will read then write then sync the disk. Ouch. Super slow. It seems that disks are not designed to write one byte at a time and guarantee that the byte was actually written to the disk before going on to the next byte. This is only 1K of data! But it is nice to see that the sync and blocksize options actually do seem to do what say they will -- tune performance.

Streaming big bursts or lots of little discrete chunks

So now we see that performance measurement depends on how we want to use the disk. Do we only care how fast we can write a giant blob of data? Or do we care how fast it can write lots of little blocks of data. If you are recording video then you probably care more about writing giants blobs of data. If you are writing small transactions in a log file (such as a server log) then you probably care more about small block performance.

If you really are just trying to test the raw disk write speed then you should just test a large burst. It really is the operating system's job to worry about how to manage lots of small requests and still get performance.

read speed testing

For these tests I want a 10MB data file, so first I create a fresh one:

When using `dd` for read testing don't forget the 'iflag=direct' option. All files in Linux pass through the Page Cache, so successive testing of read performance on file will represent the speed to read from cache, not disk. That is usually not what people want to see in speed tests, but it has it's place. Note that the data still goes through a buffer since the data goes through a DMA channel in most disk IO, but that buffer is in the userspace, not the kernel, so the 'DIRECT' option for reading disk can actually speed up IO in some applications since you get rid of the kernel overhead.

And then we can see a weird piece of machinery if we use the 'direct' flag wrong. It turns out that 'direct' requires alignment with the device block size (512 on my disk). See what happens if I set a blocksize that doesn't align with the disk block size?

That makes the direct flag a big pain in the ass to use. In fact, it's not really that useful in the real world since you usually don't want to avoid the kernel cache in the first place. So what's the point? Well, it does help with writing benchmark tools :-) And here we can see something even more mysterious.

Run a few times without 'iflag=direct'. Notice that the first run is slow, but subsequent runs are much faster -- the data comes from cache:

We can flush the kernel cache so that subsequent tests are not effected by the previous test, but notice here that they do get faster. What's this? This is the disk cache! I don't know how to suppress that!

Well-known Benchmark Suites

HardInfo This is a GUI tool intended to give an overview of system hardware. It also contains a section with benchmarks. It is simple and quick and makes it easy to compare the performance of one machine to another. If run without an X DISPLAY it will generate an HTML text report describing the system and performance.