All my experiences, bugs, hacks, tricks, ... with CentOS and Linux in general.

Thursday, November 29, 2007

How to test and monitor disk performance

In my RAID performance article I've given a lot of performance numbers, but it is indeed a good question on how to exactly measure disk performance. To do this you need 2 tools. One that does the actual benchmark and one that measure the different system parameters so you can know the impact of the benchmark.

For the benchmark tool itself I usually use 3 tools. The first one is good old dd. The problem with dd is that it can only do 1 type of benchmark (sequential read and write), but it can to this to and from many sources and that is its strength.So how do you use dd for a disk benchmark. Well usually like this :

First a word of warning. Using dd like this will mean that any data on /dev/sdb (including filesystems) will be overwritten. The dd command does a write test and the second dd command does a read test. The write test reads from /dev/zero and writes to /dev/sdb, it uses a blocksize of 64k and it reads 102400 blocks. The read test reads from /dev/sdb and writes to /dev/null, it uses the same amount and size of blocks like the read test.So in this case we are testing /dev/sdb, but this can be any blockdevice. Like /dev/md1 (a software RAID device) or /dev/VolGroup00/Logvol01 (a LVM logical volume.The amount of blocks you need to read or write to have a valid is very simple. Make sure you process at least twice the amount of memory you have in your system. Why ? Because the Linux kernel does caching you need to make sure you request more data then the size of the cache to make sure you are actually testing the disk and not the Linux kernel caching.And then the blocksize remains. Well, this depends on the type of usage you will do on the disks or RAID arrays. There are no real fixed rules for this. But generally speaking when you have a lot of small files take a small blocksize (4 to 16k), for standard fileserving and databases take a medium blocksize (64 to 128k) and for large files or applications that do a lot of sequential I/O use a large blocksize (256k and larger).

One final note on dd, since CentOS 5 dd will, when it has finished a test, tell you the speed if obtained. But on older versions of CentOS you will need to calculate this yourself. Another option there is to use ddrescue, which is a tool similar to dd but also provides the bandwidth used in its output. You can find ddrescue for CentOS in Dag's RPMForge repository.

As I said, dd only does sequential I/O but since it does not need a filesystem it is very useful to get a baseline/indication/magnitude/... for the performance of a disk. The next step is to put a filesystem on the device being tested. Then the second tool I use to test performance is iozone. You can also find prebuild rpms for CentOS in Dag's RPMForge repository.

I usually use iozone like this

iozone -e -i0 -i1 -i2 -+n -r 16k -s8g -t1

Iozone has a look of options (that is part why I like it). If you do iozone -h you will get the complete list. I suggest you go over them when you start testing so you know all your options. But the example above shows enough to get started. The -e option adds the time of the flush operation to the results and the -+n disables the retest feature. The different -i options indicate the tests done by iozone, -i0 is the sequential write test (always needed), -i1 is sequential read and -i0 is random read and write. Another interresting test is the mixed read/write (-i8). The 3 remaining options are : -r 16k is the record (or block) size used, -s8g is the size of the data tested (8 GB in this case, remember the 2x RAM rule here) and -t1 is the amount of threads performing the test (in this case 1).

Depepding in the application you are planning to use you try to use iozone in a way that it mimics as close as possible the IO patterns of your real application. This means playing with the -i (the kind of test), the -r (block size) and -t (number of threads) options.

Other nice features of iozone are the autotest. Here iozone can iterate over different tests using different parameters for file size and record size. Also iozone can output its results in a Excell file, so you later make nice graphs of the tests.

The final test that I perform is a bit less scientific, but still a good indication of performance of a disk or a RAID array. The test is a copy of a directory that contains about 1GB of data. This data is stored in different subdirectories (I think it is about 5 levels deep) and all these subdirs contains either a couple of big files and for the rest a lot of small files.This means that a copy of this directory involves a lot of metadata activity, and a mix of sequential I/O (the big files) and something that resembles random or mixed I/O (the small files). And the result is usually a big difference compared to the sequential speeds obtained with dd or iozone.

Now that we have all these benchmarks we still need a tool to monitor the system itself and see what is going on. The tool I always use for this is dstat. It is written and maintained by Dag of the RPMForge repo (and of course it also contains CentOS rpms of dstat). The nice thing about dstat is that is has different monitoring modules (and you can also write your own), its output formatting is really good and it can output the results in CSV files so you can also process the results in a spreadsheet. This is a example on how to use dstat :

All the diferent output are different output modules and they get display in the order you have put them on the commandline. With the (for example) -n -N structure you can specifiy for which devices you would like to see output. In this case the -n refers to network-card statistics and with -N I specified I would like to see the output for eth2 and eth3. A similar systems is used for the disk statistics (-d and -D) and number of interrupts (-i and -I).

To get the list of extra modules available next to the standard onces do this :

2 of my servers and one other person's service has reported that they are SLOW after upgrading to Cent OS5. Mine are 3 times slower. I have posted a bug at:

http://bugs.centos.org/view.php?id=2382

But so far no joy. The problem seems to be those using hardware RAID cards. And,when I test against an identical 4.x server, I get minor differences (like with bonnie++), but the 4.x server is still 3 times faster executing application.

Would you mind taking a peek at the bug posting to see if you can suggest a test? (I am on the mailing list for the bug. I will see anything you post to it.)

While I certainly find utilities like dd (though I really prefer robin millers dt) and iozone good for summary info, I like to look at what's really going on every second as the one big problem with just looking at total run time or average rate is you never really know what was happing during the test. Was the CPU hammered? Was memory starved? Where there too many context switches? And let's not forget the network for remote types of I/O.

In fact I apply that type of logic to monitoring everything in the system which is one of the reasons I wrote collectl - see http://collectl.sourceforge.net/ for lots of examples and if you like what you see download it and fire it up.

China Wholesale has been described as the world’s factory. This phenomenom is typified by the rise ofbusiness. Incredible range of products available with China Wholesalers “Low Price and High Quality” not only reaches directly to their target clients worldwide but also ensures that wholesale from china from China means margins you cannot find elsewhere and buy products wholesaleChina Wholesale will skyroket your profits.

Hey congrats on the new posting come outbtw i love your blog although i have just stumbled upon it =)Love the new pictures you got there! Please come visit my site marketing companies give me any valuable feedbacks.

You may have not intended to do so, but I think you have managed to express the state of mind that a lot of people are in. The sense of wanting to help, but not knowing how or where, is something a lot of us are going through. Please come visit my site Directory San Antonio City when you got time.

You may have not intended to do so, but I think you have managed to express the state of mind that a lot of people are in. The sense of wanting to help, but not knowing how or where, is something a lot of us are going through. Please come visit my site San Antonio Business Services And Classifieds when you got time.

I usually don’t leave comments!!! Trust me! But I liked your blog…especially this post! Would you mind terribly if I put up a backlink from my site to your site? Please come visit my site Divorce Advice give me any valuable feedbacks.

I enjoyed reading your work! GREAT post! I looked around for this… but I found you! Anyway, would you mind if I threw up a backlink from my site? Please come visit my site Document Shredding when you got time.

Hello, What enticed you to post this article. It was extremely interesting, especially since I was searching for info on this subject last Thursday. Please come visit my site Fortworth Business Directory when you got time.

Hello, What enticed you to post this article. It was extremely interesting, especially since I was searching for info on this subject last Thursday. Please come visit my site Business Directory Fortworth when you got time.

. Since I’m new to blogging, these articles are greatly appreciated; very useful and informative blog and every body must visit this blog. Please come visit my site Garland Yellow Page Business Directory when you got time

Do you want to get the most popular sports accessories? Please click our website, we offer many kinds of fashionable Air Jordan Shoes, NFL Jerseys, MBT Shoes, Nike TN and Puma Shoes Men , especially in NFL Jerseys, you will find more than thousand kinds of jerseys in our store. We guarantee the best quality and the best service for every consumer. If you are interest in, please feel free to contact us, we will reply you as soon as possible.

Easy Earning money in online never been this easy and transparent. You would find great tips on how to make that dream amount every Day. Sitting in the home earn around $100 per day. So go ahead and click here for more details and open floodgates to your online income. All the best.Thanks.

This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine.

Be fast to browse the newest styles and lots of affordable outfits and Nike oxygen Max and women's merchandise is not an daily affair, for that reason that in the stress of modern day time girls ordinarily do not possess a complete great offer time and vitality to go buying by means of countless many style malls, and look at special fees will make you exhausted, so which you can acquire a satisfactory goods, call up for to devote lots of time, the show up of on collection buying significantly minimizing the inconvenience of this, now you not just can every one of the sudden go to to lots of goods Nike Air Max Shoes .

If a job is worth doing it is worth doing well.What runescape gold you share here just prove this saying, i like your thought here, rs gold which is special and informative.

This topic is pretty good and the writer's views are buy runescape gold unique and not so hard to understand.I like it very much, and by the cheap rs gold way I want to share some interesting and very useful websites to you:www.rsorder.com! Iam sure you can find out funny things there.

Many people in this life deny their freedom. They sit back in their misery and blame it on their parents, or their childhood, their health, or their financial problems Cheap Minecraft Gift Code, they never once stand up and take responsibility for their own lives and their own happiness Buy Xbox Live Points, of all the wonderful gifts that we've been given, one of the greatest is freedom Ultimate Game Card.

The past is gone and static. Nothing we can do will change it.Runescape Gold, the future is before us and dynamic. Everything we do will affect it rs gold, You laugh at meRunescape Gold for being different , but I laugh at you for being the same.

I believe that we are not real [url=http://www.gw2goldguide.com]GW2 Gold Guide[/url] social workers. We may be doing social work in the [url=http://www.gw2goldguide.com]Buy GW2 Gold Guide[/url] eyes of the people, but we are really contemplatives in the heart of [url=http://www.gw2goldguide.com]Guild Wars 2 Guide[/url] the world.

The best place to learn mandarin Chinese is in China. However, we understand that it isn't always possible to move here to study Chinese language. The next best thing is to study with our experienced teachers in a virtual classroom. Online students enjoy the same excellent way of Chinese Online Courses and custom designed courseware that we provide for our face to face clients.

I was about to say something on this topic. But now i can see that everything on this topic is very amazing and mind blowing, so i have nothing to say here. I am just going through all the topics and being appreciated. Thanks for sharing..........................