I've looked through the last few SSD articles, and haven't been able to come up with the answer to this questions:What's the IOMeter profiles used for file server, web server, workstation, and database tests?

Mostly interested because I need to set up an IOMeter run to benchmark a couple of the systems here in the office, and I'm curious how it's profile sits compared to others.

Some of the TR ones were deemed not as relevant anymore and have been removed for a while. I had another thread asking the same thing and no one replied. At one point I actually remade those patterns based on the older descriptions from StorageReview.com, plus adding the patterns that they have added to complete a suite of my own. I forgot where I put them though.

Which patterns are you specifically interested in?

The Model M is not for the faint of heart. You either like them or hate them.

IIRC, the file and web results were part of the original IOMeter package, while the database and workstation patterns came from StorageReview. I was going to summarize them here but, looking at them now, I see an awful lot of variables to detail. Shoot me an email and I can send you a copy of the config file we use. Open that in IOMeter, and you can edit the individual access patterns to see how the I/Os are distributed.

OK, I managed to dig up stuff that I was planning to release to TR forums dated 2009/11! (procrastination at its best)

The database pattern that you are looking for is, as SR.com stated when they updated to Testbench3, pretty arbitrary. I personally think it is pretty lame. It is 100% access (1 row) with 8KB size where 67% is reads and 100% random. Instead of trying attach the files to web file lockers and stuff, I have decided to post my work here. It was back then when I got my shiny new 785G+SB710 and I wanted to compare it to ICH9 and stock MS vs Intel/AMD drivers.

So here's the readme that I wrote up for the zip file package (I have updated the links for reference):

The "test setups" were my own definition for my own testing. If we use the same setups then we can compare our results, or at least that was what I was hoping. Configs #4 and up are my own recreation based on the links. I have re-created the older (and deprecated) patterns, plus I added new patterns (#9-11) that I thought to be useful at the time. Why 2 .icf files? More on that later.

At the time, I did not know if single vs multiple core CPUs would make a difference in the benchmarks, so I created 2 profiles. To spawn as many workers as CPUs, just change the "Default Disk Workers to Spawn" field to "NUMBER_OF_CPUS" (minus quotes). I think what I found was that it made very little, if at all any, difference.

So number of workers... what do people tend to use? I've seen someone say the more the better, 10 as a minimum. You indicate 1 per CPU or CPU core. I got 10 working right now on a quad core CPU and it's barely 10% CPU usage.

...it runs a test with 1 worker for a while, records the numbers, then runs with 2, then 4, then 8, et cetera? I've seen someone say with SSDs you have to wait a while, like 8 hours or overnight, to get a valid measure. Is there any conventional wisdom about how long you should wait with this stuff and if it's different with SSD versus SATA/SAS?

Scrotos wrote:So number of workers... what do people tend to use? I've seen someone say the more the better, 10 as a minimum. You indicate 1 per CPU or CPU core. I got 10 working right now on a quad core CPU and it's barely 10% CPU usage.

I don't know anymore. At the time I thought each worker will be one thread, so I assign 1 worker per CPU core. I think the results turn out to be the same.

Not sure if #workers = #concurrent requests.

The Model M is not for the faint of heart. You either like them or hate them.

Our IOMeter tests run with one worker. They start with one concurrent I/O request and then ramp up from there. The test runs for three minutes for each I/O request tier. The number of workers is separate from the number of concurrent requests.

Before any tests are run, IOMeter creates a single file that spans the entire capacity of the drive, effectively putting it into a used state. We secure erase drives before running IOMeter in the first place, so they all have a fresh-state starting point. There's no need to let the drives sit in this case, since they're freshly wiped and there's no data for the background garbage collection routines to move around.

I wish you'd just post your test file(s) or, if you have described the methodology previously with that info, link to it in your SSD "how we test" section. It'd make it easier for schmoes like me who are interested in doing similar tests but aren't too familiar with the tools. For instance, I'm trying to get some decent numbers on this: viewtopic.php?f=29&t=86144

I less want to become an expert in benchmarking and more would like a relatively easy way to just get numbers and figure out if RAID5 is going to perform better than RAID6 in my scenario, i.e. if the controller is saturated anyway and it don't much matter. Either way, I don't see a reason to not post the config file y'all use for reference in the articles. After all, "Most of the tests and methods we employed are publicly available and reproducible."

The patterns TR used were not customized. Most of them were public info at one point or the other. Some patterns were removed in later distributions of IOMeter so what I did was just searching around and based on SR and Toms (!) older articles I found+reconstructed most if not all of them. I only added a couple in there from the established one. SR actually mentioned that the "workstation" pattern (or was it the database one?) is actually quite useless.

But yes, posting the file would be convenient. Although TR is not in the business of teaching you how to run IOMeter.

The Model M is not for the faint of heart. You either like them or hate them.

Yes, I'm aware of that. Thanks for the reprimand that I thought I'd already addressed. I do find it odd, though, since you seem to have spent quite some time trying to recreate TR's testing methodology yourself. I'd have thought you'd be more like, "yeah, post that biznatch!" rather than "RTFM."

I do wholeheartedly thank you for the work you did in trying to recreate the test patterns that TR uses and especially for digging up work from 3 years prior and making it publicly available! The currently available version of IOMeter has entries like:

I wouldn't even know where to start looking to try and dig up various profiles that are publically available going back 6 years or longer; even with the links you provided it seemed like a ton of reading for something I was hoping would be a quick "run this profile and find your IOps" deal. Heck, you even spent a decent amount of time researching all this and you don't even know how TR gets their concurrent requests numbers.

So again, thank you for having taken the time to get some useful profiles and information out there.

Feel free to shoot me an email if you'd like a copy of our IOMeter test file. The thing is, that file just defines the access patterns (request size, read/write distribution, randomness, etc...). You still have to configure the workers, concurrent I/O scaling, and test length manually.

Scrotos wrote:Thanks for the reprimand that I thought I'd already addressed. I do find it odd, though, since you seem to have spent quite some time trying to recreate TR's testing methodology yourself. I'd have thought you'd be more like, "yeah, post that biznatch!" rather than "RTFM."

Sorry if I came across that way. It was not my intention. I actually think the patterns are the easy part, since they are documented on the web. (see below)

Scrotos wrote:I do wholeheartedly thank you for the work you did in trying to recreate the test patterns that TR uses and especially for digging up work from 3 years prior and making it publicly available! The currently available version of IOMeter has entries like:

I wouldn't even know where to start looking to try and dig up various profiles that are publically available going back 6 years or longer

Basically it seemed like once Intel decided to make IOMeter open source, the distributions stopped including what Intel included before in terms of the profiles. It was unfortunate. Good thing the patterns were relatively easy to recreate. And subsequent analysis by SR debunked some patterns as not that useful anyway.

Scrotos wrote:even with the links you provided it seemed like a ton of reading for something I was hoping would be a quick "run this profile and find your IOps" deal. Heck, you even spent a decent amount of time researching all this and you don't even know how TR gets their concurrent requests numbers.

Dissonance wrote:Feel free to shoot me an email if you'd like a copy of our IOMeter test file. The thing is, that file just defines the access patterns (request size, read/write distribution, randomness, etc...). You still have to configure the workers, concurrent I/O scaling, and test length manually.

All that worker, concurrent I/O, test length stuff, IMO, are the "biznatch" in terms of the setup. The clunky UI does not help things either. There was no documentation and so I just picked my own settings. I can never really compare with TR's numbers (not to mention I have different CPU+mobo). So I only made comparisons based on my own baselines. That was when I realize that I don't really need TR's exact numbers, just the trends that I needed to match.

But yes, if any of us can post a quick "how to use IOMeter" guide, I am all for it. The strength in TR is not just the articles, but the forum community as well. Let's crowdsource this!

The Model M is not for the faint of heart. You either like them or hate them.

Ok, got it, thanks. So I see under Test Setup where to change the time. The # of Outstanding I/Os is the concurrent requests, I assume.

Looking at the CSVs that get generated, I see Queue Depth (is Concurrent I/O Requests is Outstanding I/Os?) and IOps which seems to be the Read IOps and Write IOps combined.

You plot IOps versus Queue Depth? I ask this because I ran the database test using 10 minute length up to 256 depth and got these IOps numbers:

1 54092 66504 65938 665016 661832 658964 6620128 6596256 6431

Ok, so the sucker plateaued after 2 concurrent requests on my workstation (Q6600, 4 GB RAM, XP SP3, SSD is my main hard drive) and I'm fine with that. I'm just trying to get an idea of how the tool works. But what concerns me is that my starting point is a few thousand higher than in the TR test. Is it because I didn't secure erase the entire drive? The sucker only has less than 40 GB free so I figure it's been mostly written to, ya? Or is something else going on here?

I know the test results aren't directly comparable but I would fully expect my results to be lower, not higher. I'm on some old G31/G33 class motherboard. I think it's SATA 2 at best? And using the computer for other stuff while benchmarking.

Maybe Cycling Options? What should that be? It is defaulting to Cycle # Outstanding I/Os. I also noticed I had # of workers to spawn automatically set to # of CPUs. I'll set that to 0 and see what happens.

BTW, am running it on a server now to see if it ramps further and it's been sitting there for quite some time. Oh lordy, it's got a 680 GB file and counting for this test run! I'll pick that up tomorrow.

Good info there. So the first test runs I did made a 10 GB iobw.tst on the SSD and SATA drives on my workstation. Why, I don't know, since I didn't limit the size at all. I'm going to try using this sucker to make some 100 GB files: http://blog.open-e.com/a-few-practical- ... t-iometer/

I'm assuming TR runs on unpartitioned raw drives for their testing so iobw.tst is not a factor in their testing, but since I'm doing it on actual filesystems it's something I have to care about. Some of the blah blah blah I read indicates that the file size should be more than twice the system's RAM so that no filesystem caching will affect the testing. So I'll rerun the server tests and see what I get. Haven't yet delved into the whole number of workers thing or messing with the RAID controller's cache.

Duh.

Dissonance wrote:Before any tests are run, IOMeter creates a single file that spans the entire capacity of the drive, effectively putting it into a used state.

So that's a bit of a bind for me since I'm running on OS drives. How much space do I leave empty, then? 1 GB? Will the OS decide to dump some temp stuff to disk during testing? Ugh. Will revisit.

Last edited by Scrotos on Wed Feb 06, 2013 6:15 pm, edited 1 time in total.

So that program I posted that made iobw.tst files seems to not work with the current version of iometer. I ran a test with 10,000,000 sectors which worked out to be 5 GB. Rerunning with 200,000,000 sectors which should get me the 100 GB I wanted. BTW, when I re-ran the test with a larger number of sectors, it didn't change the size of the iobw.tst file. I had to manually delete it to get a new size created.

So hey, makin' progress. I'm also using outstanding I/Os to be 256. (doesn't matter with this test) Wow, the iops take a dive with a larger test file. I think I have a handle on the test file size so will be working on the workers next. Ok workers ramps up to the limit quicker, but that's about it it seems.

Flying Fox wrote:But yes, if any of us can post a quick "how to use IOMeter" guide, I am all for it. The strength in TR is not just the articles, but the forum community as well. Let's crowdsource this!

I think I have a version of the TR benchmark that has all the settings pre-filled for what tests they run. I've shot off a copy to the bloke in the know to see if it's missing anything. If so, the how-to would be pretty easy!

That fellow is testing other stuff than what TR tests. I am happy with the test patterns for databases that you and TR have. I have a wishlist item for IOMeter but it doesn't look like anyone's maintaining the code anymore.

Once I get the settings fixed up for the TR test in a way it's easy to reproduce, I've got questions on the 4k alignment. Wheeee.

Now the TR one has a 4K alignment set for the 512 byte pattern but nothing else in that profile. The TPU fellow has 4K alignment set for everything. It seems to me that you'd want to set 4K everywhere, wouldn't you? Why only the 512 byte size?

And where does 4K come into play? Vista and Win7 automatically make partitions aligned to 4K boundaries. SSDs like partitions aligned to 4K boundaries. Large drives using the "advanced format" 4K sectors... also like partitions aligned to 4K boundaries? So in both cases XP is a loser unless you make a partition using something 4K aware and then install XP later. So you'd want to align everything to 4K so the results would most closely match Vista/Win7 and OS X?

Someone in the TPU thread got errors on their tests when they included smaller than 4K data sizes with their 4K sectored drives. Does this happen with the TR test, too, but the error doesn't really show up? Or are the drives not 4K sectored so it doesn't matter? Or was that TPU guy having some other issue? I don't have any 4K sectored drives I can test this on to verify.

We run all our tests at least three times and report the median of the results. We've found IOMeter performance can fall off with SSDs after the first couple of runs, so we use five runs for solid-state drives and throw out the first two.

One thing that's odd is TH uses "LBA=full span" for iometer and I don't see that setting anywhere. I'm guessing they leave the sector size at 0 so it creates a test file that spans the entire drive? That's the only thing that makes sense to me.

So I think I have an all-in-one iometer config file that replicates TR's testing exactly. Waiting to hear back if it's ok to post. I don't see why it wouldn't be, but hey it's polite, innit?

I'll make another separate thread with a how-to for testing methodology with that. I'll also throw in some musings for those who want to test systems already in production like servers based on my experiences.