i do quite a bit of FRAPS recordings. the drive that i'm recording to is a RAID0 array of 1TB seagate barracuda 7200.12 drives, which has a sustained write of over 210MB/s. a 60 FPS recording at 1920x1080 calls for somewhere around 120 MB/s, so my RAID0 shouldn't have any problems. however, i am getting pretty severe stuttering after a couple minutes recording, which indicates that the drive isn't keeping up.

does anyone know exactly what's wrong? or is FRAPS recording performance not determined by sequential write?

Edit: Another possibility -- was that 210 MB/sec for your RAID array measured near the beginning of the array, or is it an average or worst case? Even if you're getting 210 MB/sec near the beginning of the array, this will fall off as the array fills up because mechanical hard drive performance decreases by about half as you move to the inner tracks. So if the 210 MB/sec was measured at the beginning of the array, and the array is nearly full, you're probably only getting about 105 MB/sec! (And even if the 210 MB/sec is an average across the entire array, that would mean you're probably getting more like 150 MB/sec towards the end, which is cutting it close if you need at least 120 MB/sec... just a little fragmentation could eat that up easily.)

The years just pass like trains. I wave, but they don't slow down.-- Steven Wilson

Edit: Another possibility -- was that 210 MB/sec for your RAID array measured near the beginning of the array, or is it an average or worst case? Even if you're getting 210 MB/sec near the beginning of the array, this will fall off as the array fills up because mechanical hard drive performance decreases by about half as you move to the inner tracks. So if the 210 MB/sec was measured at the beginning of the array, and the array is nearly full, you're probably only getting about 105 MB/sec! (And even if the 210 MB/sec is an average across the entire array, that would mean you're probably getting more like 150 MB/sec towards the end, which is cutting it close if you need at least 120 MB/sec... just a little fragmentation could eat that up easily.)

Edit: Another possibility -- was that 210 MB/sec for your RAID array measured near the beginning of the array, or is it an average or worst case? Even if you're getting 210 MB/sec near the beginning of the array, this will fall off as the array fills up because mechanical hard drive performance decreases by about half as you move to the inner tracks. So if the 210 MB/sec was measured at the beginning of the array, and the array is nearly full, you're probably only getting about 105 MB/sec! (And even if the 210 MB/sec is an average across the entire array, that would mean you're probably getting more like 150 MB/sec towards the end, which is cutting it close if you need at least 120 MB/sec... just a little fragmentation could eat that up easily.)

just checked a 1920x1200@60 FRAPS avi file. the total bitrate is 1658880 kbit/s, or 202.5 MB/s (assuming my math is right, of course). since 1920x1080 is 90% that of 1920x1200, it works out to be 182.25 MB/s. definitely greater than the 120 MB/s that everybody's been telling me. judging by the looks of it, i'll need faster drives

Ryu Connor wrote:

just brew it! wrote:

Edit: Another possibility -- was that 210 MB/sec for your RAID array measured near the beginning of the array, or is it an average or worst case? Even if you're getting 210 MB/sec near the beginning of the array, this will fall off as the array fills up because mechanical hard drive performance decreases by about half as you move to the inner tracks. So if the 210 MB/sec was measured at the beginning of the array, and the array is nearly full, you're probably only getting about 105 MB/sec! (And even if the 210 MB/sec is an average across the entire array, that would mean you're probably getting more like 150 MB/sec towards the end, which is cutting it close if you need at least 120 MB/sec... just a little fragmentation could eat that up easily.)

so in light of my recent calculations, looks like my array isn't powerful enough, especially given how inconsistent it is. bummer. i wonder if it is worthwhile to get another drive for a triple RAID0, or replace them with something faster. i can get around this issue by using fancycache and using 8GB of RAM as a deferred write buffer (10 seconds defer), but it is not a particularly elegant solution. not to mention, it costs 8 GB of RAM. maybe if i chop it down to 4GB with 5 seconds defer...

the cache is also constantly clearing itself as well. the trick is to find a cache and defer amount that gives the drive enough time to catch up before the cache runs out. all i know is that with the 8GB cache and 10 second defer, the system can record for an hour without experiencing a single stutter. of course, i don't know what will happen if it needs to go longer than that, since i've never recorded that much.

let's do some math:

in 10 seconds, the cache accumulates 180.25 MB/s * 10 s = 1802.5 MB of data. at that point, the data start being written to the array at an absolute low of around 120 MB/s. this means that it takes at most 15 seconds for the array to write the same amount of data, which if sustained, requires the cache to retain an additional 5 seconds worth of data every 10 seconds, which works out to 901.25 MB. if this keeps up, the cache gets filled in... 7+1=8 cycles, or about 80 seconds. so obviously, the cache is insufficient when the array is nearly full.

if we use the average of 175 MB/s, the same setup results in an offset of 0.3 seconds, resulting in an accumulation of 54.075 MB per cycle, which means that the cache gets filled in 118+1=119 cycles, or about 19.83 minutes. again, not entirely sustainable.

theoretically, we can half the cache and half the deferred time. it all boils down if the array likes writing data in 901.25 MB chunks as much as it likes writing 1802.5 MB. we can also keep the same cache size but decrease the deferred time, which theoretically lowers the accumulation amount per cycle. again though, it depends on how well the array handles smaller data chunks. technically after the first large chunk, everything becomes pure sequential writes anyway, so it shouldn't make any difference in that regard.

obviously the easiest fix would be to add another drive, which technically increases its bare minimum performance up to 180 MB/s, assuming the Intel onboard RAID can keep up. i have an overclocked 2600k, so it SHOULD be alright.

theoretically, we can half the cache and half the deferred time. it all boils down if the array likes writing data in 901.25 MB chunks as much as it likes writing 1802.5 MB. we can also keep the same cache size but decrease the deferred time, which theoretically lowers the accumulation amount per cycle.

It also cuts the time it takes to *fill* the cache in half, so you gain nothing.

901 MB vs 1802 MB isn't going to make a difference, since (as you've already surmised) it is all sequential once it starts streaming anyway.

You'd probably be better off setting the defer time much lower, so that the data can start streaming out to the drives as soon as there's a few tens of MB in the cache.

moriz wrote:

obviously the easiest fix would be to add another drive, which technically increases its bare minimum performance up to 180 MB/s, assuming the Intel onboard RAID can keep up. i have an overclocked 2600k, so it SHOULD be alright.

Yeah, I *think* it should be able to keep up.

Also, don't discount the potential effects of fragmentation. If the RAID array is at all fragmented, your sequential transfer rates will go down the toilet.

The years just pass like trains. I wave, but they don't slow down.-- Steven Wilson

i DO gain something though: 4 GB of RAM. granted, i have 16 GB and have no realistic use for it, so i suppose i'll try something like 8GB cache and 1 second deferred. that should results in an average accumulation of only 5.25MB per cycle, which means that at the average write speed of 175 MB/s, we will have 1526.05 seconds, or 25.43 minutes of uninterrupted recording. which is kinda strange, since i've recorded continuously for well over twice that amount and there was no stutter.

or i can just stop being cheap and get myself another drive. :S

as for fragmentation, the array spends most of its time EMPTY. i have split into into two partition, with the larger one occupying the outside tracks and the much smaller one on the inside tracks (1.5TB and 0.5TB respectively). the smaller partition is filled with mostly static data, and the partitioning itself should keep fragmentation to a minimum. then there's defragging every week and all.

morphine wrote:

Apologies is this has already been mentioned, but why not save directly encoded, even if just to a lossless format? That should help ease the disk I/O.

technically, Fraps does encode the files. if we're dealing with true RAW files, as our head brewmaster had already mentioned, requires something around 400 MB/s incompressible sequential write. that's well into RAID0 vertex 3 pro performance, and i don't have the budget for it. if there's a way to compress the files AGAIN while simultaneously recording with Fraps, then i'm all ears.

i DO gain something though: 4 GB of RAM. granted, i have 16 GB and have no realistic use for it, so i suppose i'll try something like 8GB cache and 1 second deferred. that should results in an average accumulation of only 5.25MB per cycle, which means that at the average write speed of 175 MB/s, we will have 1526.05 seconds, or 25.43 minutes of uninterrupted recording. which is kinda strange, since i've recorded continuously for well over twice that amount and there was no stutter.

The disconnect between theory and practice could well be the difference between decimal and binary megabytes. Unless all of the numbers are using the same units, the data rate calculations could be off by several percent in either direction, and when you're close to the edge like this that could make a significant difference in how long it takes to exhaust your available cache.

The years just pass like trains. I wave, but they don't slow down.-- Steven Wilson

Apologies is this has already been mentioned, but why not save directly encoded, even if just to a lossless format? That should help ease the disk I/O.

Fraps does, exactly this. A low-compression lossless codec, specifically designed to compress screen grabs at a high rate of speed with low memory and CPU usage. They use the FourCC code FPS1.

It's possible, or it used to be, to trick Fraps into encoding to a buffer, have another codec reprocess that buffer, and write the re-compressed file, but it's prone to error, not easy to setup, and the gains are questionable.

if there's a way to compress the files AGAIN while simultaneously recording with Fraps, then i'm all ears.

In premise NTFS Compression might fit that bill. There's a chance that it won't bother trying to compress the already compressed video. It's meant to be a highspeed compression and it's heuristics are pretty smart. You'd have to test and see the behavior.

"Welcome back my friends to the show that never ends. We're so glad you could attend. Come inside! Come inside!"

actually, i need something that can compress the files AGAIN before it even gets written to disk. otherwise, there's no benefit, since i'm not looking to save disk space; i have plenty of THAT fortunately.

You may want to look into getting an SSD if it's that big of an issue. My 120GB Corsair Force 3 is considered a pretty slow drive for being SATA III. I'm still getting 451.07 MB/s with the command Ryu gave. Two of these (with MIR) would cost about $300 US, but would be a huge speed increase in RAID0 over your Seagates.

Lenovo W520IBM dx340Nokia Lumia 928Sony a7 with far too many lenses to list or even count

SSDs are unfortunately not a choice, since even two 120GB in RAID0 is too little space. not to mention, the easy fix in this situation is to drop in another 1TB seagate for $50. there's not much point in spending $350ish for SSDs.

what hard drive raid controller are you using? most on board controllers can't keep up with fast sustained writes. i'd say invest in an LSI 9260 raid card and get 4 WD RE4 drives. the enterprise level drives from seagate and western digital will do 80 to 140mbs in a single drive set up. Since you're doing a lot of HD recording i would use a dedicated true raid card, i know a decent one starts at $200 but the on board options suck for raid.