I'm building a fileserver for my small office. We do film and multimedia design. Only 3 clients connected. The server is primarily for local access to graphic assets and video files. I'm looking for advice on hardware and software required. Particularly for the RAID. I have the following objectives:

A) merged capacity
I'd like all other systems to access the data as a single mapped network drive that has an initial capacity of 10 TB. So perhaps 5x 2TB drives (plus mirror drives for redundancy).

B) easy way to increase capacity
Thinking long term, I'd like to 'easily' add more drives to the array for a potential two or three fold increase in capacity. So theoretically it could get upto a 30 TB raid array consisting of maybe 15x 2 TB drives of capacity (plus mirror drives for redundancy).

C) maximum fault tolerance
I want at least 1 mirror drive per capacity drive (in laymen's terms). So if I start with 10 TB / 5x 2TB of capacity, I suppose I would need another another 5x 2TB drives to be mirrors. So 10 drives total. But I'd also like potential for even more redundancy; with upto 2 additional mirrors per 'capacity drive' (and to be able to add them to the array anytime with ease).

D) easy way to monitor drive health
I'd like an intuitive interface for managing the raid and monitoring drive health

The other systems accessing this network drive will be running Windows, but also the odd Ubuntu and MacOS system as well.

Are these objectives attainable? What type of RAID setup do you recommend? What hardware will be required?

Also what OS do you think this system should be running? Does it really matter? I'm no network admin - just a long time Windoze user, without much Linux experience. That said, I'm not opposed to a Linux solution if it's easy enough and more practical than a Windows OS for this server. Or maybe something such as Openfiler.

Budget should hit the sweet spot for value and performance (hence my preference to use 2TB drives). The biggest focus is storage; aside from that the system just needs to keep the drives running optimally with perhaps 2 or 3 clients accessing / writing files at any given time. The hardware quote would start with something like 10x 2TB WD Caviar Blacks; about $1900 for the storage + $x for remaining parts.

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
If this question can be reworded to fit the rules in the help center, please edit the question.

1

Too bad Chopper3 is on vacation this week, he does this kind of thing for a living.
–
sysadmin1138♦Dec 28 '10 at 19:02

1

If you're looking at that much storage, why not look at dedicated NAS or SAN hardware?
–
RexDec 28 '10 at 19:06

3 Answers
3

I agree with what @sysadmin1138 said in his answer and voted it up but I have a few comments and my experience I'd like to add.

Prior to doing IT fulltime I ran a small graphics department for 8 years dealing with this sort of stuff. We had 3 artist and close to 50 node cluster of rendering computers creating and processing image sequences, video frames, up to 64 Megapixel per frame. We were a small company working in a niche market so budget was always an issue for us as well.

Keep in mind that a video server for film and multimedia with 3 users can mean many things. It can mean 3 users streaming the same web quality video or it could mean 3 users compositing shots with a dozen or more layers of 4k uncompressed or lossless compressed frame sequences. The type of network and storage requirements are vastly different in these two examples. So it's hard to really give you solid advice but I'll mention what setup we used at my previous job as it's beyond the high end of that spectrum.

Our file server/storage setup was comprised of many external RAID units from Aberdeen attached to one of their servers running Linux. We chose Linux b/c we were comfortable with it and could avoid the costly Window CALs. Hey they add up when you have a sizable rendering farm. But Aberdeen also sells their servers pre-configured with Windows. They also recently started offering some ZFS NAS/SAN storage servers that are running Nexenta which may be worth a look as well. But their hardware comes with a 5 year warranty so I would highly recommend doing something like that over building something yourself. Monitoring wise their DAS units have an Ethernet management port with a web interface for configuring. But they can alert you to any issues via email and snmp.

The Aberdeen DAS units were exposed to the server as multiple RAID5 LUNs at first but we eventually migrated all of them over to RAID6 LUNs and we were using LVM to manage the storage as a pool. Each project got it's own logical volume which would grow as the project grew. This made it easy to archive the projects as they were completed and then return the space back to the pool. The closest thing to this on Windows without going to 3rd party filesystems and management tools like Veritas is sysadmin1138's suggestion of using Windows Dynamic Disks. In our case RAID10 may have been a better option but our budget wouldn't allow us to give up half our capacity so we chose to stick to RAID6. The server was able to saturate the 2 Gbit links it had at the time so I don't think that choice slowed us down that much.

But I started that setup in 2003 with 3TB of storage and they have been growing with it since. Now it's nearly 8 years old and most of the components have been replaced over the years. But it's pretty simple to add additional storage to it and migrate logical volumes off the older units to decommission them. A couple years back they also replaced the file server part of the setup. Their current server has 4Gbit links. Storage wise the last I heard they were a little over 40TB.

A few other thoughts.
I know it's tempting to go with 2TB drives as it gets you where you need to be capacity wise pretty easily but it comes at the expense of fewer spindles which means fewer IOPs and longer rebuild times when a drive dies.

Interesting! Thanks for sharing that unique insight based on your first hand experience. At this point, I'm leaning towards keeping the compositing and rendering duties at each workstation; while this fileserver acts as the location for storing / viewing / saving assets, completed projects, and backup. I do like the idea of something like an LVM to manage the storage as a pool.
–
Mr. SchwabeDec 28 '10 at 23:30

ZFS also manages the storage in a pool in a more fluid way than LVM. Wasn't a option back in 2003 when I set this up. With LVM when you allocate extents to a virtual volume they are committed to that volume. Where with ZFS it's more like thin provisioning where any unused space can be used by any filesystem in the pool. I would have a look at Nexenta as it's not too pricey and brings with it a lot of nice features.
–
3dinfluenceDec 29 '10 at 2:57

I'm not going to give you pricing since that's so highly localized as to be meaningless.

Looking purely at the storage usage patterns you're proposing, the biggest I/O driver I'm seeing is the video and graphic design. Video tends to be very large. Large enough that you might run into network I/O bottlenecks before you run into disk I/O bottlenecks. It should be snappy for your users, and with enough overhead that when someone is committing a 1.5GB video file to the filer people can still save other files.

As for OS and server hardware, considering your background Windows will work just fine here. The server needs to be 64-bit and with a 64-bit RAID card (if you buy new, this shouldn't be a problem). 8GB RAM is the minimum I'd go with.

Now for the storage infrastructure.

Key drivers:

You want to add drives later.

Expansion of initial LUNs would be highly desirable.

Failing that, the ability to add new LUNs to a Dynamic Disk set.

You want one drive letter.

You need mirroring.

You need drive-health monitoring.

Go with a high end RAID card that supports RAID10 and live expansion. Or if costs start to eat you alive when it comes to buying disks, RAID50 and a lot of battery-backed or flash cache. Pay attention to 'Advanced Format' support (4KB sectors vs the previous standard 512B sectors) on your RAID card; if you buy from a big name (Dell/HP/IBM) this shouldn't be an issue, but if you're assembling from parts this will be a big issue.

Once you have your storage infrastructure in place, you'll be relying on the RAID card to present a single huge LUN to the operating system. Since you're 64-bit end to end, you should have no problem with a single 30TB LUN. Adding space should be as easy as importing drives into the RAID config and then Extending the LUN to fill the new space, all of which can be done online. Just keep in mind that a single volume that large will take an age to chkdsk if you ever need to do that.

If you buy from a big name they all have tools/agents that'll monitor disk health, you just have to remember to install and configure them. They usually have a way to send alerts, too. If you're doing it from parts, your RAID card vendor probably has a utility that does much the same kind of thing though may be less featured than the integrated stuff the big vendors distribute.

If you do need to go with multiple LUNs for some reason, keep in mind that Windows allows you to use mount-points. The directory F:\Bigshare\VideoArchive can be its own volume. People map Z: to \\storeserv\bigshare\ and Z:\VideoArchive is that second LUN.

Avoid DFS. Support in non-Windows land is very spotty and can't be relied upon.

RAID50 Considerations

If you have to go there, make sure your drives are enterprise-class. This will increase cost. A 30TB LUN can take a week to rebuild after a drive failure, and you really, really, really do not want a drive failure when you're doing that.

In fact, you would be much better served by making sure no RAID50 set grows past 10TB in size and just go with multiple LUNs; it reduces how long you're out as well as your exposure to a complete data loss event during a rebuild.

Going with multiple LUNs is pretty easy with Windows Dynamic Disks. The type of drive is called a Spanned Drive, and can spread across multiple LUNs of varying sizes. This is how you create a 9.5TB volume on a 32-bit server that can only use up to 2TB LUNs (3x 2TB LUNs and a single 1.5TB LUN). I/O will not be as well balanced as it would with a single 30TB RAID10 LUN, but that's the price you're paying for cheap.

Thanks! These tips clarify the options. I definitely am a little intimidated by managing a single 30 TB volume... that would be a ton of data loss in the event the RAID array fails during a rebuild. Still some more planning to do.
–
Mr. SchwabeDec 28 '10 at 22:18

Up to 30TB with two mirrors per drive? 30TBx3 = 90TB/2TB per drive = 45 drives.

When you start talking about this amount of data (and this number of drives) you also have to think about how you're going to push this amount of data around. Even the initial 10 drives will overrun a single GNIC or RAID card. How about the southbridge on the motherboard? You can chain together enough DAS drives to hit your capacity targets, but it sure will take a long time to do anything with all that data.

If you really need that much mirroring/expansion/reliability I'd recommend an iSCSI SAN. Something like the Dell Equalogic would be a good bet. You'll get a lot further with a bunch of smaller servers/arrays working in tandem, than trying to create one gigantic array.

Additionally:
If you can afford it buy enterprise class drives (15k, SAS, WD or Seagate). Since you're buying the drives over time hopefully the cost will be manageable. Running 24x7 consumer drives will be lucky to last two years. So that would be two drives failing every month at your full 45 drive capacity. You'll end up buying a lot more drives because you aren't going to find 2TB SAS drives, but that's not entirely a bad thing. More drives = better throughput.