redundant, extensible local data storage

I'm looking for advice on local data storage. My current setup is a Win 7 system with 2x 6TB drives set up as a mirrored dynamic disk. My endless stream of RAW photos is closing in on filling up the disks and in the past this has meant buying two new, larger disks, turning those into a mirrored pair, and copying everything from the old dynamic disk to the new one. Lather, rinse, repeat (and worry about eventually having too much data to fit onto a single disk.)

What I'd like is a system where I can add or swap in new disks and the system will magically take care of absorbing the extra space and providing redundancy in case of disk failure. Is that something that Win 10 storage spaces can do? Would a NAS be a better choice? In terms of speed requirements, my limiting use cases are editing large RAW photos and streaming HD video. Any pointers on where to look or what else to consider would be welcomed!

I'm looking for advice on local data storage. My current setup is a Win 7 system with 2x 6TB drives set up as a mirrored dynamic disk. My endless stream of RAW photos is closing in on filling up the disks and in the past this has meant buying two new, larger disks, turning those into a mirrored pair, and copying everything from the old dynamic disk to the new one. Lather, rinse, repeat (and worry about eventually having too much data to fit onto a single disk.)

What I'd like is a system where I can add or swap in new disks and the system will magically take care of absorbing the extra space and providing redundancy in case of disk failure. Is that something that Win 10 storage spaces can do? Would a NAS be a better choice? In terms of speed requirements, my limiting use cases are editing large RAW photos and streaming HD video. Any pointers on where to look or what else to consider would be welcomed!

My vote would be to build a FreeNAS box. Normally I recommend a pool of mirrors when performance is needed, and RAIDZ2 (basically RAID 6) when storage space is needed. But here it actually might make more sense for you to spend more money on hard drives right out of the gate. If you get 6 disks for storage, you can setup two 3 disk arrays. This guarantees that you can lose 2 disks without data loss, and allows you the possibility of losing up to 4 disks and still not lose data.

If you get 6 disks for storage, you can setup two 3 disk arrays. This guarantees that you can lose 2 disks without data loss, and allows you the possibility of losing up to 4 disks and still not lose data.

That sounds like a tremendous waste of space, buying 6 disks just to get the storage of 2.

But yes, it sounds like it's time for the OP to move to a NAS. At least this way he could start with a pair of drives in a mirror and add more mirrored pairs of drives in the future to add capacity.

Thanks for the suggestions. Any thoughts on how this would perform with Lightroom? Let's suppose I have both my Lightroom PC and the NAS box plugged directly into the router. Would the difference be noticeable?

It might be noticeable, but shouldn't be significant. You are increasing latency and limiting throughput, though Gigabit Ethernet is plenty quick for a single user. As long as you keep the catalog/database/whatever it's called on your local system (preferably on an SSD), it should stay very performant.

I think you're looking at it wrong sorta.You do want a NAS. But no matter how large your workflow is. You probably don't need all 6TB to be superfast active. You need them but you're not going to touch them daily/weekly. So once their active period is over you can move it to the NAS storage. It will be slower like USB 2.0 drive slower (well maybe not that slow) but still fast enough to work with.

So you won't need to change anything in your Main system or add more storage to it physically.You'll do that using a NAS such as Synology. Where if you buy a 4-Bay model you'll be able to expand the array by swaping out smaller drives with larger ones fairly painless when you want to grow.

Also. BACKUP because. Redundancy only means you don't have to go to your backup with one drive failure. If someone hits you with Ransomware. You'll just have a Redundant storage unit of ransomed data.

Though with Lightroom. You'll have to read up on how to move files. I hate lightroom's file handing capabilities.

Thanks again for the comments. Indeed (to Andrewcw's point) I've been thinking that a tiered approach might make sense and the idea of active vs. inactive data sounds very reasonable.

It's true that Lightroom is a bit of a PITA in this regard because all file moves have to be done within LR (worse, many of my older photos are in a separate set of Picasa albums).

Re. Backup, besides the mirrored disks, my strategy has basically been Backblaze. I don't know whether that's sufficient to protect against ransomware. I suppose the ransomware could be quietly encrypting my data while Backblaze backs it up and when the hammer drops I have redundant *and* offsite copies of ransomed data.

I've a pair of 3TB and another pair of 4TB drives in a dumb USB3/eSata enclosure. Do you think I could get reasonable performance for backing things up if I set up each pair in mirroring mode using Storage Space? I know when I tried parity using 4 identical 4TB HDDs in this enclosure it was terrible. They were great though when hooked up directly to mobo in the same configuration.

I've an 8-disk unRAID box too that I've been pretty happy with. Picked it over FreeNAS for simplicity.

^^ Came in to recommend unRAID. It's simpler, but that could be a feature in this use case. Additionally avoiding multi-disk raid means only having to spin up one disk for single-file, single-user reads. You'll be able to saturate Gig-E easy off a single modern SATA drive. I use unRAID for my media, have been for years. Have weathered numerous single drive failures as well as online expansion with ease. There is now support for double parity drives for extra redundancy.

It's a bit quirky, but it does the job. With docker support it's now easy to run a variety of containerized apps off the box as well. Certainly easier than back in the plugin days

I think you're looking at it wrong sorta.You do want a NAS. But no matter how large your workflow is. You probably don't need all 6TB to be superfast active. You need them but you're not going to touch them daily/weekly. So once their active period is over you can move it to the NAS storage. It will be slower like USB 2.0 drive slower (well maybe not that slow) but still fast enough to work with.

But now you need two back up systems, one for your main system, and one for your NAS. You'll also have to manually manage files, shuttling them between active and archive storage. Even a budget modern NAS is capable of fully saturating Gigabit Ethernet (~120MB/s), which is about 3-4x faster than USB 2.0. Plenty fast for the OP's use case. Just have a NAS with external expansion capabilities, and just grow your storage pool as your data increases. Synologies have Synology Hybrid Raid which allows you to mismatch drive manufacturers and capacities, while minimizing wasted space due to mismatched capacities. This lets you build your storage pool over time, as your budget allows and your needs grow. They also have models that support between 9 to 15 drives, so you can keep growing your pool up to I believe 108TB. Some of these models can be expanded externally, so you don't need a huge outlay of cash in the beginning, you can just buy an external expansion enclosure when you need it. Don't know how well QNAP deals with this, as I haven't used their products, but those are the two big brands in prebuilt home NAS appliances.

With a single NAS storing everything, you can setup Backblaze to backup the whole thing automatically without any intervention on your part. I believe Backblaze supports versioning (BTRFS does too, which is one of the available filesystems on higher end Synologies) so you should be safe from ransomware.

So my (recently acquired) understanding is that Backblaze will not backup network drives; not that it's technically infeasible but they fear it opens them up to abuse. The $5/month or whatever I pay for Backblaze is a pretty amazing deal so I'd be sad to lose that.

Any idea how much storage you're consuming, say, every year? That may determine how often you need to upgrade.

You can single consumer or consumer-targeted NAS disks up to 12TB right now, and this is a case where more simple might be better-- upgrade to a 2-disk setup with 10TB or 12TB disks and revisit when you need more space?

The growth rate isn't too high. Less than 1 TB/year I think. So far I've been able to buy bigger drives as needed and a single drive (+ mirror) has been sufficient. If single drive sizes ever flatten out then I'll need a new strategy. With 12TB drives available maybe this isn't an immediate concern. Though it feels a bit wasteful to have to abandon perfectly good drives when they're too small to hold all my stuff, hence the wish for something more easily expanded.

There’s a few threads in Other Hardware about it if you want to dig around, the long story short is that a simple two drive mirror and replacing the disks every few years is almost certainly cheaper and more reliable in the long run.