Storage Spaces & Scale-Out File Server Are Two Different Things

In the past few months it’s become clear to me that people are confusing Storage Spaces and Scale-Out File Server (SOFS). They seem to incorrectly think that one requires the other or that the terms are interchangeable. I want to make this clear:

Storage Spaces and Scale-Out File Server are completely different features and do not require each other.

Storage Spaces

The concept of Storage Spaces is simple: you take a JBOD (a bunch of disks with no RAID) and unify them into a single block of management called a Storage Pool. From this pool you create Virtual Disks. Each Virtual Disk can be simple (no fault tolerance), mirrored (2-way or 3-way), or parity (like RAID 5 in concept). The type of Virtual Disk fault tolerance dictates how the slabs (chunks) of each Virtual Disk are spread across the physical disks included in the pool. This is similar to how LUNs are created and protected in a SAN. And yes, a Virtual Disk can be spread across 2, 3+ JBODs.

Storage Spaces can be used as the shared storage of a cluster (note that I did not limit this to a SOFS cluster). For example, 2 or more (check JBOD vendor) servers are connected to a JBOD tray via SAS cables (2 per server with MPIO) instead of connecting the servers to a SAN. Storage Spaces is managed via the Failover Cluster Manager console. Now you have the shared storage requirement of a cluster, such as a Hyper-V cluster or a cluster running the SOFS role.

Yes, the servers in the cluster can be your Hyper-V hosts in a small environment. No, there is no SMB 3.0 or file shares in that configuration. Stop over thinking things – all you need to do is provide shared storage and convert it into CSV that is used as normal by Hyper-V. It is really that simple.

Yes, JBOD + Storage Spaces can be used in a SOFS as the shared storage. In that case, the virtual disks are active on each cluster node, and converted into CSVs. Shares are created on the CSVs, and application servers access the shares via SMB 3.0.

Scale-Out File Server (SOFS)

The SOFS is actually an active/active role that runs on a cluster. The cluster has shared storage between the cluster nodes. Disks are provisioned on the shared storage, made available to each cluster node, added to the cluster, and converted into CSVs. Shares are then created on the CSV and are made active/active on each cluster node via the active/active SOFS cluster role.

SOFS is for application servers only. For example Hyper-V can store the VM files (config, VHD/X, etc) on the SMB 3.0 file shares. SOFS is not for end user shares; instead use virtual file servers that are stored on the SOFS.

Nowhere in this description of a SOFS have I mentioned Storage Spaces. The storage requirement of a SOFS is cluster supported storage. That includes:

SAS SAN

iSCSI SAN

Fibre Channel SAN

FCoE SAN

PCI RAID (like the Dell VRTX)

… and SAS attached shared JBOD + Storage Spaces

Note that I only mentioned Storage Spaces with the JBOD option. Each of the other storage options for a cluster uses hardware RAID and therefore Storage Spaces is unsupported.

Summary

Storage Spaces works with a JBOD to provide a hardware RAID alternative. Storage Spaces on a shared JBOD can be used as cluster storage. This could be a small Hyper-V cluster or it could be a cluster running the active/active SOFS role.

A SOFS is an alternative way of presenting active/active storage to application servers. It requires cluster supported storage, which can be a shared JBOD + Storage Spaces.

So if you’re looking into the “clusterd hardware RAID” path (LSI Syncro CS 9286-8e), how do you acquire storage tiering with SSD/SAS? Since for what I understand, it’s when you’re running Storage Spaces (storage pool -> virtual disk) you will get this functionality and storage spaces on top of hardware RAID = not support. Do you use the RAID-controller’s functionality called CacheCade instead?

Thanks for the article.. There are so many different levels of abstraction these days that it’s getting tough to keep them all straight!

Can I make a CSV that consists of three drives, one in each of three separate 2012R2 servers (for example 3 servers each with a 2tb SATA drive to build out a single CSV that is a protected, shared 2TB disk)? My goal is to have a lab where each of three Hyper-V hosts houses a copy of ALL VMs on a shared CSV, so that I have fault tolerance, while allowing each Hyper-V host share the VM load.

I am envisioning this similar to how Exchange 2010 DAGs work – Each server has a copy of each DB which is all replicated and can be mounted on any node with a copy of it. In my analogy, the VMs would be like the DBs in that there will always be a copy replicated to each node, but any VM can run on any host.

The free edition allows a 128GB CSV to span 2 physical servers. You present that CSV to the Hyper-V cluster as a CSV drive and then put your VM”s on it. The StarWind software keeps the CSV in sync across the 2 nodes via the network.

The paid version allows unlimited CSV sizes across up to 3 physical hosts.

I’m confused about this paragraph: “SOFS is for application servers only. (That I understand.) For example Hyper-V can store the VM files (config, VHD/X, etc) on the SMB 3.0 file shares. SOFS is not for end user shares; instead use virtual file servers that are stored on the SOFS.

Can I store the virtual file servers on the File Server Cluster as a Role?

Can Storage Spaces combine storage attached to more than one server? (Always looking for ways to reduce hardware count — like avoiding external shared JBODs.) If so, it seem like it could allow RAID1 across servers to help provide fault tolerance in small Hyper-V clusters. I wonder how SSD cache and dedupe would fit in. I’m probably dreaming, right?

While this article is very helpful in distinguishing SS from SOFS, I was left vaguely wondering where SOFS provides best value. Sounds like it’s best suited to providing file services fault tolerance for cluster servers — including file servers — in bigger-than-minimum-sized sites. It’s totally out of scope for minimum-sized SMB-style Hyper-V failover clusters. Is that right?

So if we drop back a notch to High Availability, accept lower robustness, and go with Hyper-V Replica Server and non-shared storage (does it make sense to use SS on single servers and add SSD cache?), or even a shared non-fault-tolerant JBOD with boot-from-backup Disaster Recovery…

I thought I had it, but the more I read, the more I get confused. Thanks for the awesome site. What is the best scenario for 2 nodes with about 6TB each to host their Hyper-V environment (file shares, domain controllers, etc.)? Of course the customer only has 1 node in and the other coming after the fiscal year….

Yes. The CSV -ownership- is balanced. If you’re using Storage Spaces then SMB clients will be redirected to the CSV owner after the initial connection to get best throughput. The balancing of CSV ownership ensures the workload is balanced.

Let’s suppose we have 2 servers and 3 DAS trays with JBODs. Half of disks – SSD, other – SATA. Also we have 3 windows server 2012 r2 hyper-v hosts
Is it possible to make two storage spaces (high-speed and low-speed) with file shares and create 3-node hyper-v cluster using this storage spaces?
Will this solution be HA in case of tray or server fault?

So i have JBOD with 48 SAS Drives and 2 Dell R720. I installed windows 2012 R2 on both servers and configure windows failover cluster both nodes. Create Storage Pool with 2 Hot Spare. Now how this storage can be used by other servers as NFS or ISCSI ?

I have implemented Hyper-V clusters before using Dell SANs. But this time round, I’m going to use clustered storage spaces.

My question relates to general use file servers for end-user file shares.

Obviously, I can have SOFS and run VM fileservers no problem via my Hyper-V cluster, but can I have a SOFS and a General use file server configured from my Storage Server Cluster at the same time? so Hyper-V and SQL access the SOFS shares and my end users access the general use file server cluster role?

Hi Aidan
Thanks for your article – well explained.
We have an environment with 2 JBOD enclosures (with some SSD and some normal drives in them). They are identical. They are connected to two storage servers with two SAS controllers in each. When all connected, each storage server sees each JBOD drive twice (due to the dual controllers).

We’d like to create a highly available SOFS from this hardware. Am I right in thinking that I can simply create a big storage pool on each server and Storage spaces will work out that each drive listed twice is the same? Will storage spaces also automatically spread the data intelligently across both JBODs? DataOn tiering is also automatic too?

Why is it that to achieve JBOD redundancy you have to have 3 JBODs?

Finally I assume after setting up some virtual disks on your storage pool you can then promote them to CSVs using SOFS/Failover clustering.

Enable MPIO with LB policy. That’ll sort out your double vision issue.
One big pool will do the trick. Storage Spaces will do the spreading and tiering intelligently (that’s where the SES stuff comes in).
3 JBODs (for 2-way mirroring) is a quorum thing, like in clustering.
Yes, the virtual disks become CSVs.

I’ve been reading a lot of your articles lately and have been impressed with your SOFS mojo.

We will be setting up a two node SOFS to serve out VM storage for a 3 node Hyper-V cluster containing 90 VMs. The SOFS will contain one large CSV. I am looking for best practices for the number of shares to create to hold the vhdx files for the Hyper-V hosts. If I have one share, will all traffic be routed through one SOFS node? Or for better performace should I place my vhdx files on multiple shares?

Best practice from Microsoft is to have 1 share per CSV. There should be at least 1 CSV SOFS node – WS2012R2 CSV sharing in conjunction with SMB redirection optimizes the use of hardware to get the most performance with this configuration.

I don’t have enough information on your SOFS. Let’s say it’s 2 nodes with shared JBOD(s). Then I’d have at least 1 CSV per node, giving you a minimum of 2 CSVs, and therefore 2 shares. Each CSV is owned by 1 node, balanced across both nodes, therefore each node owns 1 CSV. When hosts connect to the shares for the first time, they hit either one randomly. If they hit the non-owner, there is a brief moment with redirected IO across the SOFS cluster network(s) to the owner node. SMB redirects the host to the owner, therefore providing direct connection from the Host->Share->Owner node in the SOFS->CSV on the JBOD(s).

I’ve done a lot of reading and I’ve never seen it pointed out anywhere else that I can build a Hyper-V cluster with only 2 servers and one (or more) JBODs. The cluster I want to build is small-ish (12-15 VMs and only two that do any intense computing or disk I/O, and that only occasionally) but I want to maximize reliability. I thought I had to have at least 4 servers (2 for SOFS, 2 for Hyper-V) and the network h/w to support SMB Multipath between the Hyper-V hosts and the SOFS.

But, if I understand correctly, I could use 2 servers, 3 dual-controller JBODs connected in MPIO to multiple SAS HBAs in each server, teamed NICs and dual PSUs to eliminate any single point of failure. The only way I would ever even have to failover VMs and run on 1 server would be if a motherboard, CPU or RAM died. If that’s correct, that’s exactly what I want!

It’s an SME scneario, and the Windows Server teams don’t have much interest there – they focus on people with 1+ PB of storage in a single unit. JBOD + Storage Spaces is cluster-supported storage. Therefore 2 servers can use it for a cluster. That’s exactly what most of the CiB’s offer: 2 servers + JBOD. And that’s exactly what you’ve described.

Concerning Steve F’s observation of a “mini” cluster there is still a single point of failure, the cluster software itself. While clustering is much, much better nowadays, it is still a level of complexity that isn’t always needed. It sounds like a better solution for a small VM environment would be two identical server with DAS and using replicas. Each server could host half the VMs and house replicas of the corresponding server. Much simpler to setup and manage. Of course, changes would require some downtime but most, especially smaller shops will have windows available for maintenance/patches.

Archives

Archives

About this Blog

This blog serves 2 purposes. Firstly, I want to share information with other IT pros about the technologies we work with and how to solve problems we often face. I've worked with technologies from the desktop to the server, Active Directory, System Center, security and virtualisation.

Secondly, I use my blog as a notebook. There's so much to learn and remember in our jobs that it's impossible to keep up. By blogging, I have a notebook that I can access from anywhere. It has saved my proverbial many times in the past.

Waiver

Anything you do to your IT infrastructure, applications, services, computer or anything else is 100% down to your own responsibility and liability. Aidan Finn bears no responsibility or liability for anything you do. Please independently confirm anything you read on this blog before doing whatever you decide to do.