A Technology Blog

Drive Extender

In my previous post I installed Windows Home Server (WHS) 2011 RC in a VMware virtual machine on ESXi (4.1).

The StableBit DrivePool Add-In is a Drive Extender (DE) replacement for WHS 2011 with the following features:

Combine hard drives into a storage pool (with the exception of the system (OS) drive).

Add and remove drives from the pool.

Create duplicated folders on the storage drive pool that are protected against single drive failure. If a drive fails on which a duplicated file was stored, the contents of that file will remain readable even without the drive.

All your data is stored in standard NTFS files. You can always access your files even if the Windows Home Server completely crashes or this add-in can’t be used.

StableBit DrivePool is in early Beta and has the potential for bugs. Because of the nature of this add-in there is potential for data loss. Only serious testers should consider downloading and installing the current Beta builds.

Installation of the DrivePool Add-In is straight-forward. First download the Add-In from the StableBit site. Next create a User on WHS 2011 (or use your WHS Administrator credentials) and copy the Add-In to the server. It does not matter which WHS share you copy the Add-In to.

On WHS browse to the location that you copied the Add-In to and double-click on it to install. Once installed you will see it under Add-Ins on the WHS 2011 Dashboard:

Once installed I added a new virtual hard drive and formatted it – the logged off and logged on again. Here you can see the DrivePool button in the dashboard now:

In the Hard Dive Pooling tab you can see that I now have two drives that can be used to create a storage pool:

It just took a couple of clicks each to add these drives to the storage pool (none of my default WHS 2011 shares were part of the storage pool though):

Clicking on the Add a folder button gives us the option to use pooled or-non-pooled storage (this only works under the StableBit DrivePool section of the dashboard. Adding a folder under Server Folders and Hard Drives will not give you the option to use pooled storage):

Here you can see my first pooled share:

It may seem that DrivePool is not quite as intuitive as the original DE implementation. WHS 2011 can set aside storage to backup the whole server onto another set of disks and so DrivePool needs to be able to designate pooled and non-pooled storage (and not just add all drives to the pool as per the original DE).

I hope that the default share will be able to be added to the pool in a future release (unless there is something obvious that I am missing).

I’ll play around with adding data and removing pooled drives at a later date.

A number of companies have now announced Drive Extender (DE) replacements for Windows Home server 2011 (Vail). This is certainly good news for WHS which, in my opinion, became a much less valuable proposition without DE. wegotserved has been covering these replacements and in this post I will briefly review the options so far.

Drive Bender is a Windows driver and service that will have a dashboard add-in for configuration. A beta is planned for February 21st. Drive Bender should be available for all (modern) versions of Windows:

seamless expansion and shrinkage of the storage pool – add new drives or partitions

folder level duplication

standard NTFS filesystem structure

64 bit – will support x86 and x64

self balancing when a new drive is added

StableBit DrivePool is a forthcoming add-in for WHS 2011. A Technical Preview should be available in the coming weeks:

seamless expansion and shrinkage of the storage pool (excluding the system drive)

DataCore are looking to pick out storage pooling and data protection features from their existing technologies and package them for WHS and Small Business Server (SBS) 2011. Their intention is to do more than simply replace DE so the least that we can expect is:

drive pooling

data protection

DataCore are looking at other advanced features such as block level snapshots to determine their value to the WHS / SBS user. Initial plans are to distribute through WHS OEMs potentially in Q2.

Update: DataCore intends to ship a Beta in May 2011. DataCore is leveraging an existing product that currently is in production on Server 2008 R2 (this bodes very well for their WHS 2011 Add-in). Pricing should be “reasonable” and the add-in will not be restricted to OEMs.

Certainly all of these options are contenders at this point. After the issues that WHS v1 had with data corruption it is essential that these vendors get it right. Data integrity is a paramount concern. It will be interesting to see how much each of these solutions will cost. $20 for DrivePool sounds quite reasonable to me.

I’m looking forward to seeing what else DataCore might bring to the table, but in my opinion, they will need to look beyond solely distributing through OEMs. OEM solutions have typically not been too expansive in terms of the number of drives that they support and many WHS users and supporters (myself included) have built systems themselves that go beyond four drives. It would be unfortunate for DataCore to limit themselves to OEMs only.

I think I speak for many users when I say we want reliability and performance on standard NTFS drives without breaking the bank. Advanced format drive support should be a given I hope. Only then will an upgrade to WHS 2011 become interesting again …

Otherwise Drive Bender on my Media Center might be a great option too … we’ll have to wait and see!

Microsoft dropped the Drive Extender (DE) functionality from Windows Home Server (Vail) this week, to the surprise and disbelief of the WHS community.

Drive Extender allowed users to dynamically shrink and grow their WHS storage pool and enable folder level duplication to protect important files. In Vail DE promised some great new features – such as real time data duplication, background storage operations, uninterrupted media steaming and more. At this juncture I have to say that I am very disappointed with DE being dropped from Vail and find it hard to see how it will gain much traction in the market without it.

I had recently been pondering what the alternatives to WHS might be and was already aware of a similar technology for Linux called Greyhole (as found in products such as Amahi) and was also looking into Nexenta – a ZFS based server platform.

My plan is to evaluate the basic workings of both in virtual machines before I make any decisions – but I did receive a timely email from the Amahi group today promoting Amahi (and Greyhole) as a WHS (Vail) alternative.

The feature set for Greyhole certainly looks to be good – though there are apparently still some “obscure” bugs to be worked out. This is probably to be expected given that Greyhole is still in Beta. I can’t say that the prospect of trusting my data to Beta software gives me much comfort – but equally WHS had some very bad issues in its early days that took a long time to be resolved. This explains why I was not an early adopter of WHS either.

One of the key tests for me will be to see if the ‘automatic free space balancing across disks’ interrupts other server operations such as streaming media or not? I certainly don’t expect it to, but I have not seen anything documenting how this feature works either.

Another feature that sways me towards Greyhole is that drives can be removed from a Greyhole storage pool and read directly by another (Linux) computer. This was certainly something that I appreciated in WHS (although this was not to be the case in Vail).

After Window 7 I had high hopes for Vail and it really is a shame that Microsoft has thrown in the towel with Drive Extender in Vail. While DE certainly was not perfect in WHS version 1 it was a big step in the right direction for the consumer market as it was a huge part of what made WHS “simple” to administer. I will wait and see what the final version of Vail looks like next year – but at this point I am hardly waiting with baited breath.

I still plan to use WHS to backup Windows machines and will probably run it as a virtual machine again in the future.

Mark Vayman, Program Manager for the Windows Home and Small Business Server Team, has posted a summary of the major new features in Drive Extender for Windows Home Server (WHS) Vail.

Of course not everything is new and some Drive Extender (DE) functionality is carried over from the current version:

Duplication can be turned on/off per folder.

Duplicated folders can survive a single hard drive failure.

Storage pool can be easily expanded using different drive types and various sizes.

The new and improved features are listed below (I have added my own emphasis):

For duplicated folders, data is duplicated in real time to two separate drives – there is no hourly migration pass.

File system level encryption (EFS) and compression are now supported for Drive Extender folders.

File conflicts are gone, duplication works as intended for files in use as it is performed at the block level now.

The remaining amount of data to synchronize/duplicate is reported per storage pool.

All storage operations are executed in the background without blocking other server operations. Specifically, drive removal can be issued without impacting the online state of shares.

Drives in a storage pool can be named with a custom description to enable physical identification of the drive in the server.

Drive serial number and exact connection type is reported for each drive.

Drives which are bigger than 2TB can be added to a storage pool.

iSCSI storage devices can be added to the a storage pool.

The system drive can be excluded from the storage pool.

A new low-level storage check and repair diagnostic operation was added.

All storage operations are performed with very low I/O priority to ensure they don’t interfere with media streaming.A new “folder repair” operation is available which runs chkdsk on the folder’s volume.

To protect against silent storage errors (bit flips, misdirected writes, torn writes), additional information is appended to each 512-byte sector stored on drive. In particular, each sector is protected by a CRC checksum, which enables Drive Extender to detect data read errors, perform realtime error correction and self-healing (up to 2 bit errors per sector if duplication is disabled, and any number of bit errors if duplication is enabled) and report the errors back to the user and application. The overhead for this additional data is roughly 12% of drive space.

Data drives in storage pools can be migrated between servers, and appear as a non-default pool. A non-default pool can be promoted to a default pool if no default pool exists.

Some features though have been lost, the most significant of which (in my opinion) is that a “data drive from a storage pool cannot be read on machine not running the ‘Vail’ server software“. This is explained elsewhere in another thread by Bulat Shelepov, Test Lead (Drive Extender), Windows Home and Small Business Server Team:

Drive Extender v2 is a volume driver that sits under the file system … [and] presents its data as regular NTFS volumes (each share, e.g. Music, Videos, etc. is a separate volume with its own drive letter) … Internally, these NTFS volumes are sliced […] into 1 GB chunks, which are distributed (in multiple copies if duplication is enabled) across multiple physical disks […] DEVolume.sys [currently a Vail only driver] is the only driver […] that can parse this layout and present the aforementioned NTFS volumes to applications. Consequently, any system not running DEVolume.sys is currently unable to retrieve the data from directly connected DEv2 disks – that includes all client Windows OS.

I certainly hope that DEVolume.sys can be packaged for other Windows operating systems (at least for Windows 7 which is a close enough relative to Windows Server 2008 R2).

Beyond this it is also interesting to note the following point:

Internally, the “Vail” software has been tested with up to 16 hard drives and with up to 16 TB of total storage capacity. We’re aware of a number of bugs that occur beyond these limits, so please keep your beta installations under 16 drives and 16 TB total drive space.

This goes someway towards explaining the ten hard drive limit for the Vail Beta, and again this is something that Microsoft are working on, though it appears that the number of disks is less important than the total size of the storage pool. Again, hopefully this issue will be addressed prior to the release of Vail.