New Notification Options

StableBit Scanner 2.5 – Notification

The StableBit Scanner 2.5 features a completely brand new notification system. You can receive notifications via Email, SMS, Speech, Twitter, or you can have them sent to your mobile devices (Android / iOS / Windows Phone / Windows).

Cloud Integration Enhancements

This actually took a lot of work, and is something that’s mostly invisible to the user, but I think that it was important and worth it, as it greatly improves the quality of the product as a whole.

Here’s a summary of what this means:

First of all, thanks to an update to the engine that powers the StableBit Scanner’s unique SMART interpretation system, the StableBit Scanner now knows about more specific information about each drive model. Things such as the maximum operating temperature and drive reliability figures are now available for each drive model.

This lets us do much more intelligent temperature control and customized overheat warnings, depending on the the use case scenario (Desktop vs Server vs Laptop, etc..). Everything is automatically configured for you, but you can tweak the settings if you want to.

The new data also enables us to issue more intelligent SMART warnings. For example, the StableBit Scanner now knows about the maximum load cycle count, per drive model, so it uses that information to determine when to issue a warning.

When available, the warranty period and drive reliability information are now available under Disk Details.

SSD SMART Interpretation Improvements

Since SSDs operate fundamentally differently than hard drives, the typical set of SMART interpretation rules that apply to spinning drives mostly don’t apply to SSDs. But unfortunately, instead of using one unified set of SMART attributes for all SSDs, each SSD controller manufacturer has chosen to use their own proprietary set. To make things worse, they generally refuse to publish their SMART specifications, which makes interpreting SSD SMART data correctly all that much more difficult.

But this is where the StableBit Scanner can really shine. Because our SMART interpretation rules are cloud powered, we can keep them updated with new rules as new SSDs are released, without pushing out software updates. So your SMART data can actually improve over time as our SMART interpretation rules evolve.

I am proud to say that, as of right now, the StableBit Scanner has SMART interpretation rules for every SSD that it has ever seen, and it can only get better from here. This took a lot of effort and I’d like to make a point of it.

I’d also like to thank those who have chosen to submit their SMART data to BitFlock, this helps us improve our SMART interpretation rules and makes our job a little easier.

New UI Themes

I’d categorize this as a nicety, as it doesn’t really improve the core functionality of the product, but it was requested a number of times, so here you go.

StableBit Scanner 3.0

Let’s talk a bit about the future of the StableBit Scanner.

I think that the StableBit Scanner 2.X line turned out nicely and is a worthy followup to StableBit Scanner 1.0 (which ran on the original Windows Home Server). But it’s time to grow the software into something bigger.

StableBit Scanner 3.0 will be the next major release and it will add a fantastic new capability to the core scanning engine, among other features. I don’t want to talk about this yet but I can’t wait to get a BETA of this out to the public, it may just knock your socks off.

StableBit Scanner 3.0 will also feature StableBit Cloud integration. The exact specifics of the StableBit Cloud are still being fleshed out and I’ll talk about it in some detail once there’s a working prototype.

You can also find a list of some of the other things that we’re working on right now on that wiki.

Our Next Product

Coming up next, we’ll introduce a brand new StableBit product called StableBit CloudDrive. Stablebit CloudDrive is a huge project, on the scale of Stablebit DrivePool, and has been in development since late 2013.

I’ll have a blog post ready along with some screenshots, once I have a 1.0 public BETA.

Hello, I’m Christopher, and I’m the Director of Customer Relations here at Covecube Inc. For those that may not recognize me, I have been very active in the Windows Home Server community, where I usually go by the username of “Drashna”. I have even been awarded the Microsoft MVP Award for Windows Home Server for the tech support I’ve provided in the forums and how-to guides that I’ve written for Windows Home Server.

We tend to get a lot of questions about the StableBit Scanner, what it does and some of the values that it presents. So let me try to answer some of those questions here, and explain a bit more about what the StableBit Scanner does, and why it’s a great utility for maintaining the health of your disks.

I will apologize now for for the amount of text here. There is a lot of information that I want to cover, and I don’t want to skim over any of it. So if you will bear with me, let’s cover exactly what the StableBit Scanner does, and why you should install it.

S.M.A.R.T. Data

First, lets talk about the SMART data that the StableBit Scanner is able to pull from the disks. This data is pretty much universally accessible on any drive you can buy, whether it’s a “spinning” hard drive, or if it is a Solid State Drive. Most of the information is pretty standard, but there are some more device specific values depending on the manufacturer of the device. And there are plenty of utilities out there to read the SMART data from your disk. For the most part, they all read the SMART data from the disks and interpret that data in a meaningful way for users. Some just show the raw output and let you know if the values are outside of manufacturer specification.

Let’s talk about some of these SMART values and what they mean for your system. It’s always a good idea to know what’s going on.

“Reallocated Sector Count” and “Reallocation Event Count” is probably the value that you will see increasing most often. What this means is that the disk has detected an issue with a bad section of the disk, and has reallocated the sectors to a special reserved (spare) area on the disk. This happens automatically, and prevents the disk from using these spots in the future.
This is normal, and typical on a HDD and one or two appearing once in a while isn’t necessarily a bad sign. However, if you see this value rapidly increase on a disk, or you have a lot of them, then there may be damage to the physical medium of the drive and you may want to replace it immediately.
Though, as this value increases, the performance of the disk may be adversely affected. The remapped data will be at another location on the drive, causing the read speed to be decreased due to “seek time” for the new location. And the more Reallocated sectors you see, the more that this will happen. So if performance is very important, it may be worth replacing the drive sooner rather than later.

“Spin Retry Count” is a value only found on HDDs, obviously. It shows the number of times that the drive has failed to spin up to full speed and had to retry to spin it up. This indicates a serious mechanical failure of the platters. There are a number of possible causes, but none are good. It means that you should remove the data from the drive immediately and replace the disk.

“Current Pending Sector Count” and “Uncorrectable Sector Count” - These two values tend to go hand in hand. These means that the disk has encountered issues reading from the drive. In fact, if you force a surface scan at this point, you may end up with the same number (or more) sectors as indicated by this value. The drive will attempt to write to these sectors eventually, and when that happens, it’s either able to and clears this value for that sector or it fails and forces the disk to remap the sector. By “remap”, I mean that this will trigger a “Reallocated Sector Count” increase. This all happens automatically in the course of normal usage. Things like a full format, writing zeros to the disk, or utilities such as SpinRite try to force this process to happen quicker.

“Load Cycle Count” – This is a value that we get asked about a lot and one that can rapidly increase. Specifically, this is the number of head parking cycles that the drive has performed. Parking the heads is a normal process of the drive, and helps prevent accidentally damage to the drive. This occurs when the drive idles. Depending on how this is configured on the drive, and how active the drive is, this can grow very slowly or can increase by 100 or more in a single hour. Western Digital Green drives are particularly notorious for being poorly configured and rapidly increasing the count.So this is a value that should be taken with a grain of salt. Watch it yourself, and if only slowly increases, then you may be able to trust that it’s accurate. And in that case, it may be a good indicator of age and usage. However, this value doesn’t necessarily indicate an issue. It’s much like the “Power on hours” or similar statistical information.

This isn’t a comprehensive list, by any means. But these are some of the most common SMART warnings you will see. And definitely, some of the more important values to know.

SMART data can be a good indicator of mechanical problems, however, it is reactive technology, for the most part. It’s designed to predict immediate failure, and it can’t predict into the future the exact point in time at which the drive will fail, and it’s not designed to. It’s akin to klaxons on a ship, letting you know that something is wrong, and to scramble to fix the issue.

That brings us to the next subject.

Surface Scanning

Now for the “blood and guts” of what the StableBit Scanner does.

By default, the StableBit Scanner is configured to do a surface scan of the disks in the system. What do I mean by a “surface scan”? The StableBit Scanner does a sector by sector scan of the entire disk, ensuring that each and every sector on the drive is readable. And when it finds sectors that are not readable, it flags them and keeps on scanning the rest of the disk.

Now, why is this important? Because after time, the “bits” on the disk may degrade. Over time, if the data is not accessed at all, it can lose it’s stored state and this is what is referred to as “bit rot” usually (different from “random bit flips”). By reading this, it gives the drive’s onboard diagnostics tools the opportunity to repair the section, or just remap it if it needs to, before becoming an issue. This process is called “Data Scrubbing“, and helps your disk identify potential problems before they affect your data. This can be noticed by changes in the SMART Data values on the drive (such as the lowering of the uncorrectable sector count, or an increase in the Reallocated Sector Count).

Wear and Tear

Though, there is a good question that has been raised to us at least a couple of times: Does this surface scan put additional strain on your disks?

For Solid State Drives? Absolutely not. They are designed to be read from many times without any degradation of the drives.

For conventional hard drives? That’s not as straight forward. But basically, any time the drive reads or write data, there is a chance of damage occurring. However, modern drives are very, very good at preventing this from happening.

The other concern here would be wear on the mechanical parts of the drive, the parts that spin the patters, and the parts that move the read/write heads. By default, the StableBit Scanner is configured to do this intensive surface scan every 30 days. What does this mean for the disk? That it’s reading the entire surface of the drive, the full capacity of the disk, once a month. That’s a good amount of work for the drive, and that will happen often.

Well, what if I scan a 3TB drive once a month? That’s about 35TB read in a year. Once a week? That’s about 140TBs read in a year. Okay, that’s a lot of reads over a years time. However, how does that compare to normal usage? Well, do you backup the drives? If so, the entire contents of the disk are read, or every sector is read, depending on the backup utility. What about Windows Search? Or Previous Versions? Or how about streaming from the disks? And how often does this happen? Well, that really depends on your usage.

And to get some perspective here: I have several 3TB drives in my system, that are getting close to 2 years old. I move data around a lot. So what do my drives look like? Well, most report in the ballpark of 50-100PBs of reads and writes. That’s PB (petabytes). Each Petabyte is about 1000 TBs. So thats 50,000-100,000 TBs of reads. If I scan once a month? That’s not even 1% of the total reads from the disk. And while I may not be a typical user in a lot of ways, it should give you a good idea how little of an impact these surface scans have on your disks. And disk are designed to last years, even under heavy usage.

Damaged Sectors

Now what are these damaged sectors that the StableBit Scanner finds and what does it mean to you?

Damaged sectors are bad sectors on the disk that the Surface Scan has issues reading. It means that… well, that it is likely damaged, and during normal operation, you may get an error accessing affected files (or even experience file system errors). These damaged sectors are the same ones that are identified by the “/r” switch on the CHKDSK utility.

Now, you may be asking yourself, why run StableBit Scanner and let it recover that data instead of CHKDSK? Well, you should and shouldn’t. It depends on what you want to do.

CHKDSK does a “best effort” to recover the data. It attempts to read, and then move the data. However, once it determines that it can’t recover it, it reallocates the bad sector and makes that data unrecoverable. And depending, it could potentially corrupt the data, or even lose sections of it.

StableBit Scanner cares about recovering data firs and foremost. Once it’s identified damaged sectors, you can run a “file scan” which attempts to figure out what was damaged on the system and what files are affected. Then it lets you attempt to recover that data. In fact, StableBit Scanner uses 20 different “head placement profiles” to attempt to read the data. This is a lot more aggressive that the CHKDSK utility’s attempt to read the data.

StableBit Scanner does not repair this damage on the disk. If it fails to read the files and cannot recover that data, you can still run a data recovery utility to attempt to recover that data as well.

Again, if you run CHKDSK with the “/r” flag, it fixes the sector by reallocating it, meaning that you lose the ability to ever recover this affected data. This is because the data has been overwritten or the location remapped. So the data is no longer available for recovery.

And to re-emphasize here: StableBit Scanner does not fix damaged sectors. We are more concerned with recovering your data, than about repairing the damage here. The disk will eventually take care of this, or you can force it by using the “/R” flag for CHKDSK.

Conclusion

All in all, the StableBit Scanner is a great tool to inspect and maintain the health of your disks, and the data that is on them.

New Themes

People have asked us, when is the StableBit Scanner is going to get a visual refresh to get a more modern look? Well, here you go.

The new version has new “Aero Glass” style themes that match the style of Windows 7:

Aero Glass Themes

It has new “Modern UI” themes that match the style of Windows 8:

Flat Themes

It even has a touch friendly theme:

Touch Theme

Cloud Enabled Drive Reliability Information

Drive Reliability

Starting with this build, the StableBit Scanner will now show drive reliability information that has been published by the manufacturer. You can find this new information in the Drive Details window under the new Reliability section.

In the latest version of BitFlock, the back-end service that powers the StableBit Scanner’s powerful SMART interpretation engine, in addition to providing the usual SMART interpretation data, we now provide drive reliability information for each specific drive model.

The new types of information provided are:

The Warranty Period

Not all drive models have a published warranty period, but for the ones that do, you will see the warranty period shown in the new Reliability section. Sometimes warranty periods differ from country to country. The warranty periods that the StableBit Scanner displays are based on information published for drives sold in the US.

Mean Time Between Failures (MTBF / MTTF)

Theoretically speaking, the MTBF (sometimes referred to as the MTTF) is the average amount of time that is expected to pass before a particular model drive is expected fail. When this number is calculated, the manufacturer assumes that the drive is running at an optimal temperature (typically around 40 degrees celsius) with a typical workload (or duty cycle) for that model. The expected duty cycle is typically based on the type of drive. For example, an enterprise level drive is typically expected to have a 24×7 duty cycle (also known as 8760 hours per year).

Unfortunately, knowing the MTBF is of limited value for traditional spinning hard drives. Unlike SSDs, spinning hard drives don’t have anything that is “used up” and predicting their failure rates using MTBF numbers is really of limited value. The hard drive manufacturers know this and that’s why most newer drives now report their failure rates using the Annualized Failure Rate (AFR), which is a more useful metric. I’ll talk about AFR in a bit.

Unlike spinning hard drives, knowing the MTBF for SSDs can be more useful. Due to the nature of the technology that powers all SSDs, they all have a finite amount of data that can be written to them,. As as result, a finite amount of time can be calculated until expected drive failure. In addition, SSD manufacturers typically publish drive Endurance numbers which are even more useful for determining the expected lifetime of a SSD drive, and I’ll talk about those shortly.

Annualized Failure Rates (AFR)

Given the limited value of MTBF as it relates to spinning hard drives most newer drives publish their expected failure rates in terms of an AFR. In short, the AFR is the percentage of drives of a particular model that are expected to fail every year.

The StableBit Scanner is able to calculate the AFR for a drive model if there is a published MTBF. The calculation is done assuming a 24×7 duty cycle. It will be noted in the UI if the AFR is coming from a published specification or is calculated from the MTBF.

Component Design Life (CDL)

This is an interesting metric that is rarely published but is useful to know when it is. It essentially tells you how long the drive manufacturer expects the drive’s components to last. This is in contrast to the warranty period, which is typically shorter.

If there is a published CDL for your drive model it will be shown in the Reliability section.

Endurance

Drive endurance generally relates to SSDs exclusively. Because of the technology that SSDs employ, they have a finite amount of data that can be written to them over their entire lifetime.

SSD manufacturers typically express drive endurance in the amount of data that can be written to the SSD per day and the amount of data that can be written over the total lifetime of the SSD.

Drive Reliability

In the above screenshot you can see that the selected SSD is rated for 37.3 GB of writes per day or 66.5 TB of writes over 5 years.

As a technical sidenote, the StableBit Scanner reports all byte measurements in binary (see http://en.wikipedia.org/wiki/Binary_prefix) and not decimal. Manufacturer published data is converted to binary where applicable.

Maximum Operating Temperature

SMART Temperature

In previous versions of the StableBit Scanner, a drive overheating warning was issued when the drive’s temperature exceeded a static temperature threshold that was specified in the Scanner’s settings. You were also able to override the maximum temperature of each drive in “Disk Settings”. Starting with this build the drive overheat warning behaves more intelligently.

In the new build, the maximum operating temperature of each drive in the system is retrieved independently in the following way:

Determining the Maximum Temperature

In general:

If there is a maximum temperature value specified in disk settings, then that value is used.

If the drive’s firmware publishes the maximum operating temperature via SCT, then that value is used.

Alternatively the StableBit Scanner uses the data from BitFlock in order to retrieve the maximum operating temperature for a particular drive model, as it is published in the manufacturer provided data sheets.

If the above methods fail, then the StableBit Scanner uses the static maximum temperature defined in Scanner Settings, as it did in previous versions.

The source of the maximum temperature is shown in the SMART window, under the “Temperature” attribute.

Temperature Source

But simply getting a warning when the drive already exceeded the maximum operating temperature may not be ideal. You may want to receive an overheating warning when the drive’s temperature starts to approach the maximum operating temperature so that you can take corrective actions in order to prevent the drive from exceeding the maximum operating temperature.

Scanner Settings – Heat

To fulfill this need the StableBit Scanner now allows you to specify a temperature “warning threshold”. In other words, if a drive’s temperature approaches its maximum operating temperature limit within the specified number of degrees, then an overheating warning will be issued.

The quick settings profiles will set up an appropriate temperature warning threshold for you.

Quick Settings – Server

Server

Static maximum temperature: 44 C

Warning threshold: 15 C

All Other Profiles

Static maximum temperature: 54 C

Warning threshold: off

Considering that servers are meant to be on 24×7 and that the typical maximum operating temperature of a spinning drive is typically 55 C to 60 C, and that the optimum temperature should be around 40 C to 45 C, a 15 degree warning threshold seems to be appropriate.

SSDs are sensitive to high temperatures as well and typically have a maximum operating temperature of around 70 C. Some newer models will even slow themselves down in order to prevent themselves from overheating.

Load Cycle Counts

SMART – Load Cycles

Starting with this build the StableBit Scanner now retrieves the maximum load cycle count that each drive model is rated for, according to the manufacturer’s published data sheets. If your drive exceeds this limit then a warning is shown. Previously a warning was shown if the drive exceeded 300,000 load cycles, which is a common figure but does vary greatly among different drive models.

I’ve gotten asked a number of times about the severity of this warning and whether it indicates impending drive failure. My experience is that it doesn’t, but it’s still useful to know when the drive is operating past its designed tolerances and is potentially in danger of developing problems in the future.

Cloud Data Quality

This build of the StableBit Scanner comes with a number of new features that rely heavily on the quality of the data provided by the BitFlock cloud. In the next few months BitFlock is getting a major update in order to improve the number of drives that it recognizes and has metadata for. I’m expecting to see 100% drive model coverage for all of our StableBit Scanner customers, so if you don’t see metadata shown for your particular drive model today, the data is being updated.

Right now we have full coverage of all Western Digital and Seagate drives that any StableBit Scanner installation (with Cloud Features enabled) has ever seen, while other manufacturers may be a bit more spotty. SSDs are also a top priority so the quality of SSD SMART data will improve across the board as well.

Unfortunately some hard drive models don’t have any publicly published specifications for them. In particular, it’s difficult to find any meaningful specifications for drives that are sold in external enclosures. So some drive models will end up having little to no additional drive reliability information available about them. But overall, these seem to be few and far between.

Release Finals

Coming up next for the StableBit Scanner is the 2.5.0 release final build. StableBit DrivePool 1.3.6 will also be released as a final and the StableBit Scanner 1.0.6 will be as well.

In this post I’d like to make a quick recap of what’s new in this build since the last release final (2.0.0.420).

Bug Fixes

It has been quite a while since the StableBit DrivePool 2.0 release final and this build has accumulated quite a few bug fixes since then. Many issues were addressed in this release including UI glitches, service issues and file system issues.

Here are some of the highlights:

User Interface:

Minimize / Maximize buttons were added to the horizontal UI.

Tasks such as balancing and background duplication can now be boosted in I/O priority or completely aborted from the UI.

There is a new Troubleshooting menu under the settings menu which allows you to collect a boot time file system log, file system tracing data or to reset StableBit DrivePool’s settings to a clean install state (with your duplication settings and pooled data not affected).

You can now send a test email message after entering an email address for notifications.

Disk tooltips were not showing up sometimes.

The disks list was sometimes redrawing in a “glitchy” way causing an annoying flicker.

The disk index (or number) is now shown in the disk tooltip.

Performance statistics will now update in 2 second intervals (instead of every second) in order to make them more readable.

Pool performance statistics were being reported for all pools in aggregate instead of for each pool individually.

When the last disk that is part of a pool goes missing or is disconnected, the pool’s settings will be kept for 365 days just in case that pool comes back. Previously, that pool’s settings would have been reset.

Balancing plugins are now given 30 seconds to make their balancing calculations or else they will be forcefully aborted. This prevents a balancing plugin from stalling a balancing pass.

The pool will always be measured first before performing other tasks, such as the background consistency check. This change was made mainly to thwart user confusion over incomplete pool measurements showing up in the UI.

Improvements were made to the disk emptying algorithm when balancing.

Disk removal no longer requires a remeasure pass after it completes.

File System:

The virtual disk enumerator now registers as a storage controller making our virtual disk be more compatible with 3rd party software that query for its controller.

Reparse points can now be created on the pool. This adds support for symbolic links, junction points and mount points on the pool. This one took a lot of effort to get done.

Fixed a consistent system crash on x86 systems triggered by heavy I/O.

Byte range locks were not following the same behavior as NTFS leading to issues with all kinds of 3rd party software.

On drive removal, empty “PoolPart…” files were being created needlessly.

A file size tracking inconsistency was fixed that was causing the pool measurement to drift.

File Placement

File placement is a new feature of StableBit DrivePool 2.1 and it gives you the ability to specify which disks will be used to store files in one or more folders on the pool. Further down I’ll show you some examples of how I’m using this feature to organize my pools in a way that wasn’t possible before.

Balancing…

File placement is accessible from the Pool Options menu under Balancing…

File Placement

This new feature adds the flexibility of per disk file organization with the convenience of drive pooling. It’s designed to give you the best of both worlds.

I’ve already talked extensively about the file placement UI and how it works in my previous blog posts, so I’m not going to repeat that. If you’d like you can refer to those blog posts here:

Instead, in this post I’m going to give you some practical examples of how I’m using file placement myself in order to optimize my virtual machine usage.

File Placement and SSDs

First, I’m going to show you how I’m using file placement combined with SSDs in order to seamlessly optimize the performance of one or two VMs on the pool.

In one of my machines I have a pool with a small SSD drive and a few larger spinning disks. I have a few virtual machines that I use, but some of them I use more often than others and I would like to place those frequently used VMs on the SSD drive for better performance. I started out by placing 2 virtual machines on the SSD drive using file placement.

Example

This gave me a tremendous performance boost whenever I used those VMs and it worked out great.

But as my usage changed, and the virtual machines grew, I decided to get a second SSD for my virtual machines and to spread them out over both SSDs. All I had to do is add the new SSD to the pool and designate my virtual machine folders to be placed on that disk as well.

Example

This approach allowed me to create a type of “sub-pool” that can be easily expanded without manually moving files around from disk to disk.

In the future, I can even designate a separate SSD for each VM, all on the same pool.

Example

The great thing about doing this with StableBit DrivePool and file placement is that the path to the VMs doesn’t change, regardless of where they’re actually stored. My virtualization software (WMWare Workstation) always sees my virtual machines on the same pool drive letter and I don’t have to worry about managing multiple pools or dealing with OS licensing issues (that tend to crop up when moving VMs around). All I do is flip a few check boxes in StableBit DrivePool and that’s it.

File Placement and Disk Throughput

I’ll give you another example that doesn’t involve SSDs.

On another system that’s mostly dedicated to running virtual machines, putting all of the VMs on SSDs is cost prohibitively expensive. So instead I’ve opted to use file placement to group VMs that I tend to use at the same time to be placed on separate spinning disks. This allows me to power on and use multiple virtual machines at the same time without oversaturating the disk I/O.

Example

With this setup, I can use virtual machines 1, 2, 3 and 4 at the same time, without incurring a huge disk I/O penalty. The same goes for virtual machines 5, 6, 7 and 8.

In the future, I can add more spinning disks to the pool and reorganize my VMs to let me run more of them at the same time without incurring a disk I/O penalty.

Again, doing this with StableBit DrivePool’s file placement is a breeze and allows me to keep all of my virtual machines at the same path while at the same time gives me the flexibility of future expansion and lets me choose which disks should be used to store each VM.

Coming up Next

Now that the 2.1 release final of StableBit DrivePool is out, StableBit Scanner 2.5.0 is on its way to a release final as well. Coming up within a week the next public BETA of StableBit Scanner 2.5.0 and hopefully soon after that a release final.

After that we’re going to see a public BETA of StableBit DrivePool 2.2.

What’s New

Now that we’re approaching the 2.1 release final and file placement is implemented, I’d like to start talking about “the next big thing” for StableBit.

But before I get into that, let’s check out what’s new in this RC. Since my last post about version 2.1.0.528 BETA there have been a few new features added and a bunch of general fixes, not really related to any single area.

I’ll just highlight the noteworthy changes here.

New Features

Finalized all translations.

You can now send a test email message after entering your email address for notifications.

Minimize / maximize buttons were added to the horizontal UI. Stretch the window wide to see them.

The Importance of File Placement

Let’s talk a bit about the future, the future of StableBit, and “the next big thing”.

As a general concept, file placement is the ability to tell StableBit DrivePool which disks are allowed to store files placed in particular folders on the pool. I’ve been planning this concept for some time now. I’ve posted the first Nuts & Bolts discussion on file placement way back in September of 2013. But in that post I’ve only talked about specifically controlling which files go onto which disks. Well… there’s actually a lot more to file placement that will extend the value of StableBit DrivePool tremendously.

“Product 3″

I’ve talked about Product 3 in the past, which is the code name for the next StableBit product. It’s currently in development and actually gives StableBit DrivePool’s file placement feature a much more prominent role.

In order to understand why, let’s talk about how things work right now. Right now, when you add a disk to the pool, a hidden “PoolPart” folder is created on that disk. Any pooled files that need to be stored on that disk are simply stored in that hidden Pool Part folder. So in reality, when you add a disk to the pool, you’re actually adding a Pool Part to the pool, and that Pool Part happens to be stored on a local disk.

I hope that you see where I’m going with this. Product 3 will allow you to add Pool Parts to the pool that are not necessarily stored on physical disks. This is going to open up a whole range of very exciting possibilities.

It’ll be possible to store Pool Parts on virtually anything that you can imagine that can store persistent data. Email servers, FTP, UNC shares, cloud storage might be some examples, all you’ll need is a plugin for which there will be an open API.

In this context, StableBit DrivePool’s file placement will gain a whole new use. It will allow you to define which folders on the pool are stored on which mediums. Moreover, with per-folder duplication, you will be able to specify which specific mediums will store the duplicated file parts, of each folder.

A Standalone Product

I’ve just talked about how Product 3 can be used together with StableBit DrivePool, but really it’s much more than that. It’s going to be a full standalone product with some unique functionality. I’ll be talking about it some more as we get closer to the first public BETA.

StableBit DrivePool 1.3.6 and StableBit Scanner 2.5

Both of these have been in BETA for some time now. I’ll be working on getting these out into release final form soon.

StableBit DrivePool 1.3.6 is a bug fix release and StableBit Scanner 2.5 has some noteworthy new features.

After the release of build 503, which introduced the new file placement balancing rules, we’ve had some fruitful discussions over on the forums about how file placement can be improved and this build incorporates some of those requested features. It also features some important bug fixes to file placement.

Folder Placement Changes

Folder Placement Rules

There are a few UI changes as well as functional changes that have to do with how folder placement rules are applied.

Folder Placement Changes

Folder Placement Rule Inheritance

As you can see, there are now new icons next to folders that have folder placement rules applied to them. This makes it easy to tell which folders have rules defined on them and which are inheriting rules. The icon is green for a folder that has a folder placement rule defined on it and it’s black if the folder is inheriting a folder placement rule from one of its ancestor folders.

This brings us into a functional change that affects how all file placement rules behave in this build compared to build 503. In build 503, if you defined a folder placement rule on some folder and then defined a different rule on a subfolder of that folder, the rule for the subfolder would get combined with the rule on the parent folder. This is counterintuitive and is really not obvious, so the latest build doesn’t do that anymore. From now on, multiple rules are never combined.

Adding New Drives

There is now a new checkbox that lets you control what happens to each file placement rule when you add new drives. In build 503, adding a new drive would add that drive as a selected destination for each file placement rule that you’ve defined. In the latest build it’s the exact opposite. New drives are never added to your existing file placement rules by default, but you can bring back the old functionality on a rule by rule basis using this new checkbox.

File Placement Changes

File Placement Changes

There are some functional and UI changes to the pattern based file placement rule interface.

File Placement Rule Priorities

A very important functional change in this build is that only one file placement rule is ever chosen for each file, when selecting a destination for that file. If multiple patterns match a given path, then the highest priority pattern is chosen. This introduces the concept of rule priorities. In the UI, you can now rearrange your file placement rules by clicking the up / down arrows or with drag and drop.

When arranging your file placement rules, one thing to keep in mind is that priorities are automatically managed for folder based rules (defined under the Folders tab). For folder based rules, rules defined on deeper directory structures must always have a higher priority than any rules defined on shallower directory structures. This is so that the user doesn’t have to think about priorities when defining folder based rules, and their rules work as expected. So you are allowed to rearrange your rules, given that you don’t violate that restriction.

Pattern based rules can be arranged in any order and don’t have the same restriction. If you had any file placement rules defined in build 503, they will be automatically rearranged for you.

Multi Select

File Placement Multi Select

Just to make setting up and managing many rules easier, you can now select multiple rules (using the standard hold SHIFT, CTRL paradigm) and make changes over all of them at the same time.

Other Fixes

There are some other noteworthy changes in this build:

Tasks such as balancing and background duplication can now be manually aborted or boosted in priority from the pool organization bar.

The progress percent on rebalancing and drive removal will now move continuously, even when moving large files.

Background duplication is now file placement aware and will try to respect the rules when possible. For example, when enabling duplication on a folder, if that folder has a file placement rule then the second copy of the file will be stored on one of the disks that match the rule (if possible). The same thing goes for disabling duplication. If a file part is violating a file placement rule, it will be cleaned up first.

It is now possible to disable a parent folder based rule for a particular subfolder by having all the disks checked on that subfolder.

When renaming a file on the pool violates a file placement rule, the service will be notified and a balancing pass will be scheduled.

Multiple disk list issues were fixed that were causing the disks list in the main UI to flicker and the tooltips to sometimes disappear.

This is a major addition to the already extensive balancing options and it basically allows you to select which disks will be used to store the files in a particular folder (I believe that this is a first in the industry). Let’s take a look at how this new feature works.

File Placement

Even though this feature required extensive code changes to all 3 parts of StableBit DrivePool (the driver, service and UI), the UI to configure file placement sits under a single new tab in the Balancing window.

Pool Options

File Placement – Folders

I’ve set up a sample pool for this post, and on the left you can see the folder structure of it.

Selecting a folder will calculate duplication and file placement statistics of that folder. You can toggle between the duplication or file placement statistics view from the toolbar at the top.

Statistics Toggle

File Placement Statistics

Each bar and pie slice has tooltips for more finer detail.

Folder Placement Rules

To define a folder placement rule, you simply select a folder and uncheck one or more disks on the right. Files in this folder and subfolders will only be stored on the checked disks.

Folder Placement Rules

Here you can see that I’ve defined a new rule stating that I don’t want any files in \Software\MSDN, a duplicated folder, to be stored on K:\. I also have the option to choose what happens when the other disks start running out of disk space. I’ve chosen to keep the default behavior and allow the files in \Software\MSDN to overflow to other disks if my chosen disks get used up to 90%.

Pool Organization

After hitting Save you can see that the Pool Organization bar drops down to 0% and you are offered the option to start a balancing pass immediately. If you have automatic balancing enabled then a balancing pass will start as per your settings.

If you’re in the horizontal UI mode (the window stretched wide) then you can also see a little file icon next to each disk space usage bar indicating that this disk had a file placement rule changed recently and that StableBit DrivePool needs to go through the files on that disk and reorganize them based on the new rules.

After we let the balancer run, we can see that the file distribution for that folder now looks like this:

Folder Placement Limited

Interaction With the Existing Balancers

For the example above, just to make things simple, I’ve disabled all of my standard balancers. But let’s see what happens when I enable the Duplication Space Optimizer.

Duplication Space Optimizer

Balancing Markers Adjusted

You can see that the balancing markers have adjusted and that StableBit DrivePool now wants to move some unduplicated files from both M:\ and L:\ onto K:\. This is so that you would have more disk space available to your duplicated files on M:\ and L:\.

Of course this would normally be taken care of for you automatically in a single balancing pass right after you define your file placement rules and you wouldn’t have to take any additional steps. In this case, I just wanted to show you this behavior.

Advanced File Placement Rules

If you want to define more advanced rules on a per file basis, you can do that as well. Under the File Placement tab select the Rules tab and add a custom rule.

Add File Placement Rule

You can type in any pattern that matches one or more files.

A * matches 0 or more characters

A ? matches any one character.

For example:

\Software\MSDN\*.ISO – Match all .ISO files under \Software\MSDN and its subfolders.

\Users\*\Downloads\* – Match all downloads in any user’s folder.

Don’t forget that you’re defining patterns to locate files and not folders, so you must at the very least include a trailing \*.

As you may have guessed, the simpler Folders interface for defining rules actually uses the same rule definitions under the hood. Those folder based rules can also be seen from the Rules tab.

Rule Conflicts

With all this power there is now the potential so set up conflicting balancing rules. Let me show you a simple conflict in order to demonstrate.

I’m going to enable only the File Placement Limiter and I will tell it that I don’t want any duplicated files on K:\.

File Placement Limiter

I will then go ahead and change my existing file placement rule for \Software\MSDN to tell it to place files only on K:\ and L:\.

Conflicting Rule

You can see the problem here, \Software\MSDN is a duplicated folder, and I just told StableBit DrivePool to move the contents of that folder onto drives K:\ and L:\, but at the same time I’ve told the File Placement Limiter that I don’t want any duplicated files on K:\.

Let’s go ahead click save then rebalance and see what happens. The balancing pass completes rather quickly (because there’s not much to do) and you will see this:

Pool Organization Conflict

You will see a new tooltip message over the pool organization bar telling you that some files could not be moved.

If you open up File Placement once again, you will get a more descriptive message.

File Placement Conflict

Conflict Resolution Settings

StableBit DrivePool now features a number of automatic conflict resolution settings and they can be found on the Settings tab of the Balancing window.

File Placement Settings

How it All Works Together

I don’t want to get too technical in this post but if you want to read up a bit on how the architecture works I’ve posted that over here:

The StableBit.com Affiliate Program

I just want to wrap up this post by mentioning that there is now an affiliate program available for anyone to join. It’s completely free and basically it means that if you refer anyone to us and that reference results in a sale you will get 25% of that sale deposited to your account. Check it out, we could definitely use your help, and really do appreciate it. All you need is a PayPal account.

What’s Next

StableBit DrivePool 2.1 has been in the works for a while now, and the list of fixes is also impressive (some of those were particularly difficult to pin down). This build features a whole lot of fixes and a major new feature, which is exactly how I like it.

Next, StableBit DrivePool 2.1 goes final and “Product 3″ is actively in the works which should integrate with StableBit DrivePool quite nicely.

This build took a while to get done (almost 2 months) and the main new feature here is support for reparse points. Reparse points are the underlying technology that Microsoft Windows uses to create links or shortcuts between files or directories, on the same drive or even among different drives.

Link Shell Extension

Unfortunately Windows Explorer is not very savvy when it comes to working with reparse points. I prefer to use a free shell extension called the Link Shell Extension in order to make working with reparse points easier.

Command Line

As an alternative, you can also use mklink and fsutil on the command line to work with reparse points.

A Simple Example

Reparse points can get very confusing if you try to think about how they work, so instead of doing that let’s dive right in with a simple example.

2 Folders

For this example, I’ve created a new pool (G:\) and have created 2 folders in the root directory of that pool, Folder1 and Folder2.

G:\Folder2\File2.txt

Now, in Folder2 I’ve created a new file called File2.txt. In this file I’ve put the text “This is file 2.”

What we’re going to do now is create a symbolic link in G:\Folder1 that points to G:\Folder2.

Pick Link Source

The first thing that we’re going to do is right click on G:\Folder2 and select Pick Link Source.

Drop Symbolic Link

We then navigate to G:\Folder1 and right click, select Drop As… > Symbolic Link. You’ll notice that there are a lot of options here, so if you don’t know which one to pick, symbolic link is generally the best bet.

G:\Folder1\Folder2

What we now end up with is a symbolic link G:\Folder1\Folder2 that points to G:\Folder2.

You can right click on G:\Folder1\Folder2 and select Properties > Link Properties to see some additional information about the symbolic link.

Link Properties

From here we can see that G:\Folder1\Folder2 actually points to ..\Folder2. You’ll notice that the link target doesn’t contain a drive letter and so this is called a relative symbolic link. If the link target were G:\Folder2, then it would be an absolute symbolic link. The Link Shell Extension will create the appropriate links for you automatically.

So let’s navigate into G:\Folder1\Folder2.

Reparse Point

Notice that we see File2.txt which actually exists in G:\Folder2. Let’s open it up.

File2.txt

And there we have it, the same text that we placed in G:\Folder2\File2.txt in the beginning of this example.

It is important to understand that this is not a copy of File2.txt, but the original one and that G:\Folder1\Folder2 is not a copy of G:\Folder2. For example, if we place another file in G:\Folder2, it will automatically appear in G:\Folder1\Folder2, and this essentially illustrates what symbolic links (which are a type of reparse point) can do.

File Symbolic Links

In the example above I’ve shown how directory symbolic links work, but symbolic links can work with files, and there is no limit on how many symbolic links can point to the same file or directory, 1, 10 , 100, it doesn’t matter.

Multiple Symbolic Links

In the screen above I’ve created 3 separate symbolic links in G:\Folder1 to the file G:\Folder2\File2.txt. All of these essentially are the same file.

Links Across Volumes

A symbolic link can technically point to any kernel path, which includes another disk volume.

Links to Other Volumes

In the screen above, I’ve created a symbolic link in the G:\ pool that points to the F:\ pool. Isn’t that cool?

Other Types of Reparse Points

Symbolic links are only one type of reparse points, and in my opinion they are the most flexible. But in addition to symbolic links StableBit DrivePool supports junctions and mount points.

Other Reparse Points

In the screen above I’ve created a mount point to C:\ and a junction to F:\ in G:\Folder1. Junctions and mount points are precursors to symbolic links, which were not implemented until Windows Vista. I suggest using symbolic links unless you have a special need not to.

Hard Links

In addition to the other types of links, the Link Shell Extension has the ability to create Hard Links. Hard Links are not implemented using reparse points and so this build of StableBit DrivePool does not support them. Hard Links are implemented directly in NTFS and can only point to a file on the same volume.

Implementation Difficulty

The reparse functionality here was notoriously difficult to implement correctly and that’s why this build took so long to get out. As a result, some functionality that I planned to include in it (such as TrueCrypt support) didn’t make it. But I just wanted to acknowledge that it’s on the way.

This BETA of the StableBit Scanner features a complete overhaul of the notification system. The entire back-end was mostly rewritten and a new UI now allows you to configure our new, much more feature rich, notification system.

The StableBit Scanner can now send notifications using these providers:

Email

SMS (US only for now)

Speech (through local speakers)

Twitter

Mobile

Pushover (iOS and Android)

Pushalot (Windows Phone and Windows 8)

Notify My Android (Android)

The New Back-End

You may be wondering why a rewrite of the notification system was necessary when the old one was working fine. The answer is that different notification providers send notifications in different ways.

Because the old notification system was written essentially for tray icon notifications it only had the notion of an alert either being raised or not. The new notification system is much more flexible and supports a wide variety of notification scenarios.

The New UI

Because the new notification system is very comprehensive it deserves its own Window, so the old “Notifications” tab in Scanner Settings is now gone.

Settings

The updated drop down menu is now called “Settings” and features the new Notification settings option.

Let’s take a tour of the new notification settings window and talk about some of the new features.

Site Name

StableBit Scanner – Notifications

One of the first things that you will notice is that there is now a place to configure a “Site name”. Assigning a site name is basically a simple way of grouping all of your copies of the StableBit Scanner running at some site. When a notification is sent out, be it email, SMS or anything else, a site name will be included in the notification.

For example, you may see something like this in a notification:(StableBit Scanner on “SERVER3″ at “Main Office”) A disk is overheating…
Case: Rack 7B
Bay: 3

Cooldown Threshold

There was a minor annoyance with the old notification system related to overheating warnings. If a disk’s temperature was hovering at the overheating threshold you would tend to get lots of notifications that the disk is overheating and then not, over and over again.

This was kind of annoying so the new notification system has a “cooldown threshold”. Basically this means that an overheating disk does not exit its overheating state (as far as notifications are concerned) unless its temperature drops some degrees below the overheating threshold.

Test Notifications

Another thing that the old system didn’t have was a “test” button. In the new system, every notification provider now has a test button that sends a test notification using that provider.

Email Test

This lets you easily confirm whether you’ve configured the notification provider correctly and that your notifications will be delivered to you.

Email Sent

Let’s now take a tour of all the notification providers.

Email

Email

This is pretty much the same functionality as before, but with a nicer UI. You can enter multiple email addresses and test them by sending test email messages.

Emails are dispatched using our paid for Amazon SES service and we do not store the email addresses on our server only in your copy of the StableBit Scanner.

SMS

SMS

This is brand new and really took a while to get right. There are a number of ways to send SMS messages out there.

One way is to use a paid online SMS gateway that offers an API and then charges you per message sent. This can be expensive, but it’s the easiest to implement.

Another way is to use a SMS Email gateway. Most cell phone providers offer SMS Email gateways which forward all emails received at your SMS Email address to your cell phone as a SMS. This approach is not reliable and requires the user to select their carrier. This is not what the StableBit Scanner uses, but if you’d like, you can certainly enter your SMS Email address in the Email notifications provider.

The StableBit Scanner uses a physical SMS gateway that sends your SMS directly to the cell network. We subscribe to a business SMS plan and a physical SMS gateway (in the form of a SMUSH Box). Your messages only stay in our system for the sending process and then all information about the message, including the telephone number is erased.

For now this is enabled for US domestic telephone numbers only, but the plan is to add international support as well.

Speech

Speech

Speech notifications have been updated to let you select the rate of speech and a different voice, when more than one is available (E.g. Windows 8).

Twitter

Twitter

Yes, that’s right, twitter notifications are now possible. Why? Because twitter is the de facto social network for sending out notifications and status updates. It can be used as a point of integration with other cloud services (IFTTT comes to mind). You can tweet to all of your followers or direct message another twitter user (including yourself).

Mobile

While we don’t offer any mobile apps yet, but that doesn’t mean that we have to exclude smart phones and tablets from notifications. Whether you’re running iOS, Android, Windows Phone or even the Microsoft Surface the StableBit Scanner has you covered.

Here are the notification services that we support on these platforms, starting with my favorite:

Notify My Android

This one is a bit clunky but it seems to be popular. The free account is limited to 5 notifications a day, but you can upgrade to an unlimited account for a one time fee.

Notification Templates

Notification Templates

One other cool thing that came out of the notification rewrite is the ability to completely customize your notifications. You see, every notification provider (except speech) uses XSLT to generate the notification text or HTML. So if you know how XSLT works you can easily tweak your notification text for each provider.

Here’s an example of the Email template:

Email Template

As you can see, you can change the text and structure of how the email is generated by editing this file.

Conclusion

This BETA features a significant notification update, and more features are on the way. The next builds of the StableBit Scanner will concentrate on improving SMART reliability and accuracy by making greater use of cloud features.

The next BETA of StableBit DrivePool is taking longer than expected due to some complications with reparse points. But a new build is in the works, so stay tuned for that.

If you’d like the StableBit Scanner to support additional notification providers, let me know. I’d like to hear your suggestions.

After going through a lengthy public BETA testing period, StableBit DrivePool 2.0 is now being tagged as a Release Final. Thank you everyone for testing the BETA and reporting any issues that you’ve encountered, most of those turned into direct fixes applied to the code.

Operating System Support

For those of you who may be visiting this page for the first time, and have never heard of StableBit DrivePool, let me give you a brief summary.

StableBit DrivePool 2.0 is a disk pooling application. It can combine the free disk space of one or more hard drives without altering the existing files on those drives or anything about the disk formatting itself. It creates a new pool, as a new virtual drive in the OS. You can freely add and remove disks to and from the pool at any time and treat the new virtual drive just like any other disk.

Your files are stored as standard Windows NTFS files on one or more disks that are part of the pool.

StableBit DrivePool also provides folder duplication. Basically you can designate the pool, or one or more folders on the pool as duplicated folders. Any files placed in a duplicated folder will actually reside on two or more disks in the pool.

StableBit DrivePool – Folder Duplication

StableBit DrivePool 2.0 OS support starts at Microsoft Windows Vista and newer. It supports both x86 and x64 variants of the OS.

Upgrading From 1.X

For those of you who are running the 1.X version and are happy with it, there’s no need to rush and upgrade to 2.0 immediately unless you want some of the new features. The 1.X line is still being worked on, just earlier today StableBit DrivePool 1.3.5.7572 was released.

Pros of upgrading for WHS 2011 users:

Remote control.

More flexible folder duplication settings.

Duplication counts > 2

Setting duplication counts on sub-folders.

Redesigned UI.

Things that 1.X has that 2.X doesn’t:

A more “native” looking Dashboard UI.

However, StableBit DrivePool 2.0 does install a Dashboard tab.

Support for Dashboard alerts.

Native .wssx installer.

Some of these will probably show up in 2.X down the line, but for now those are the differences.

For me personally, I love the super efficient Remote Control. It means that I don’t have to load the Dashboard every time that I want to manage StableBit DrivePool or the StableBit Scanner. So I already upgraded my personal server.

Upgrading Instructions

If you do wish to upgrade to 2.0, there is no additional cost and you are free to do so. But because StableBit DrivePool 2.0 uses a completely different installer, you will need to follow these steps:

Uninstall StableBit DrivePool 1.X using the Dashboard’s add-ins tab.

Reboot the server.

Install StableBit DrivePool 2.0 by logging into the server using remote desktop as Administrator.

Reboot the server.

Uninstalling StableBit DrivePool does not alter your existing pooled data. In addition, StableBit DrivePool 1.X and 2.0 use the same file storage scheme so they are backwards and forwards compatible.

A New Product

New Product

The initial StableBit DrivePool 2.0 BETA is now over, but that’s not the end of its development. A new StableBit DrivePool 2.1 BETA should be out within a couple of weeks, and of course we will continue to listen to your feedback and fix anything that’s broken.

But there is something else in the works! A new product is under development right now which I am very excited about. No, it’s not related to BitFlock or anything remotely similar. While I can’t tell you any more than that for now, hopefully we can have a new public BETA of that in a few months.