Bit Flock

Creating .NET applications that just run on any version of Windows can be tricky.

ForÂ BitFlock,Â the requirement was to create a Windows application that will be very easy to run. There’s no need for installing it into the user’s system and the need for dependencies has to be minimized, if not eliminated completely. The application was written in .NET, but the challenge is, how do you deploy a .NET application over the web, without an installer and in a single EXE?

One solution would be to use a virtualization technology that creates a single EXE file that effectively emulates the environment that the application needs to run in. There are a couple of solutions out there that do this but, understandably, the EXE files that they generate are too large (60+ MB). This is not an options for deploying a simple application meant to be opened directly from the web browser. At least one solution I’ve seen use a form of streaming technology to make the download process bearable, but that solution is too expensive and frankly overkill for what the BitFlock application was supposed to do.

Alternatives? Delphi? Assembly? If you’re already invested in the .NET Framework then these are not an option. Delphi is another technology to learn, and you may not like it. Assembly is a hard. So what’s the best answer if you want to do this in .NET?

.NET Ecosystem on Windows

The first question to ask is, if I compile my application targeting a specific version of the Framework, what versions of Windows will it run on out of the box?

To survey the landscape, here’s a neat little chart to show you the bigger picture.

What are all those dots?

If you look at the black solid ones, those are the important ones. Every black solid dot means that the .NET Framework comes pre-installed on that version of Windows. Knowing that, we can now answer the original question. If you target your compiler for .NET 2.0, your EXE will run on the .NET 2.0, 3.0 and 3.5 Full Frameworks. And, we can see that at least one of those come pre-installed on Windows Vista, Server 2008, Windows 7 and Server 2008 R2.

Note that .NET 2.0 assemblies files are actually installed when the user installs .NET 3.5 and .NET 3.0, so your .NET 2.0 application is not actually running over .NET 3.0 or 3.5 in the strictest sense. But the result is the same.

.NET Dependencies

We can quickly see that it’s not possible to deploy a single compiled .NET EXE that will run on everything Windows XP and later without including some sort on .NET Framework installation method. Also, more troubling, is the fact that our .NET 2.0 compiled assemblies will NOT run on a system with only .NET 4.0 installed. This is a distinct possibility since Microsoft’s still popular Windows XP SP3 platform came with no .NET pre-installed, and .NET 4 is being offered as an optional update on that platform.

So what’s the answer?

For BitFlock, our solution was twofold.

– 1 –

Have 2 separate binaries generated from the same code, one targeting .NET 2.0 and one targeting .NET 4.0. We will try to detect the .NET Framework installed from the browser’s user agent, this works for Internet Explorer and Firefox with the .NET Framework Assistant extension installed. If the user agent doesn’t contain a string identifying the version of the .NET Framework then we will check the Windows version reported in the user agent string against any known versions of Windows with the .NET Framework pre-installed. If the user is using one of those versions of Windows we send them the appropriate .NET EXE.

– 2 -ï»¿

If we can’t identify the user’s .NET Framework and the user’s version of Windows is not one that comes with a pre-installed version of the .NET Framework then we rely on something called a universal binary.

The Universal Binary

The universal binary consists of a few pieces.

A small native code bootstrapper, with a size in the tens of kilobytes, that runs on all copies of Windows starting with Windows 2000 SP4 and up.

A .NET 4.0 Client Web Installer.

A .NET 2.0 version of the BitFlock application as a single merged EXE.

A .NET 4.0 version of the BitFlock application as a single merged EXE.

Assets, such as settings and graphics.

The binary itself is a single EXE packaging all of the above as resources. The total size of the universal binary ended up being around 4 megabytes. An acceptable compromise.

How does it work?

The bootstrapper launches and reads the embedded XML settings file which defines the minimum requirements for this application. A requirement can be a particular Windows version range, existence of a particular file or registry key. Requirements can be grouped and logically combined with AND / OR.

If the system meets these requirements then the bootstrapper checks the individual requirements of each embedded application. In our case, there are 2 embedded applications, a .NET 2.0 version and a .NET 4.0 version. The first application the meets it’s requirements fully is executed and the bootstrapper’s job is complete.

If none of the embedded applications meet their requirements on the first pass, the bootstrapper then check if any application can potentially meet it’s requirements. How does it do that? Each specified requirement can optionally contain information on how that requirement can be met. In our case, it contains information saying that the .NET 4.0 requirement can be met by executing an embedded installer.

If the bootstrapper finds such an application, it displays a minimalistic UI to the user informing them that the application requires that you install certain dependencies before it can run. The user has the choice to either proceed with the installation or exit.

User Experience

At the end of the day, our primary concern is the user experience. This is the user experience running BitFlock as it’s implemented now:

The user visits the web site, they are identified as a Windows user from the browser’s user agent string,Â and they click the Run link.

The web site tries to identify the installed .NET version. If it can, then an appropriate EXE is sent.

The web site then tries to identify the user’s version of Windows. If it can, then it checks whether it’s one of the known versions of Windows that come with .NET pre-installed. If it is, then it dispatches the appropriate EXE.

If the two tests above fail, then the web site sends the user the Universal Binary.

The universal binary checks the version of Windows that it’s running on and the service pack level to determine whether it is supported by BitFlock. If not, then it displays a message indicating that it can’t run and lists the supported Windows versions.

It then checks to see if the user has .NET 2.0, 3.0, 3.5 or 4.0 installed. If they do, then the appropriate embedded EXE is launched, and the bootstrapper is done.

At this point, the bootstrapper will check to make sure that the system meets the minimum requirements for the .NET 4.0 installer. If not, a message is displayed to the user indicating that they must install .NET 2.0 or 3.0 or 3.5, manually. This is rare.

If .NET 4.0 can be installed, then a minimalistic message is shown to the user informing them that .NET 4.0 must be installed before the application can be run. The user is given the chance to exit and not install the .NET Framework.

If the user chooses to proceed, .NET 4.0 is installed over the Web and the application is started.

The goal of this process is of course to make sure that as few people as possible make it all the way down to #9. Ideally, #6 is as far as most people should go.

I’ll share a bit more on the internal architecture of the bootstrapper and the settings file that makes this all happen in a later post.

This is the first in a series of posts describing the technology behind BitFlock. This time Iâ€™m going to focus on the cloud aspects of BitFlock.

BitFlock is really a service consisting of 2 pieces, a web page and an application designed to gather health information about your hard disks.

The application is designed to be as simple as possible and does not have much of a UI. Iâ€™ll talk about the different types of information that the application gathers in another post, but Iâ€™ll just mention that it doesnâ€™t read any partition data, so your files are never read. Also, no writing to any of your drives is ever done.

Once the application gathers health information about your drives it uploads it to a web service running at https://bitflock.com using 128-bit SSL . From there, itâ€™s assigned a unique â€œNest IDâ€ which is linked to the health data. The purpose of the Nest ID is to uniquely identify your set of data so bitflock.com can show it when you call it up from the web browser.

Also, itâ€™s worth noting that the entire web site operates over 128-bit SSL and will redirect any http traffic to https automatically.

Why Online?

When the BitFlock application gathers data from your hard drives, the information is retrieved and uploaded as binary chunks. When these binary chunks arrive on the server theyâ€™re stored in the database as binary chunks. No interpretation is done. Only when you view your nest is the information put together and interpreted in a way that makes sense.

This means that neither the BitFlock application nor the web service does any interpretation of the data at collection time. This is important because the interpretation of S.M.A.R.T. data in particular, is manufacturer dependent. So when a new drive model becomes available, or more information becomes available about an older drive model, BitFlockâ€™s view of the health data for those drives is automatically updated. There is no need to deploy a new version of the application, or to re-run the scan process.

This is kind of an oversimplification. In reality, BitFlock does cache some interpretation data for efficiency. But there is a process to force a full nest update when necessary. Regardless, the result is the same.

How is S.M.A.R.T. Interpreted?

Just in case youâ€™re not familiar with S.M.A.R.T., let me do a brief summary.

Summary of S.M.A.R.T. Data

S.M.A.R.T. consists of many different pieces, but the part that most interests us is S.M.A.R.T. attributes and thresholds. This is what you typically see in many S.M.A.R.T. reading application. Itâ€™s usually shown in a table, and if youâ€™re not familiar with its format, it can look very cryptic and daunting.

To the uninitiated, a complicated list of seemingly incomprehensible statistics.

This is how BitFlock presents the same data:

Of course you donâ€™t even have to look at this if you donâ€™t want to. Thatâ€™s because BitFlock will show you a summary for each drive in plain English right on the front page of each nest.

Attribute / Threshold Pair:

It is generally accepted to say that an attribute / threshold pair consists of the following parts:

Status â€“ Indicates whether this attribute is purely related to drive age or is indicative of failure.

Attribute Value â€“ This is a number from 0 to 255. It has no meaning other than the fact that it shouldnâ€™t fall below the threshold.

Threshold (or Warn) Value â€“ The attribute value should not fall below this value. If it does, then depending on the Status it can mean different things.

Minimum (or Worst) Attribute Value â€“ This is the lowest that the Attribute Value has dropped to in the past.

Raw Value â€“ Depending on the Type, this gives us different statistics about the health of the drive.

This is how S.M.A.R.T. predicts drive failure, when an attribute value falls below the threshold value for any attribute with the status of â€œPre Fail Advisoryâ€ the drive is expected to fail within 24 hours.

This is useful and of course BitFlock detects this condition and flags the drive with a red health icon if this ever occurs.

BitFlockâ€™s Further Interpretation

The power of BitFlock is that it goes one step further than the general S.M.A.R.T. interpretation. BitFlock works directly with the Raw Value to check if any of them are indicative of trouble. But before it can do that it needs to identify your disk type. It does this by reading the model / firmware pair from the drive. It tries to find an interpretation group from hundreds defined in the database. This list is constantly updated as new drives are released and more information becomes known.

Once it finds an interpretation group it builds an interpretation table, where it lists each attribute type that should be checked for certain conditions in the raw value that indicate trouble. For each attribute that indicates potential trouble, the check algorithm and the plain English warning type is also listed.

BitFlock generates a way to check your drive, that is specific to your hard disk model. If any of these attributes trigger a warning condition, a yellow icon is shown for that disk with an explanation.

In addition the check data, the attribute table contains information on how to decode each individual raw value into a human readable format.

So basically for each attribute BitFlock knows:

The name and description of the attribute

How to display the value of the attribute in a human readable format.

Whether this attribute should be checked for out of the ordinary values and which algorithm to use for this check.

If the attribute has triggered a warning, how to display it in plain text.

And this information is specific to your drive model.

An Example

Time for an example to clear this up. I like realistic scenarios, so letâ€™s take a real drive from the demo nest.

Letâ€™s say you have a hard disk with the model ST3500630AS and firmware 3.AHG.

BitFlock sees this and tries to find an interpretation group than matches that serial number / firmware pair. Failing to find a specific interpretation group, BitFlock will try to find an interpretation group that is not firmware based. This type of firmware-less interpretation group is used where all the drives for a particular model behave the same way, regardless of firmware.

With this method we can target a particular model with a particular firmware with one interpretation group, and then have another catch all group for all the other firmwares. Itâ€™s useful to do this if a firmware version has a bug in it that affects S.M.A.R.T. This way we can work around the bug automatically. In fact, BitFlock does this very thing for a number of drives.

Our interpretation table is built up for each attribute as described above. Letâ€™s just go through one of these attributes to show how cool this is.

Say weâ€™re looking for an interpretation of Attribute ID 5 (Reallocated Sectors Count). We know that the drive belongs to the interpretation group Seagate Barracuda 7200.10 family, because we identified it earlier. We then ask the system if there is a model specific interpretation entry for a Seagate Barracuda 7200.10 family drive. If not, then we ask for a generic, non-model specific, interpretation of attribute ID 5.

Once we retrieve the interpretation entry we know a few things.

We know the attribute name and description:

We know how to combine the raw attribute bytes to display them in a meaningful way.

We know how the check for a warning condition on that attribute.

We know how to generate the plain text warning message if the warning is triggered:

This makes the system very flexible and powerful. We can now essentially build a custom health report tailored to your specific drive.

By the way, I picked attribute 5 on purpose. Attribute 5 is generally the number of sectors in the G-LIST (see the Wikipedia entry for bad sectors). This is the number of sectors that went bad since the drive left the manufacturing plant.

A SSD Example

An example that takes better advantage of this system would be a SSD drive.

Back to our demo nest, letâ€™s look at the INTEL SSDSA2M160G2GC drive. This is a Generation 2 Intel SSD drive. It comes with an on-board lifetime indicator. However, the attribute is encoded differently, and in order to calculate the percentage you need to do some math on it.

None of this is a problem for BitFlock. We donâ€™t need to deploy a new version of the application, all we do is add some new interpretation entries targeting that specific interpretation group. Thatâ€™s it. Now the system knows how to tell you about drive lifetime. It also knows to warn you when youâ€™ve used up more than 90% of it.

Alright, I have to admit, it seems like everything these days has a cloud service. So the question is, who needs Bit Flock? With so many good paid and free S.M.A.R.T. utilities out there, why another one?

In order for something to be useful it needs to solve a concrete problem and it needs to do it well.

Letâ€™s start at the beginning.

Why should anyone care?

If youâ€™re reading this, chances are you own a computer. All your data is stored either on a Solid State Drive or a Hard Disk Drive. Both of these devices will stop working after some number of years. At that time, the data on those drives may become damaged and or inaccessible. Drive recovery shops are very expensive. Iâ€™ve heard anecdotal stories of some old 20MB hard drive still going strong today, but letâ€™s face it, modern hard drives donâ€™t work for that long.

You should care because you donâ€™t want to loose your data. Even if you have a backup you will want to know if your current drives are in trouble. A hard drive thatâ€™s not healthy can cause system crashes, performance problems, or worse, data corruption.

What is S.M.A.R.T.?

Thereâ€™s a great article about S.M.A.R.T. on wikipedia, but Iâ€™ll summarize. At the end of the day S.M.A.R.T. has only one job, to tell your system whether your hard disk is going to fail within the next 24 hours. The answer to that question is either yes or no. If itâ€™s yes, then you better hope that you have a backup.

But S.M.A.R.T. isnâ€™t good at predicting drive failure?

Thatâ€™s the common knowledge floating around on the Internet these days. Based on my experience, I absolutely believe that itâ€™s true. But letâ€™s redefine the problem.

What is important to me, and I think computer user everywhere, is not to know when the hard drive needs replacement, itâ€™s to know when my data is in danger of being lost. These are not necessarily the same things.

Look at it this way, if you saved an important file to your documents folder and then come back a week later and canâ€™t access it, is that a problem? S.M.A.R.T. wouldnâ€™t think so. It has a backup plan for situations like this. It remembers the sector on the drive that canâ€™t be read and the next time you write to it, it will be â€œre-mappedâ€ to a known good one from the spare pool.

In other words, it assumes that itâ€™s ok for your important file to get lost. It wonâ€™t sound any alarm bells or set off any warnings but will instead quietly cover up the evidence the next time you write anything to that damaged sector, self-repairing the drive and making it appear as if nothing had gone wrong.

While S.M.A.R.T. is not good at detecting drive failure, itâ€™s generally really good at detecting when the situation above happens. It just doesnâ€™t do anything about it. What you need is a secondary utility to go in and look at the data to let you know that something like that happened.

This is just one example, and Bit Flock is designed to look for these kinds of problems and report them in a clean and user friendly manner, while at the same time not dumbing down the information.

What about other S.M.A.R.T. clients, why do we need a cloud based solution?

The problem with detecting these types of problems is that you need to dive deep into the different SMART attributes and interpret them in specific ways. Thatâ€™s because each manufacturer uses SMART in their own proprietary manner. The meaning of an attribute may change from one drive to another and the format of the attribute data may also change from one manufacturer to another. Showing you the actual numbers of the attribute doesnâ€™t tell you enough.

Iâ€™ll give you two concrete examples:

1. Seagate

On some Seagate drives, the Seek Error Rate attribute has a special encoding where itâ€™s actually a percentage over a time period.

A SMART client thatâ€™s not aware of this fact will just show you some large number. If itâ€™s a good client, it might explain what a seek error rate is, leading you to assume that your drive is having a huge number of seek errors and may fail at any moment.

This would be incorrect. First of all, seek errors are normal for a hard disk, they happen all the time. Second of all, that particular number is not a number at all. On specific Seagate drives itâ€™s actually a rate. Properly decoded it can be interpreted something like â€œ< 0.01% (16 / 261039)â€. That means that for the past 261,039 seeks, 16 of those were not successful and the drive had to try again. This is exactly what Bit Flock does for you, and more.

Having a seek error rate of < 0.01 % is not a problem, but having a seek rate of 5.21%, combined with 14 reallocated sectors, should probably alarm you. Bit Flock will let you know that your data may be in danger with a friendly warning.

This example is taken from 2 real Seagate hard drives part of Bit Flock.

Bit Flock can detect this type of problem because:

It knows that you have a particular Samsung hard drive that encodes itâ€™s seek error rate in this way.

It understands that a 5% seek error rate is too high, given that other similar models in the flock donâ€™t experience this.

2. Solid State Disks

Solid state disks support SMART just like any other hard disk, but their SMART data needs to be interpreted differently. For example, the â€œReallocated Sector Countâ€, which normally implies data loss, is not that critical on a SSD. Thatâ€™s because a SSD reallocates damaged sectors before writing your data, unlike a HDD that typically reallocates sectors on the next write after some of your data canâ€™t be read. This means that you donâ€™t typically lose data when a SSD reallocates sectors.

Also, a SSD may have a lifetime indicator which can be shown as a percentage of useful life left. Just like the seek error rate above, this percentage may be encoded in different ways for different manufacturer.

So whatâ€™s the point of all of this?

The bottom line is that you need a SMART client that can identify the model of disk that you have, and one that can look for the thing thatâ€™s most important to you, data loss.

It needs to be able to intelligently compare your data with data from other similar hard drives in order to determine whatâ€™s abnormal.

It needs to be able to quickly adapt to new drives on the market and update existing data if more information becomes available about any specific drive.

This implies a large database of drive types and attribute interpretation data that can be updated on the fly.

BitFlock.com was launched a few weeks ago as a public beta. Everyone is free to go to the web site and make a nest.

So what is it?

To sum it up, Bit Flock is a cloud service for hard drive S.M.A.R.T. health data.

Bit Flock supports Windows and is very simple to use. You download a small stand-alone executable and run it. It very quickly analyzes your hard drives and creates an online â€œnestâ€ for you containing the result of the analysis among other things.

If you want to update your nest, just run the EXE again and it will bring your online nest up to date.

Launched mostly as an outgrowth of other technology developed for another product that weâ€™ll be releasing soon, itâ€™s an application that reads your hard driveâ€™s S.M.A.R.T. data, uploads it online and tells you if your drives are healthy.

All of Bit Flockâ€™s features are outlined on its web site so I wonâ€™t go into it here. If you go through them youâ€™ll see that in fact itâ€™s a modern, state of the art S.M.A.R.T. client. I just want to point out one of those features here.

Bit Flock is cloud driven.

Why is this important? Well, interpretation of S.M.A.R.T. data by its very nature is something that evolves. Thatâ€™s where Bit Flock shines, since itâ€™s presentation is web driven, the interpretation can change to keep up with the times.

Iâ€™ll go more into this some more in future blog posts, but for now check it out and make a nest.