Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

An anonymous reader writes "Flash SSDs are non-volatile, right? So how could power failures screw with your data? Several ways, according to a ZDNet post that summarizes a paper (PDF) presented at last month's FAST 13 conference. Researchers from Ohio State and HP Labs researchers tested 15 SSDs using an automated power fault injection testbed and found that 13 lost data. 'Bit corruption hit 3 devices; 3 had shorn writes; 8 had serializability errors; one device lost 1/3 of its data; and 1 SSD bricked. The low-end hard drive had some unserializable writes, while the high-end drive had no power fault failures. The 2 SSDs that had no failures? Both were MLC 2012 model years with a mid-range ($1.17/GB) price.'"

We encountered extensive and progresssive file corruption on SSDs in an industrial device. It used the FAT file system, and after every loss of power, it ran its equivalent of chkdsk/f at the next boot. If power was lost again while this command was running, then it was guaranteed that the file system would become corrupt (despite the fact that we were writing nothing to the SSD; it held only files which were opened for reading). The window of opportunity was described as "very short", and the possibility of corruption was "very small" according to the vendor. In our experience in the field, and in our internal testing, the window of opportunity exceeded 20 seconds, and the possibility of corruption was "utter certainty".

The vendor fixed the problem in a very easy way. They changed the file system from FAT to a commercial journaling FS. In our subsequent tests, we never found any file corruption, even on iterated power loss at random intervals after power on.

If it was a drive being used to read schematics for CNC for instance, there isn't a manufacturer out there that currently offers a machine-tied UPS for the CNC machine. If the CNC machine loses power, then so does the drive, and vice versa, since it's all on the same circuit (usually you'll find the power stuff hidden in a cabinet along a nearby wall, and that stuff takes power directly from the mains).

We use USB flash drives for a data logger. Most of the time the data is being buffered in the ARM based Linux board's RAM to save power. Once we get a complete file's worth (4MB at the present) we power up, validate, write the file, and power down.
Supercaps have been a lifesaver. There's even enough capacity to do the write cycle if the flash was powered down when a power fail is detected. That allows to not lose whatever what was already in the RAM buffer.

I hate a lot of USB drives and CompactFlash. They're all designed as dumb commodity devices for the undiscriminating user, and trying to get any solid spec sheets out of the manufacturers is impossible if you're not also a giant corporation. Instead their data sheets are just marketing literature (you rarely get anything more technical than "8x speed"). Almost all are designed to work with Windows with no concern to work with embedded systems or production automation, etc. So you end up buying a wide variety to test with and see which ones are barely adequate to work with your system.

Not just a lot of them, most of them, to the point that my former contract rolled their own due to flaky controllers, etc put out by the SSD manufacturers. Yes, they found it cheaper and more efficient to make their own SSD drives, and to incinerate the ones that failed in a blast furnace than rely on the crap the manufacturers are currently foisting on the market.

You might check into adding supercaps into the power supply, across the DC output lines.For a less DY method, you could try this: http://www.beam-tech.com/093001/prd_pgs/internal_ups.htm# [beam-tech.com]It's an internally mounted, UPS. There are also some PC power supplies that have the UPS built-in, but expect to pay a premium for those.If your application allows it, you might want to just mount your SSD into a laptop. It already has internal battery power, and there isn't any exotic hardware you have to pay through the nose for.

Do laptops ever monitor health of the battery if external power is never removed? I'm aware that laptops can tell when the battery is eventually trashed in nornal use (Dells, in particular, seem to be pretty bitchy about it with continuously-blinking lights, and report their findings to the OS if it bothers to ask).

But being plugged in forever is not "normal use" for a laptop.

I like your idea (and no, I'm not the AC you're replying to), but I have this vision of a small laptop that has been running with ex

Right, sure: All of this battery information can certainly be gleaned under any operating system, given appropriate software.

But the question is (restated): If the machine never runs on battery, does the machine know the health status of that battery? Does it really have any idea what those figures really are? Can it possibly know, without ever having run on (or otherwise discharged) the battery what the operational status of that battery really is?

The problem with voting has always been that the idiots get to vote too. So while you might "vote with your dollars" to select the most reliable drive, they will vote for the one with the cute name, or the shiny case, or the "free gift", or the special price, etc.

space is at an extreme premium in those drives. There's a reason they feel so heavy/dense. Given the quilting layout of the chips, adding a single cap would prevent several memory chips from fitting. So you may as well then fill that remaining space with more caps. But you will reduce capacity, and that's what sells SSDs.

There's already a substantial amount of circuitry in them, far from "basic". It's essentially a CPU. I'd be interested to see some numbers as to average power drain during idle, read, and write.

The ones that did the best during the power blips probably did have caps and a bit more in their power system to handle it though. It certainly does surprise me that the mid-range, not the high-end, were the best performers in this test.

I don't know what SSDs you've been using, but I've never picked up an SSD (OCZ Vertex 2/3, Intel X25-M/320/330/335/510/520) that didn't feel light and sound nearly hollow.

Consumer drives are usually lightweight, they don't need the extra cooling. Enterprise drives depending on who they're made by and what they're for can have heatspreaders or heatsinks within, or attached to each chip adding to the weight.

but I've never picked up an SSD (OCZ Vertex 2/3, Intel X25-M/320/330/335/510/520) that didn't feel light and sound nearly hollow.

Rip it open and have a look. There's not much weight at all to a piece of fibreglass and some plastic resin encasing some silicon. Circuit boards and components are really quite light when they don't require cooling or even large bits of metal for simple thermal mass.

You'll find that even though it's light and looks hollow it'll be packed quite full. Now combine that with the problems associated with creating some form of energy storage. Storage can come in some electrical form, i.e. battery which would be

There is all kinds of extra space in a 2.5" SSD. They have a lil' CPU, some flash chips, and that's it more or less. They are quite small. In smaller form factors, then ya space can become an issue but there's plenty in a 2.5" unit.

You're assuming a desktop-sized drive in a desktop computer, yet nearly all computers sold today are portables, and laptop users are more likely to buy bus-powered external drives than mains-powered drives.

So the five most likely causes of power failure in a consumer hard drives (and presumably, in the future, SSDs), ordered from most likely to least likely, are probably:

Most enterprise SSDs do have small supercapacitors or capacitor arrays onboard for exactly this reason. Some of the higher-end consumer drives do too. But most consumer drives don't.

The answer? Get a UPS.

A UPS is no panacea: I experience grid failure very rarely.

However, relatively speaking I experience many more kernel lockups that require an ACPI-initiated poweroff by holding down the power button until the machine abruptly powers off. What do you do when a reboot/poweroff command causes your Linux/BSD machine to hang? I/O handle leaks in the Samba SMB client (ie. *not* the smbd daemon) and the Samba Winbind code are notorious for this. The only times I have ever had to "yank power" from a production Linu

Instead of a supercap I'd rather there be a couple of replaceable Lithium coin cells inside of an SSD, to just finish finish writes after the power unexpectedly dips for some reason. They're cheap commodities, they seem to have predictable failure rates, and I don't remember the last time I changed one in any computer (though it used to be a fairly frequent repair).

By using them only once in a blue moon and occasionally monitoring the voltage and setting a SMART error if they're getting worn out, I'd estim

Assuming you have sysrq keys enabled, you can hit alt-sysrq-s, wait for the sync to complete, alt-sysrq-u, alt-sysrq-b. This performs a filesystem sync then remounts all filesystems read-only then boots the system. Also if you have a stuck mount point you can always use a lazy umount (umount -l) to remove it from filesystem hierarchy so you don't need to reboot in the first place.

I don't understand how if they claim that it takes up to 20 sec for the final write to finalize that a computer that simply shutsdown in 10 sec won't have the same problem.

Drives support a blocking "sync" command that is only supposed to return when the drive has flushed all pending writes and has reached quiescence. If there is nothing pending to flush then the command will return immediately. If not, it may take the cited 20 seconds to return. Normal reboot/poweroff procedure in the OS waits for this condition, and this has been around forever (the HDD equivalent is to flush write cache and park the heads). That's why a 10 second shutdown can be safe even with the putative

You would think.
The only SSD I'm running is on my computer at home and my house is sufficiently UPS'd. It's always cool when the power goes out at my apartments but all my electronics keep going. I just wish there was a battery on that Time Warner box outside my door.

Uhhh...we solved this problem ages ago with UPS. If you care about your data put the machine on a UPS. I've had my business customers on UPS systems for years, showed them how to test the batteries and swap 'em when they get worn out, no problems. I just had to swap the PSU and HDD out of my netbox at the shop because a transformer blew on my block and managed to give the old gal enough of a shock even through a surge protector that it cooked the PSU and the HDD, but since its just a netbox I don't care eno

After the third used APC UPS that still didn't work properly after battery replacement, I gave up. None of them could handle vaguely anywhere near the load they were supposed to. I don't know whose UPSes to buy, but I wouldn't buy anything from APC any more. It's unfortunate, because they used to follow a simple formula (fat traces, quality components, sturdy enclosures, priced accordingly) and they were a good value proposition.

After the third used APC UPS that still didn't work properly after battery replacement, I gave up. None of them could handle vaguely anywhere near the load they were supposed to. I don't know whose UPSes to buy, but I wouldn't buy anything from APC any more. It's unfortunate, because they used to follow a simple formula (fat traces, quality components, sturdy enclosures, priced accordingly) and they were a good value proposition.

I have tried many other UPS devices and APC are the only ones I trust. Its not clear if you are buying used UPS devices or are the original owner and are just replacing the battery. If you are buying used units (i.e. off of ebay), you never know if they have been hit with a surge, etc. Beyond that, they contain a battery. Batteries degrade over time, usually between 3 to 5 years, same as your car battery. This means that while the UPS will have the rated capacity with a new battery, it will become less over

Sadly I have to agree APC was once a good company, now they lowball the shit out of everything and they suck. Now as far as power? Double it. I know, sounds like a bullshit way to raise rates but I have found they ALL vastly overestimate what the unit will take so if you need 300w? Buy a 600w. Need a Kilowatt? Buy 2. Yes it sucks but you have to play the game and that I've found is a good hard fast rule that works for pretty much any company in the UPS business. Now if its a home user you can often get by w

In other words that's why you should NEVER plug in a laser printer behind a UPS. The initial current surge when the laser printer wakes up can easily overload the UPS and shut itself off, cutting off the power to the other devices on the UPS completely.

That multi-grid relay was supposed to be a warm standby approach, but it was too clever for its own good, which brings us back to your opening sentence. Manual switches would have been preferable. A brief power outage while a maintenance guy scrambles for the relay (hopefully there's someone stationed near the control...) is acceptable when the utility goes down. Having the system screw up and imagine itself an emergency is just lame.

You can design the firmware to avoid corruptions as well, it doesn't need a hardware solution. The manufacturers just have to be aware that power failures will occur and take that into account during the design. Extra capacitors won't fix the problem of shortcuts in the design.

Some SSDs already have capacitors that do just this, so yes, they did think of it. Did you really think that SSD manufacturers aren't aware of this issue?

But when a few dollars can sway a purchase decision, and it's hard to convince consumers through a few sentences on the side of an SSD box that power protection circuitry is important to have, it's hard to justify putting it in. And since most SSD's are probably sold as OEM equipment where a few pennies can make the difference between getting the sale or n

But when a few dollars can sway a purchase decision, and it's hard to convince consumers through a few sentences on the side of an SSD box that power protection circuitry is important to have, it's hard to justify putting it in

This isn't buying a car. $3 or even $20 isn't going to be detrimental to the purchase oppritunity when the consumer can TELL it is of quality above the competitors. Blaming the consumer in this case sounds like you are on the other side

Exactly, this is buying consumer computer equipment. Put a label on the side with a bullet point touting your unexpected power fault protection and I can pretty much guarantee it will have no impact on your product sales. You know what will? The extra $2 price that puts you below the other guy on the "lowest price first" product sorting.

But when a few dollars can sway a purchase decision, and it's hard to convince consumers through a few sentences on the side of an SSD box that power protection circuitry is important to have, it's hard to justify putting it in

This isn't buying a car. $3 or even $20 isn't going to be detrimental to the purchase oppritunity when the consumer can TELL it is of quality above the competitors. Blaming the consumer in this case sounds like you are on the other side

How can the consumer TELL if its quality is above the competitors? The presence of capacitors doesn't mean that it's a better drive than a drive without capacitors. It just means that you have more protection from one rare set of circumstances -- potentially with less reliability overall, since big electrolytic capacitors are known to fail, especially cheap ones.

I suspect that most SSD's are bought as OEM drives buried inside laptops and desktops where the end user may not ever know what brand and/or mode

Component manufacturers target OEMs for the bulk of their sales. They will build to the price point that may win them a sale. The $3 to $20 amount may not make a difference to a consumer purchasing one from NewEgg, but to someone who purchases in blocks of 1000 it may.

it's hard to convince consumers through a few sentences on the side of an SSD box that power protection circuitry is important to have

I disagree.

To pick an example: Gigabyte advertises on the box that they use high-quality Japanese capacitors for their motherboards. And since every. single. motherboard failure I've seen in a decade has been due to bad caps, these words mean a lot to me.

"Built-in power backup to help keep your data safe!" sounds like a good enough slogan to lure me in.

>Flash memory is accessed in blocks and only blocks. Even if you need to write to a single bit, the entire block that that bit resides in needs to be re-written. This means before you can write, the entire block has to be read and stored temporary ram. If power is interrupted during a write operation then there is a very good chance the entire block will be lost because the contents of the flash controller's ram will be lost.

You can write individual words in a flash chip.It takes longer to write than read because you have to force a bunch of electrons through an insulator.

If you want to write over existing data, you have to erase the block it is in, because you can only erase whole blocks, but there is nothing to stop you incrementally writing to unused parts of a block.

Of course it doesn't. Naming/Shaming is not allowed.I was sarcastic, of course. They don't do it, though, because it'd probably put them in a crossfire of lawsuits coming from powerful companies. Nobody wants that. They will lose simply by being bullied financially. It's all about who brings more lawyers to the table, not who's right or wrong.

I created a Reliable Writes [postgresql.org] page for PostgreSQL that talks about this and gives some known good and bad examples. Intel's 320 and 710 drives are the only two SATA SSDs still on the market that have survived the tests for clean shutdown I've advocated everyone run. They are units with a supercapacitor to enable power failure cleanup. If a drive doesn't have a battery for that sort of purpose, you will lose data at shutdown one day. And, no, a UPS is no cure, because all it takes to ruin a system on one is

Last time I checked, standard platter-based disks had the same issue -- a problem that is solved in server/enterprise environments by placing a write-cache battery in the RAID controller.

In a desktop environment I suppose one could embed a write cache battery into the SSDs to abate the issue, but in a laptop environment it'd be unlikely you'd even encounter it since you'd have to be writing data while running out of battery, in which case, you might well deserve it:)

Actually, you might be surprised at how large it would be. F=It/V. 10mS write time for a sector, 2S for a file.

You don't need enough power to finish the OS-level task...you only need enough to write out the data in the drive's RAM cache. Since that is 256MB or less on most current SSD drives (512MB is found on some drives greater than 500GB), it's not as much as you estimate.

Then, too, when you use the correct timings (10ms for a sector is about 200 times too long), you see that even the slowest SSD takes only about 5ms to write out 1MB (with an average around 3ms), that's around 0.75 seconds to flush the whole cac

These devices have an elaborate internal database for the management of block remapping. For this to survive power failures it needs to use transactional updates. Getting this right is hard - it takes years for file systems and databases to become robust. I'd guess that many devices don't even attempt to do it and the ones that do probably have obscure failure modes. A UPS is essential.

I had some original Vertex drives from OCZ that kept absolutely corrupting when my laptop got accidentally unplugged and I powered on the machine. I had to RMA them over and over and over again. I finally figured out that my battery was getting old and, although everything was functional even on battery power and it would boot, the initial large draw of power on boot must have created a voltage drop (i.e. brownout) which the SSDs weren't designed to compensate for. Within an hour of boot (even back on plugged power) they would choke, freeze the OS, and be rendered unusable from then on out.

Several SSD manufacturers are probably not engineering well for fluctuating power. Rather than fixing the problem with better engineering, OCZ simply changed their warranty policy to void the warranty if the customer is not providing proper power which, correct me if I'm wrong, I don't think rotating disk hard drive manufacturers have had that in their warranty clauses.

Well thats probably becuase you were using OCZ crap. I have never had a quality product from that company.

However that said, I have noticed the same thing with the crucial m4s I have. In one particular laptop, it keeps bricking drives becuase the battery doesnt hold much of a charge any more. Luckily, i can "unbrick" them by plugging in the power (but not data) for 20 minutes, then plugging in the data connection, then rebooting the machine. Has worked more than once.

The OCZ Vertex 2 series were prone to spontainiously bricking. The original OEM Samsungs that Dell used would exhibit all sorts of strange disk I/O issues that led to strange Windows hard locking. You would think it would be causes by CPU, RAM, Video, or chipset. No, it was the Samsungs.

When it comes to reliability, I'll stick with Intel offerings. Not the fastest things on the block, but good enough in its class. Personally, I gambled on a OCZ Vertex 4 because Newegg slashed some insane 30% off MSRP via Sh

I think you make a good point about warranty clauses, and it would be hard to imagine HD manufacturers singling out their SSDs with an inferior warranty in this respect.

Considering the paper cited by TFA won't spill the beans on which models were tested, it may be a safer bet to purchase SSDs from traditional HD makers (at least I hope that is the case with my Samsung).

I had some fun with trying to mount some Crucial M4 drives in USB external enclosures. They kept getting unmounted and the SMART block remap count kept running up, and up, and up. One of the drives outright failed and the other was at 55% spare sectors remaining when I figured out the issue. When there was a write, the current consumption from programming the FLASH chip would cause a voltage sag and the write would fail but it wasn't usually enough of a drop to make the drive reset. Once I bought the "Y

If they do that, they won't get any more free SSDs to test, and that'll impact their ability to write papers criticizing SSDs. What would you prefer? A paper biased towards SSDs too small/cheap to be useful to you, or one that doesn't name names? Anonymity is VERY important in this kind of research.

If they do that, they won't get any more free SSDs to test, and that'll impact their ability to write papers criticizing SSDs. What would you prefer?

I would prefer research to be done by someone who is not manufacturers bitch.You dont need a ton of money to test commodity hardware, the trick is to SELL stuff after the test, not take home and pretend it wasnt a bribe.

What some of folks don't realize is its the seesaw nature of many power events that's primarily behind both data corruption and SSD failure. It's a rare rack system that has its own power conditioning and UPS these days (HP NonStop comes to mind) and without it you're subject to whatever the event provides in the way of under/over voltage, spikes, drops, etc. Many times these happen in timeframes too fast for power switching equipment to react and in some cases its that stuff that gets fried first.

There is a protection mechanism that I know exists in Crucial SSDs which makes the drive appear dead after some unclean shutdowns of the drive while it performs a firmware-level integrity check of the drive. It may exist in other brands as well. Sometimes it takes 2 runs of 30-60 minutes to get the drive to re-enumerate via SATA.
I'd be curious to know if the "dead" drive was affected by this bug.

There is a protection mechanism that I know exists in Crucial SSDs which makes the drive appear dead after some unclean shutdowns of the drive while it performs a firmware-level integrity check of the drive.

I don't know if they're violating a spec or not and it's probably a life's work to find out, but that seems very rude to me. They really ought to identify as busy or something, so that they don't just scare the piss out of you. If you almost-brick an Xperia phone by scragging the bootloader so bad you can't even reflash it, whatever handles the comms is still working and lurking in the background and it will enumerate via USB with the service interface. That way you know whether you should even bother. Woul

I agree. The first time one of our engineer's laptops HD's did this, it was rather uncomfortable to say the least. I think a good compromise solution would be to have it enumerate with a "useful" drive textual model identifier like "M4 ERROR CHECKING, LEAVE ON 30 MIN" or some such. I'm sure it violates some standard, too, but it would at least give the user some indication of what is happening.

You got this too? I just ordered a Crucial M4 on sale a few weeks ago. the day after I installed and cloned it, I had the same situation where it wouldn't start. I called Crucial, expecting to need an RMA. Luckily I got an informed gentleman on the phone who told me to leave it at the failed POST screen for 20 minutes, reboot, and give it another 20 minutes, and reboot again. It worked. Supposedly it's not so much a 'bug' as an 'obscure feature'....I'm keeping my spinning rust drive around just in case.

Overall, it is a good thing. The data isn't organized linearly for wear leveling purposes, so a power outage can leave the metadata in an inconsistent state. Also, make sure you have the latest firmware on the drive.
They had a fun one earlier that caused a drive lockup hourly after the power on counter hit about 35k hrs (or some such). I've got about 2 dz M4 drives in service, so I've seen a lot of the bugs.

Power loss protection (super capacitors) was stated on four of the drives (the four least expensive to boot). Only three performed flawlessly in the unserialized writes test. Those aren't great odds. In fact only two drives passed all tests with no errors, and it wasn't necessarily the SLC "enterprise" drives, though those two also passed the serialized writes test.

In case you aren't aware, unserialized writes invalidate *every* assumption, including write ahead, journaling, even your fancy BTRFS/ZFS. His example is a database where the transaction log write was sync'd before the data page write, then after a power failure the data page is persisted but the log write is gone.

You can recover from many of the other errors or at least detect them but unserialized writes can silently corrupt data or even ruin the entire filesystem.

Obviously the metadata/dead failures are the exception... Those render the whole SSD useless.