I have encountered several USB sticks with a corrupted filesystem in only 2 years. In a Windows-only environment (Vista and newer), what can be done to reduce the chance of filesystem corruption and dataloss on a single USB drive?

Which filesystem is the most robust?

Which technologies or labels (xyz certified, etc) indicate that USB sticks supporting them are less likely to become corrupted?

We don't do product recomendations on Superuser so specific manufactors won't be listed. Have you determined if the data corrutption came from wear on the device itself or because of it being abused ( i.e. being removed without being ejected first )? I only suggest NTFS, data corruption can be handled a great number of ways, data duplication with verification is only one way.
–
RamhoundMay 13 '13 at 13:07

@Ramhound I don't know if that would be better, since NTFS doesn't inherently protect against data corruption. FAT32 might be better for flash memory, since there's no journal (and thus less sectors are written to over the life of the drive; I believe this is why most manufacturers suggest FAT32). Might be best to stick with your second suggestion, and either duplicate & verify, or checksum & generate some recovery archives.
–
BreakthroughMay 13 '13 at 13:50

@Breakthrough - The use of FAT32 would limit the filesize of a given file. Without understanding the cause of the data coruption in a case like this, its tough to say, it wouldn't happen in the future with the FAT32 filesystem.
–
RamhoundMay 13 '13 at 14:03

@Ramhound agreed there, the 4GB limit is a huge pain. I certainly have used NTFS on some of my own USB drives, but I think it's still worth noting that doing so will probably decrease the lifespan of the drive (depending on the average size of file you write to the drive over its' life).
–
BreakthroughMay 13 '13 at 14:07

The devices were from at least 2 manufacturers (no brand) and could always be reformatted and used again. Before restoring, it was always possible to extract the majority of the data, although some of the filestructure got lost and some files were corrupted. It is highly likely that they were ejected properly.
–
PeterMay 13 '13 at 14:25

5 Answers
5

What can be done to reduce the chance of filesystem corruption and
dataloss on a single USB drive?

Commonly used filesystems like FAT32 or NTFS don't store any data validation information (only on the internal filesystem itself). Keep backups of your data, validate the data with checksums (you can generate MD5/SHA1 hashes for your files only to check if any data has been corrupted), and/or store recovery archives.

And lastly, regardless of the filesystem, you should always properly unmount the drive. This ensures that any existing file reads/writes are completed, and any read/write buffers have been flushed.

Which filesystem is the most robust?

Robustness comes at a price - compatibility. Arguably, you'd want a filesystem with built-in data validation and checksumming (or redundant data) like ZFS, but that isn't very portable with Windows/OSX. If performance is a concern, you might want to try exFAT, which appears to be supported in most major operating systems out of the box or with some slight configuration.

Which technologies or labels (xyz certified, etc) indicate that USB
sticks supporting them are less likely to become corrupted?

Anything that keeps flash memory alive longer, most notably wear leveling and over provisioning. If the drive supports wear leveling, a larger drive will keep more available sectors in the case some wear out.

At the end of the day, flash memory doesn't last forever. All current flash memory has a limited number of read/write cycles, which inherently causes data loss over time. You can mitigate this risk by taking regular backups, and validating your data with checksums to determine when a file has been corrupted.

It's also possible to use a filesystem with built-in data integrity and recovery, but these are uncommon in many non-UNIX environments as of writing this. They also may be slower and actually wear out the drive faster, due to the requirements of storing additional checksums & redundant information for each file.

For each case there's a solution, you just need to weigh the portability/integrity/speed considerations.

"filesystem with built-in data integrity and recovery" - are there some that work on a version of Windows, without an initial setup on the machine (driver installs, etc)? How does the journalling of NTFS help?
–
PeterMay 13 '13 at 14:35

@Peter the journal makes sure each file write is a transaction - it either finishes completely, or doesn't finish at all. Ideally, if a file isn't completely written, nothing is changed, and the old copy still exists (as the journal wasn't updated). If it so happens that the journal gets corrupted, at least you then know that something is wrong (and you can attempt to recover it) instead of working with data which might be corrupted. The journal helps to ensure that the filesystem is still valid and functioning.
–
BreakthroughMay 13 '13 at 14:43

Sorry for the question, I don't know much about the topic so I might not make sense: is it correct to say NTFS has atomic writes and that the FAT family has not? And therefore that NTFS would only be corrupted by hardware failure, corrupt drivers, or malicious intent?
–
PeterMay 13 '13 at 14:47

@Peter new versions of the NTFS file system (Transactional NTFS) are indeed atomic, however older versions of NTFS are not. Even so, the older versions would still better protect against filesystem corruption because the journal keeps a log of the most recent changes to the filesystem. If you're curious, I recommend reading the Journaling File System article on Wikipedia.
–
BreakthroughMay 13 '13 at 14:51

If you use your drive on multiple operating systems you need to use FAT32 because it is the most compatible filesystem, but also the most unreliable. Any of the above choices are a better option if compatibility is not an issue. Note: if you use Linux and Windows then you can use NTFS for both computer filesystems and your flash drives, but if you use Mac OS X then getting NTFS working is a pain and not worth it.

Brand - never base any purchase ever on brand

Technologies - @Breakthrough listed some that you could look for, but just about all flash drives today will have these features or some other proprietary alias for them that isn't worth looking into

In general, don't spend too much time looking into it. As @SethCurry touched on, redundancy is always the better answer for keeping your data safe. Any storage device can and will fail eventually, so you don't want to get comfortable with one solution.

The most common reason flash drives get corrupted is impatience. I often refuse to wait to eject flash drives, and I know I'm not the only one. (In my defense, I also tend to make sure nothing critical is only on a flash drive, and you should, too.)

Drives get corrupted when you don't safely remove them because of something called "write cacheing." Essentially, write caching is a feature that improves write speeds. Rather than writing each request as it is received and forcing you to wait, your OS will cache these requests, and fulfill all of them in one fell swoop. When you tell your computer to safely remove or unmount your flash drive, you essentially warn the OS that you're going to remove it, so it writes all requests in its cache to the disk, and tells all background programs to stop accessing it. If you don't wait, you could have items waiting to be written to disk, which could result in a corrupt filesystem.

As for format, I personally prefer ext4 for my flash drives. For Windows, I would say go with NTFS, as ext4 tends to cause problems in Windows. NTFS supports large files, and journals, so it will work pretty well. Filesystem is largely a personal choice, and typically anything that is less prone to corruption is also going to be significantly slower. ZFS is becoming popular, though I don't know whether that works on Windows, and I don't know whether it can be put on a flash drive.

In terms of brands, I don't find a large difference in quality from one to the other. Some have better protection for the connectors, some certainly feel less 'flimsy' (although, surprisingly, I've found that the flimsy ones break less often). I usually just use whatever is cheap.

You should recognize that nothing important should ever be kept solely on flash memory. USB sticks are too easy to lose, step on, or drop into the toilet, etc. Important data should be backed up and kept on at least two distinct drives, and preferably in at least 2 separate physical locations (think fire risk, flood risk, etc).

I removed the first part of your answer, because everything else is correct. Flash memory is indeed subject to physical damage, albeit in terms of electrical signals rather than gravity or kinetic energy. Software usually can't corrupt a flash drive (see Can some software physically damage hardware?), only the filesystem (and if using write caching as you said, possibly any outstanding file writes).
–
BreakthroughMay 13 '13 at 14:18

I didn't mean to say that it was impervious to physical damage, simply that it is less common (because I drop my drives far more often then I accidentally electrocute them), and by software, I meant filesystem corruption, though I suppose you are correct in that I should have made that distinction more clear. In the software regard though, wasn't there a virus made at one point that was able to destroy hardware through overclocking and increasing the RPMs of disks? Not a point of contention, as it was an isolated event, merely a slight correction.
–
Seth CurryMay 13 '13 at 14:29

1

By default current versions of windows disable write caching for flash drives by default; so impatience yanking is much less likely to be a problem now than it was in the XP era.
–
Dan NeelyMay 13 '13 at 14:53

@DanNeely I believe this is correct. Note that for internal hard drives, this feature is usually enabled (although even in power loss, the device may be able to flush the buffer if power is restored).
–
BreakthroughMay 13 '13 at 14:55

It is supposed to, but according to a couple of article's I've seen lately, sometimes the flash memory isn't tagged as removable memory, so write cacheing is still enabled unless you manually disable it. That also doesn't solve the issue of programs currently writing in the background.
–
Seth CurryMay 13 '13 at 14:55

One thing is not mentioned in the answers yet, and that is the type of flash memory used. This will only be a criterium when purchasing a Flash memory device.

There's SLC (Single Level Cell) and MLC (Multi Level Cell) flash memory.
MLC technology was developed to increase the density of data that can be stored.
It stores more than one bit per cell, usually two. The way it does it is by storing four voltage levels in the cell. That gives you two bits of data for every cell.

There are a couple problems with MLC.

MLC has about a factor of 10 fewer write cycles that it's able to handle. It can do 10,000 writes before it begins to have problems - whereas SLC can do 100,000.
As the cell's ability to store degrades, with MLC the ability the distinguish the levels will decrease faster than with just two levels.

Although all of these technologies use ECC, Error Correction Code, there's a limit to what it can do. Also, the wear leveling, as mentioned in an other answer will not 'fix' the difference. Maybe there's still some really cheap USB sticks that don't 'wear level' but you'd be hard to tell from the outside.

Also, whether a device uses SLC or MLC will be hard to tell. In the race to cram ever more data into the same 'surface' I assume that most manufacturers will have switched to MLC.
If reliability is a prime factor, maybe you can look around and still find some SLC memory devices.

The management of the writing is different: if you select "Quick removal" (default), Windows tries to write the data immediately on the USB pen, while using "Better performance" option the system will cache data instead of writing and will flush all only before unmounting the volume using Safely Remove Hardware (usually by left-clicking on the USB icon in the tray).

Be aware that with "Better performance" option you can loose not written data in case of unplugging of the stick without using the safely remove hardware way or in case of sudden shutdown of your machine.