Posted
by
samzenpus
on Wednesday January 04, 2012 @09:06PM
from the check-it-out dept.

diegocg writes "Linux 3.2 has been released. New features include support for Ext4 block size bigger than 4KB and up to 1MB, btrfs has added faster scrubbing, automatic backup of critical metadata and tools for manual inspection; the process scheduler has added support to set upper limits of CPU time; the desktop responsiveness in presence of heavy writes has been improved, TCP has been updated to include an algorithm which speeds up the recovery of connection after lost packets; the profiling tool 'perf top' has added support for live inspection of tasks and libraries. The Device Mapper has added support for 'thin provisioning' of storage, and a support for a new architecture has been added: Hexagon DSP processor from Qualcomm. New drivers and small improvements and fixes are also available in this release. Here's the full list of changes."

So does this mean I can start using btrfs, at least for personal workstations? I've got a new box at the office waiting to be setup, with a 120GB Corsair SSD as the main system disk, normal 2TB harddisk as backup/media storage. Will be using Debian. Should I use btrfs?

Long answer: Please! The more people exercising the code, the more bugs will be revealed, and the more confident the developers can be. But you will have to be ready for some performance regressions and data loss danger. For the brave.

Don't know about this new version, but I tried btrfs on ubuntu a few months ago, on a box I use mostly with photo and video editing software. It was slow to the point of being completely unusable. Specially for non destructive photo editing, where the software creates and modifies (replaces) small files with metadata for each action you perform on the images. Have being using ext4 since then, and everything is ok.

Depends are we talking about the ordinary 'user' here or the ordinary "user of btrfs", which is pretty extraordinary when compared with the population of general PC users. A PRENDEND bug report that I would think might be useful would be:

I have ~?TB volume composed of three drives each with a single partition of sizes X, Y, and Z. I have noticed that after I remove a snapshot really big files, in my case some VM images of size T, get corrupted, in that they seem to contain old data that would have been i

You're absolutely correct that clearly most useful will be a developer with proper debug setup, but there still might be some occasions where a normal user can roughly describe the actions that led to the filesystem crash, such as "I was torrenting like a mad", "I cleared my browser cache", "I copied some files off a USB drive".

Even without any crashes or power fails, I've managed to corrupt BTRFS in testing, with just a defrag (as recently as 3.2-RC6). I wouldn't look at BTRFS for a while, at least until it's no longer marked experimental. By all means test away, but don't assume you'll be able to get to any data you put in it.

If this is the case, whats the fucking point really? BTRFS was heralded are the replacement for ZFS, but you are seriously telling me that after all this time, you can still lose a large amount of data and end up with a corrupt filesystem after such a trivial thing as a powerloss? Really?

BTRFS has only been around for a short time and I don't believe any OS uses it as the default filesystem yet. And you're surprised that it still has problems?

And I don't remember anyone claiming that BTRFS would replace ZFS, merely that it would eventually have many of the same capabilities that ZFS has.

I had a home server with 6TB of disk using btrfs and a second server with the same amount of disk, also using btrfs for backup. I had a power failure while the rsync backup was running, and both btrfs filesytems were mangled. I managed to recover most of the stuff after a couple days' worth of work, but I definitely changed back to xfs on LVM2.

Btrfs was so much simpler, and I was able to maintain > 60 Mb/s write speeds to a set of crapy disks that only manage 12-14 Mb/s writes, now. I know I could RAID my disks to get back to those speeds, but with btrfs I was able to grow my server by replacing one disk at a time. With RAID and LVM2, it's not worth that much effort for a home media server.

Windows may not be a filesystem, but it only ships with one sane choice of general-purpose filesystem, and that choice is NTFS. Therefore, Windows == NTFS for the purpose of a discussion in the context of filesystems. Not literally, but plainly for all intents and purposes.

(Counterarguments about Windows also shipping with crappy filesystem(s) needn't apply, since Linux/*BSD ships with those too. And HPFS, the only other then-viable choice from IBM/MSFT, and perhaps the only o

I'm just one man, but I've tried hard over the past decade to find a real problem with NTFS (and I've been sternly bitten by ReiserFS and ext2/3 over that same period), and just haven't: It's worked on the many hundreds of computers I've fondled just fine, and even seems to survive mild hard disk failure with some amount of reasonableness.

What, in your opinion, makes NTFS a pain in your ass? (I ask because I'm curious and want to avoid any such scenario, not because I am predisposed to attack your observations.)

Performance can be a pain with very large files and with a lot of very small ones, and it's the people with such uses that are pissed off with NTFS - I'm not sure that can be classed as a "real problem".Any security considerations can be ignored since if you took those seriously you wouldn't be advertising something even worse in that respect than plain 20 year old FTP.

Which is why I wrote that people are pissed off with it but I don't know if it's a real problem. Some other file systems have been tweaked specificly for those corner cases anyway, and that's where I don't use NTFS if I can help it. Sometimes I can't help it and users complain until I move the stuff off the local drive onto a network drive with a different file system that even if it physically has a lower data access speed than local drives gets the job done more quickly.

Well, the file system should be loosened so it can move about the whole partition. Otherwise you lose space.

Besides, the defragmentation engine in NT, which was licensed from the folks who make Diskeeper, is extremely robust. I'm as much of a critic of Microsoft as they come; my opinion of Microsoft borders on hatred. But I've got nothing bad to say about NTFS other than to comment the need for constant defragging is a pain. But as far as robustness, in almost 20 years, I can honestly say I've almost ne

Yes that can be a real problem for people with lots of files. Years ago I did programming on Windows (thank god anymore) there were literally thousands of files in my home dir.The problem was there was this threshold of number of files which if you crossed it brought the FS to a crawl. I could notice that by deleting a few files then suddenly performance was back (except for the defragmentation induced one) once you crossed this boundary performance were back to a crawl.Add to that the windows inherent pess

The fucking point is to encourage beta-testers. Bleeding-edge users who know what they are doing and don't care about data loss are being offered the chance to test a new and interesting filesystem and (ab)use it in ways that upstream developers had not thought of, hopefully uncovering major bugs before the thing will get marked as feature-complete and enabled by default for new installs by major distros.

Bleeding-edge users who know what they are doing and don't care about data loss are being offered the chance to test a new and interesting filesystem

Amen.

fsck's only job is to make that junk that was a filesystem look something like a filesystem again. Nothing in there about making it look like the particular filesystem you used to have. fsck is not backups. fsck will not (necessarily) get your data back. fsck may eat kittens on a bad day. What fsck does hand back to you should not be trusted and should certainly be verified.

If you think that pulling most of what was/home,/var,/srv or/opt out of lost+found is fun, just remember that corrupted directory and filenames get named after their inodes. Nothing like trying to figure out of 1234567 or 1234568 was the start of the quarterly financials report.

If you are relying upon a fsck to get your data back after a power outage, you have more faith in your filesystem than you should. It's a nice validation tool, with the caveat that a False Negative means you go back to using a damaged filesystem for more fun later, rather than now.

BUT if you have backups, please do test. Having talked to the BTRFS team directly at LINUXCON, Mr. Chacon and folks are pretty cool about getting feedback. And you can do nifty things like snapshots for backups on RAID10 or thin disks on virtual machines which don't inflate during formatting.

For many filesystems, failing a fcsk means reaching for the format tools and the last (verified) backup. You are backing up everything, right?

BTRFS is journaled (the log tree), so you don't lose data due to a power failure. It just replays the journal when you mount it again.

btrfsck is only really needed to recover from something unanticipated happening. Like software bugs. The kind that you'd expect in a new filesystem. So it's absolutely not ready for prime-time until a fsck is released.

For non-critical systems it's completely usable for people who like to experiment with the new toys. I've been using it for a year on my laptop (including multiple power losses and other shenanigans) with no problems.

My experience so far:

Good:

Snapshots (and subvolumes) are a killer feature. Having hourly and daily versions of everything is wonderful. I have subvolumes for root (@) and/home (@home). If I want to roll back my entire system but keep my homedir, I can simply: "cd/snap ; mv @ @-2012.01.04-broken ; btrfs subv snap @-2012-01-02--00-40-39-apt @ ; reboot". Translated - Save a copy of my current root; create a new root from the last snapshot that was automatically created when I last ran apt-get ; then reboot (all shutdown continues in the @-2012.01.04-broken subvolume, it doesn't corrupt the new @). Killer. Feature.

Bad:

fsync is god.awful.slow. And dpkg does a whole lot of fsync. It's completely unacceptable performance, and either btrfs has to get faster or dpkg has to be a little more miserly with fsync. If dpkg could send write barriers instead of syncs it'd probably solve it, but who knows. For the time being: "apt-get install eatmydata; ln -s `which eatmydata`/usr/local/bin/aptitude ; ln -s `which eatmydata`/usr/local/bin/apt-get" . eatmydata preloads a library that overrides sync and fsync (they're simply ignored). dpkg is now screaming fast, but I run the risk of completely screwing its metadata if there's a crash or poweroff while it's doing something. So the solution is I have a/etc/apt/apt.conf.d/80-snapshot that creates a snapshot before every apt run. If something goes wrong, I just roll back the system like I mentioned under Good.

So in summary: Some good, some bad, definitely not fully baked yet, but completely usable if you're adventurous. No fsck required. Yet. Keep backups.:)

I work at a school, and lazy students are constantly turning computers off by holding the button or turning off the plug to save a few seconds. NTFS is resistant to power-loss-induced damage, but it is not entirely immune: We get a couple of machines a week with such problems. Fortunatly we have a backup partition as well, so all we need do is boot off of a USB stick and copy the backup over the damaged partition.

Ok, but the tree can go for a pout in a corner and you'll just get kernel panics on restart - this happened to me. No power outage, just a normal reboot and poof... it was repairable using the btrfs-zero-log tool.... but it wasn't a pleasant 10 minutes while I was thinking.. "What did I do, why why why did I think it was good to try out btrfs?" Of course once I got it booting again, I kept right on using btrfs:-) I love the snapshot thing... very useful

I have herd a number of recommend that for a personal desktop you could use it for non user files (ext for home and boot and btrfs for ever thing else).

The idea being that if something goes wrong you can just reinstall (30 min for me if I cache my updates).

You just have weigh up the risk and minimize the damage. Unless you are using it with something like snapper (fs snapshot and roll back) or a stopwatch you will never know the difference unless it gets corrupted.

btrfs is tanatlizing for VMs because of the copy-on-write file behaviour (i.e. "cp --reflink a b" creates b instantly regardless of the size of a), but http://lists.fedoraproject.org/pipermail/devel/2011-July/154251.html [fedoraproject.org] is still an issue, as far as I'm aware. So storing VMs, where you access them with O_SYNC, just gets slower over time until it's unusable. I'm not quite brave enough to suggest that any of our customers use it, at least until there's a working fsck.

Waiting to see the usual fanatical wars over filesystems... people calling for the death of the EXT3/4 system.

Personally the whole fanatical thing seems a bit silly - who'd have ever thought that people would lynch each other over having different options for different purposes/tasks, the very core of the whole idea of what we do and strive for. I'm fine with ext4, thanks:)

I still say we should lynch EXT3/4(even though I use it) due to it's complete/inability/ to undelete files.Because, as we all know, people/never/ manage to accidentally delete files and/always/ have recent backups handy.

People generally only make that mistake once or twice before they become a bit more clued up and invest in a backup option, even on OS's that provide undelete (Windows). Agreeably it doesn't save you when you create and then lose a file between the backup times.

It might be a nice option to have, so long as it doesn't inhibit/hinder the existing system. I think an entirely different filesystem would be a better option, something with inbuilt versioning/history.

Then what you should do is change you shell so rm is a functIon that moves stuff to the "trash" rather than compromise the on disk format of the file system so an operation "unlink" which is supposed to be destructive can be undone. Solve the problem in the correct place.

Then, one day, I was using someone else's computer. I used "rm" with the expectation that it would prompt me, but this person never bothered to set it up that way, and I had the fearful experience of worrying whether it was deleting too much. I hadn't been too careless that time, but it got me thinking. It's dangerous to use "rm" when I really mean "rm -i"; habits are strong things.

So I made a change that I still use. I now alias "r" as "rm -i". "r" by itself does not have default behavior on most computers. Now if I absent-mindedly type "r *.txt" on someone else's computer, I get "r: command not found" and I edit the command to say "rm -i".

I suppose I should have used "rmi" or something like that, just in case I am a guest somewhere that "r" was aliased to something crazy. In practice, it hasn't been a problem. I use more aliases than most people seem to; they seem to be content with the defaults. I seem to be the only one I know who likes one-letter aliases.

Hmm, I guess I might accidentally run the R statistics package someday?

LOL, yes I'm aware of that danger. Unless I'm wiping out an entire folder recursively or something, I don't bother with the -f flag. FreeBSD has a -I now (capital i) which only prompts you if more than 3 files are going to be deleted. This isn't available on Mac, but I think Linux has it as well.

The filesystem needn't be burdened with it. But if was a kernel function, you'd have the option of saving clobbered files as well as deleted files. You could also design a friendly retrieval system, without requiring a versioning filesystem, per se.

Three, only works with EXT2. Which it works perfectly well on. But nothing later.Y'know, tossing a copy of the deleted inode entries(in a different format) somewhere else on disk, to be used only when space runs low would solve all of the many, many topics out there about people losing data due to a mistake etc. Perhaps an option that could easily be disabled for the tech-savvy(See: the number of people who use the stock desktop on, e.g. ubuntu, even when a seperate desktop is available easily via apt-get).

And when your backup hasn't run yet what than? I've worked on 100+ TB of live data several PB of capacity systems. Sometimes users are dumping 10Gbps data to disk for days at a time. Than someone deletes something and woops the tape hasn't archived it yet. Admittedly a "recycle bin" probably wouldn't handle TBs of data but still scale the problem done and it is still a real issue. Downloaded a pdf and accidentally deleted it rather than the paper you just read. Oh crap and can't remember where it came from

Nope doesn't wash. Sometimes you don't know what is important to you until someone else tells you its important. Like studying for exams and someone tells you oh you should look at that file again the prof said there will be a question on it on the exam, a book you finished reading you find out 3 weeks later someone else wants to read etc. It seems pretty ridiculous to me to assume that everyone will always only delete things that they will end up never needing in the future when no one knows the future, no

That's really anything to do with the file system anyway. Recycle Bin isn't a feature of NTFS/FAT, it's a feature of Windows, and there's plenty of things available in Linux that do the same thing for EXT3/4 and I assume work fine for any natively supported FS. It's not like it's magically "undeleting" files, it just moves them to some other location, which the OS may choose to handle differently to normal directories.

Even after you remove it from there, the data's still on the disk, until that part of the

Yep and could be much faster. Scenario: download loads of files say pictures. Delete in the standard way and you have to move all those files around which is some function of size of individual files and how shotgunned across the filesystem they are. So they might be contiguous after they get written to one location all in a very short interval but not before. Versus something like a "I'm 'deleted'" bit on the file you lose a bit of space but don't have to move the file to "delete" it. Someone mounts the fi

Versus something like a "I'm 'deleted'" bit on the file you lose a bit of space but don't have to move the file to "delete" it.

How is that different from moving the directory entries for loads of files into the "I'm deleted" folder, as has been done on the Mac since 1991 and on Windows since 1995? None of the data gets moved; the inodes (or whatever FAT, HFS, and NTFS call them) stay in the same place.

None of this as you get on a windows system deleting files on a remote system being permanent and not making it to your recycle bin etc

I seem to remember at least some versions of the Nautilus file manager creating an invisible ".Trash-1001" folder on an SMB share and then "deleting" files by moving them into that folder.

That's not silly! There are two reasons for silliness. Surprise and
fear. Fear and surprise... and ruthless efficiency. There are *three*
reasons for silliness, these being fear, surprise, and ruthless
efficiency... and an almost fanatical devotion to the Pope. *Amongst* the reasons for silliness are such elements as fear, surprise, ruthless efficiency and... Ok, you're right, fanatical is silly after all.

Why? The last time I had a look at BTRFs it was at least in a VM significantly slower than EXT4.This was noticable at every corner of the OS.I would not use it for production, at least not in a scenario where I/O performance matters, but the features are neat. I lovethe snapshotting. But my personal guess is it is at least a year away from being a viable EXT4 replacement.

Do you remember when Ext4 came out? A bunch of people had data loss after system crashes. It was explained that this was intentional behavior, as specified by the POSIX standard, and the safer behavior of Ext3 was unintentional. Of course, no one cares when they're looking at a tree full of zero byte files, so the file system fanatics rose hell and got them to patch Ext4. So file system fanatics have their uses.

The first kernel I compiled was 1.2.10, I know there are people who have here who have been it longer than I, so this is not an ego-trip. I just feel old. I need doctor Carol Marcus to make me.... "Feel young, as when the earth was new."

Sort of. The actual version number of java is still 1.6, 1.7, etc, but it's always referred to (by Oracle officially and by people generally) as Java 6, Java 7, etc. I'm not entirely sure when they did that, but the guy sat at the desk next to me has a "Java 2" book. I guess they got to 1.2 and wanted to make it sound big, but didn't want to change the version number?

This looks to be a really strong, likely to be long-supported, kernel. Providing that the Googleification of the TCP stack doesn't hurt local 1-10Gbps performance, that is. Have a care if you do your own kernel compiles... the whole Ethernet driver subsystem has been merged together.

Engineers kept using ext3 (or reiserfs...) for a while because ext4 was "too new".
Now that it's stable and used, is it safe to extend it with such a powerful "block" option, and risk a potential regression?

Very good news: standard Linux box will read my NAS' drives without using specific package and user space FS tools.
The NAS has a Debian Sparc system with ext3 16k blocks. Recovering the data when the NAS is dead (or has problems) is always a concern. Knowing I will be able to start a PC with Linux Live CD and plug those disks to recover my data is a relief.

I hope this NAS will also accept USB drives with ext3 16k FS made by x86 Linux (it doesn' read ext3 4k FS). I've prepared some with thorough blocks rw

That was Ubuntu, as I recall, and not necessarily the greater kernel community. I'm sure they'd rather play it safe and have a slightly more power hungry but stable system than risk crashing people's systems because OEMs are incompetent and can't report their shit properly.