XFS vs Reiser

Last night, Christer and I were playing with the Ubuntu 8.04 installer, trying to break it, and get any last bugs reported before final is released tomorrow.

What I noticed last night, during the install really surprised me. ReiserFS is a screaming filesystem compared to XFS. After doing the install 3 times on 2 identical machines (Dell Precision 490), in every case, ReiserFS was nearly twice as fast completing the install. Didn't matter how the disk was partitioned either. It is a screaming speed machine. Further, launching applications, and playing with KVM, it was also noticeably faster than it's XFS competition.

However, XFS did have much better storage management in the way of sqeezing out every last byte on the drive. In fact, on bare bones installs, XFS gave me 10% more storage space than ReiserFS.

Conclusion? If you're after speed, from what I gathered last night, ReiserFS screams. If you're looking for maximized disk space, XFS is the clear winner there. Was the disk space gain worth the wait of the install? Yeah, probably. But man, post-install, launching applications with ReiserFS was still noticeably faster than XFS.

I'm curious about these, and will be benchmarking some more in the future. Expect follow-up posts on this topic.

Filesystem benchmarks are always a touchy and controversial subject. I've spent a year of my life chasing after all the Linux filesystems in terms of performance. Reiser is *REALLY REALLY FAST* at metadata operations (i.e. the creation and deletion of millions of new files, such as installing an Ubuntu system) however it has many drawbacks:

(1) DEAD upstream, DEAD migration path. The founder doesn't care about it anymore, and the only guy maintaining reiserfs in upstream Git is one SUSE developer to support SLES bugfixes. We can only guess what happens when that version of SLES EOLs. There is also no migration to a newer filesystem other than reformat and restore.

(2) Metadata latency: Metadata operations lock up the whole kernel. Even if you have an entire MP3 in memory, removing a large folder can cause your music to skip, games to be choppy, etc because the kernel does not allow anything else to multitask while that goes on.

(3) Performance degrades over time: A 1-month-old reiserfs volume can be an order of magnitude slower than before due to tree packing / balancing algorithm issues. This is addressed in the reiser4 rewrite, but heavily affects reiserfs3. Many seasoned reiserfs geeks have scripts to rebuild certain parts of their filesystem with mkreiserfs periodically.

(4) Heavy file fragmentation with large files. A 4GB DVD torrent was finished in 4,000 or so individual fragments and read back at 8MB/s on a 40MB/s drive. Even copying that file over itself only yielded a small improvement, while under ext3 a copy-back will yield raw disk performance and XFS doesn't even fragment the file to BEGIN with.

With that said, I think you also hit XFS at its weakness: metadata operations. XFS is very slow at creating files, very slow at deleting files. Well, that's not entirely true: In ext3 and reiserfs, deleting files is an O(n) operation: Time taken to delete a file increases linearly with the size of the file. In XFS, it's nearly O(1): deleting a 50TB file and deleting a 1KB file take the same amount of time (in case you're curious, deleting 50TB worth of 200GB files in ext3 apparently takes 2 weeks on a decent SCSI storage cluster), but that constant amount of time sucks for small sized files. This is the effect that you see during an installation.

However, XFS does have unique advantages:

(1) Very nice handling of large files, as mentioned before. If you work with 200+MB sized files regularly, XFS is your filesystem!
(2) Only filesystem with online repacker. XFS has a command xfs_fsr for "defragging" online filesystems. This can be helpful with slowly torrented or other large files that can become fragmented enough to impact performance.
(3) Only filesystem with online dump tool: xfsdump is awesome for backups and that alone is why I choose it personally.

It does suffer from poorer handling of bad blocks (kernel panic) though, so it shouldn't be used on hardware that is prone to such defects.

I've been a ReiserFS fan for years, since building my first in-house mail server from Gentoo and Qmail, many years ago. A few tips on its use (the first for ANY filesystem):

Check your hard disk before committing your personal data to it. Run badblocks shortly after installation, and if there are any, consider trashing the disk. If you can afford the time to set it up, install smartmontools to use modern disks' ability to warn you if the disk is going bad.

When mounting a partition that boots, (/ or a separate /boot partition), use the -notail option. This prevents small files from staying absorbed in the filesystem, which boot managers don't like. You can also use it for partitions you want maximum performance on at the expense of disk space. The old Gentoo docs used to specify a separate /boot partition because of this. That's too much to maintian for me. I use a / partition with -notail, but a separate /home without the option. Perhaps next time I'll use -notail on /home too, but my systems are working great for now.

Most filesystems (I don't know about XFS or other esoteric ones) bog down when approaching maximum capacity. I've found (through observation only, no benchmarks) that somewhere between 90% and 95% full the disk screaming stops and my own screaming starts. I set a monitor (I use monit, but a quick cron/bash script would work well) to warn me at 80% full. Then it's time for me to start purging back to 60% or start planning to increase capacity.

Note that we're NOT talking about Reiser4, we're talking about ReiserFS.

Not long ago, there was a guy on Linux kernel planet that blogged about creating a new filesystem. He did exhaustive benchmarks on the main Linux filesystems to see what he was up against. Here are some non-biased benchmarks for you to look at:

reiser3 and resier4 get pretty impressive results in a lot of areas. From experience, reiserfs is the WORST filesystem at recovering from hard shutdowns. It is more likely to corrupt itsself and require an fsck than any other filesystem.

Interesting Jeff. In my experience, using reiser for several years, it is the best at recovering. Actually, I've found that it's better at recovering than xfs. I built 2 machines with xfs and had multiple corruptions, especially with my firefox profile for some reason. I've never lost a byte to reiser.

XFS is not particularly suited for a desktop machine ... man it felt slow.

Err, when your reiser requires a fsck, good luck It's about the MOST sensitive fscker to random data on the disk and easily gets side-tracked and rebuilds a trash B-tree and calls it quits.

XFS is not designed to be a desktop filesystem where a good proportion of your files are small and constantly created/deleted. It is good for a storage partition for your multimedia content though -- possibly the best suited for the job.

Unless you have redundant power, I'd stick with with ext3. Like Kees, I've seen XFS corrupt data on power failure. The FAQ on the subject used to say "If it hurts, don't do it" as if the only reason this happened was people forcing power off.

If you're really after benchmarking etc, it might be informative to the general Linux using public to examine multiple versions of ext4, as we can expect the performance characteristics to change over time.

I plan on reformatting my HDD next week and starting anew when stable Hardy comes out. As such, it's one of my goals to try out new filesystems to fit my needs. Previously I've been dual-booting, so I had ext3 for Ubuntu partition, and NTFS for data. But, I'd like to try ReiserFS for the Linux partition, and perhaps XFS or JFS for the data partition. I use MythTV and store a lot of large video files on the data partition, so I've heard XFS and JFS are both good options. Is one better than the other?

I have to agree with jdong and most of the others about XFS. It's certainly not good (possibly very poor) for regular desktop filesystem partitions, but I've been using it for years on my storage drive(s) without any problems. It's varied from 200gb drive up to a ~1 tb RAID 5 array.... holding movies, mp3's, backup tar archives, and other large files it excels greatly at. Never had any corruption issues in the past 5-6 years I've used it. It also excels at very large partitions and guaranteed I/O rate. Supposedly JFS is somewhat decent at this as well but cannot give any real experience.

I've actually heard ReiserFS was the worst for stability issues although this was at some time ago. Have not used it but know it's fast for large amounts of small files especially with lots of creation/deletions.... /var or /tmp perhaps?

Personally use XFS on my fileserver RAID array (running NFS) and ext3 for all partitions on all other boxes. Highly recommend against using ext3 for any large partitions, tried it - it's crazy slow.

Update 4/29/08: John Dong (jdong) has enlightened me regarding the waning support of ReiserFS. The website of the developing company, Namesys, has now gone offline. OpenSUSE did have good reasons to stop using it as the default filesystem, and I wished I'd run across their reasoning earlier (link below). I'm reevaluating, and will probably start using Ext3 instead.