Article Index

File System Shootout

I haven't written about the kernel mailing list before, but something
with direct correlation to clusters recently came across the list.
On October 24, 2003, Mike Benoit posted an email to the kernel mailing list
announcing some updates to his file system
shootout. Mike used two
file system benchmarks, Bonnie++ and IOZONE, that are designed to
test hard drive and file system performance. Mike had previously
posted to the kernel mailing list when he first had file system
performance results. There were several suggestions for tuning
specific file systems, and a request for using better hardware.
He made the suggested changes and retested. He now used an Opteron
240 system with 512 Mb of RAM and a PII/450 with 512 Mb of RAM for
the shootout. He tested EXT2, EXT3 with several options, XFS, JFS,
ReiserFS v3 with two options, and Reiser4 with two options, all
using several recent versions of the 2.6 test kernel. He ran the
tests 3 times each on both a SCSI disk and an IDE disk and presents
all of the results in a nice tabular form with some results
highlighted.

Why are file systems so important to clusters? There are many
clusters applications that read and write data to a file system.
This file system can be local (i.e. in the node itself) or part
of a central file system (e.g. NFS). Also, applications could be
using a distributed file system like Lustre or a high-speed parallel
file system like PVFS (Parallel Virtual File System) or GPFS
(General Parallel File System). For all of these configurations,
applications that are I/O (Input/Output) bound, spend a great
deal of time writing to file systems. Hence, file system
performance is important to them.

Mike has several interesting observations. First, in his opinion,
based on his benchmark results, XFS and JFS give the
best bang for the buck. That is, they are close to EXT2 in
performance with a small amount of CPU usage. It's interesting to
note that journaled file systems are slower than the the
non-journaled file system EXT2. So if you don't mind an occasional,
potentially long file system check (fsck), then EXT2 is still
pretty fast. It might be very useful for relatively small
read-only file systems.

For applications that are bound by I/O performance, he recommends
Reiser4, or XFS, or ReiserFS v3. Remember though, that Reiser4 is
still experimental and has not yet made it into the 2.6 kernel.
However, according to Mike, the results are very encouraging. He
also mentioned that if your file system uses lots of small files,
then ReiserFS v3 is the way to go. However, if your file system
has medium to large size files, then he recommends XFS. Mike
goes on to mention that if you are CPU limited, he recommends
JFS.

Finally, Mike makes some observations comparing SCSI disks to IDE
disks. He ran the tests on a SCSI disk that was running at
10,000 RPM and an IDE disk running at 7,200 RPM. He found that Reiser4
had about a 50% boost in speed using SCSI disks compared to IDE disks.
Both JFS and EXT3 gained the least speed by moving to the SCSI disks,
only gaining about 5-20%. He also mentions that in one case JFS
actually ran slower on a SCSI disk than on an IDE disk. He finally
suggests that a 5 times cost difference for SCSI drives may not be
worth the cost if an average improvement of 20% is all that is is
seen over IDE drives.

This article was originally published in ClusterWorld Magazine. It has been updated and formated for the web. If you want to read more about HPC clusters and Linux you may wish to visit Linux Magazine.

Jeff Layton has been a cluster enthusiast since 1997 and spends far
too much time reading mailing lists. He occasionally finds time to perform
experiments on clusters in his basement.