Just a quick question, but why are there so many file systems still competing and in use today? (ntfs, fat32, ext3(ffs), etc)

It seems that file system designers could agree upon the best aspects of each type of system and implement a "best" filesystem, no? Just a thought, since these filesystems have been around for a while now, and it should be at least somewhat apparent which ones have good qualities over others, and we could just combine the good in each and create an ultimate system that is much better

closed as not constructive by Mark Trapp Oct 12 '11 at 7:39

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
If this question can be reworded to fit the rules in the help center, please edit the question.

38

Should we try to combine a Ferrari with a backhoe too? Design is based off a series of trade-offs; perfect doesn't exist.
–
PubbyOct 10 '11 at 6:20

We'll do that about the same time that we make a camera that has a big, long high-quality lens, and is really lightweight, and fits in a pocket or purse, and is very inexpensive. The idea that you can take all the good parts and combine them into one best thing doesn't actually work out in practice. A list of good things is either incomplete or full of items that contradict each other.
–
Eric LippertOct 10 '11 at 15:03

11 Answers
11

Let's think about the specifics here for a moment, using examples you've cited:

ntfs - Proprietary to Microsoft. Anyone who is not Microsoft cannot use this, therefore would have to use/create something different. Now, if you are Microsoft, you want to use this over FAT because of the issues of the next bullet point.

fat32 - Not sufficiently modern. The maximum file size is 4GB. Directory entry lookup is O(n). The allocation table is a linked list, rather than something more efficient like an allocation bitmap (where it's really quick to find contiguous free space). Does not support permissions. Does not support hard links or symbolic links. Does not support journaling.

ext3 - This was an extension of ext2 mainly to support journaling.

So, it seems there are a few reasons:

An earlier filesystem lacks something. In the case of FAT it is lacking a lot: both in terms of (1) features and (2) performance. In the case of ext2 it did not have journalled updates, so recovering from a crash took more time.

An existing filesystem would probably do, but it is not yours. (eg. NTFS if you're not Microsoft). In this case you don't really have much choice but to come up with your own.

Nitpick: actually, the maximum file size (according to the official specifications published by Microsoft itself) for FAT32 is 2 GiByte, but nobody, not even Microsoft implements it that stupidly :-)
–
Jörg W MittagOct 10 '11 at 8:11

15

Nitpick 2: the NTFS file format was invented by Microsoft, but it is a published format and there are no known patent encumberances. Various non-Microsoft operating systems implement NTFS, including Linux.
–
Stephen COct 10 '11 at 12:38

6

Xfs - designed to allow for quick streaming retrieval of massive files (designed for video workstations). ZFS - a filesystem for the 21st century. ReiserFS - an attempt to create a "killer" filesystem. Each filesystem has it's strengths and weaknesses, each was created to fulfill a certain need.
–
Timothy BaldridgeOct 10 '11 at 13:47

@StephenC Are you sure about that? As far as I know NTFS is not documented. Most open source implementations warn you that there will be severe damage if you try to write to them using that driver when Windows has left the disk in certain states.
–
asveikauOct 10 '11 at 17:13

There are trade-offs. For example if you want a journaled FS, you pay (efficiency, complexity, etc.) for it but get something out of it. Some don't feel the need for a journaled FS and don't want to pay for it, some do. Same with other "features" of the FS.

When was the last time you decided which FS fits your specific needs? I'm more on the coding side than on operations, but i never saw such a decision. Especially, since there is often no real choice: MS->Ntfs. Linux: Ext(latest), maybe Reiser.
–
kepplaOct 10 '11 at 13:40

@keppla: Decision are often made by the OS designers. File systems are not balkanized by the end users.
–
ZanoOct 10 '11 at 14:15

2

@keppla: Any time I decide to format a flash drive, I turn off journaling and decide whether it is for my own usage only (then ext2, because it has POSIX file permissions) or for sharing with others (then fat32). Yet for hard drives I tend to use NTFS (when on Windows) or ext4 (when on Linux)...
–
lioriOct 10 '11 at 15:58

1

@keppla Almost all Linux systems come with the capability to use several different filesystems, not just ext[2..4]. When I set up a Linux box, I look at the purpose of the box. For instance: my Linux DVR machine runs JFS on the Video Storage drive because it handles large files much more gracefully than ext3 and uses less CPU overhead than XFS. But for the user accounts and system files on the same machine I run EXT3 because I am dealing with smaller file sizes.
–
jwernernyOct 10 '11 at 16:42

@keppla: Additionally, on Mac OS X too you can have a choice between various combinations of capabilities of their FS. Moreover, originally I intended the tone to be that of someone who designs the FS. Because, the OP asked why doesn't a designer design a FS which has the best of everything. And what I try to point out is the best is relative to where it is going to be used. Some places a feature is useful some other places the same feature is an unnecessary overhead.
–
Jungle HunterOct 10 '11 at 17:03

There can't ever be one "best" of anything because there are so many opinions about what "best" is. The decision is specific to the needs and limitations of the user. Designs are always based on their ability to fit within constraints.

A basic mobile phone needs to store a few hundred contacts, text message history, and a few small apps. Does its filesystem need to support a hierarchical directory structure on multi-terabyte drives in a RAID configuration? Is there sufficient RAM on the device to run such a filesystem? Does the filesystem need complex ACLs? Probably not -- for all of these questions -- so a simple, resource-sipping filesystem would suffice.

Companies will also develop different products to maintain a competitive advantage. For instance, Apple touts the ability of its HFS+ filesystem to track which files have changed recently so that backups are quick. On the flip side, the drivers for a floppy disk filesystem (FAT) can fit in just a few KB of memory.

Consider FAT for a moment: its support for long file names is kludgy (to put it nicely), and searching through files in a directory is linear so it gets slow very quickly if a directory contains a lot of files. At the same time, it has a bare minimum of metadata to raw write speed is very good, and since it's so simple over all, code to implement it can be quite small.

Something like ext2 or ext3 adds a lot of features and capabilities that are simply absent from FAT. Searching for files is also quite a lot faster. At the same time, raw writing speed is probably a bit slower, and the code to implement the file system is undoubtedly quite a lot larger.

Just a quick question, but why are there so many file systems still competing and in use today? (ntfs, fat32, ext3(ffs), etc)

It seems that file system designers could agree upon the best aspects of each type of system and implement a "best" filesystem, no?

Let's suppose that there weren't tradeoffs, and file system designers did implement a "best" filesystem, free from patent worries, and released as dual-licence BSD/GPL so that it would be acceptable to MS and Debian alike. What makes you think that the other file systems would vanish overnight?

I don't think anyone's used FAT32 on a new hard drive for 10 years, but it still persists as the de facto standard for formatting USB drives, SD cards, etc. Camera and mobile phone manufacturers have tried and tested firmware for using it. Arduino hobbyists have stable libraries for using it. They're all going to need big incentives to change.

And then you have the issues of backwards compatibility with older OSes (especially Windows, whose users won't want to install new filesystem drivers).

As is so often the case in computing the answer is (a) because of historical circumstance and the need to maintain backwards compatibility and (b) because some methods are better suited to some tasks than others.

On (a) you need to remember that the "Winchester drive" - I am just about old enough to remember them being called that - (what the rest of the world calls a 'hard drive') has only been around for about half the time of electronic computing and even then it has not been accessible to most users for even that long for cost reasons. The FAT file system worked well on floppy disks and also on the original small hard drives as it was reasonably efficient and required low overhead. Once it started to be used - and its use spread widely because it is simple to implement - manufacturers could not tell its users that their old data was suddenly invalid.

Similarly, for Linux users, say, a stable NTFS driver was a long time coming, so keeping devices formatted as FAT meant they could be read and written across multiple systems.

On (b) - think of the differences between a system that, say, stores, billions of text-based database records and one that stores DVD-length media files. For the databse each record could be very small - perhaps just 30 or 40 bytes and certainly a filesystem that allocated a whole 'segment' (however you want to define that) of disk is likely to be wasteful of diskspace. Not so with the DVDs - bigger 'segments' (within reason, obviously) are likely to be highly efficient in space terms.

Yet another example of why there can never be a perfect file system for everyone: HDDs and SSDs have very different read/write access characteristics. An SSD-optimized filesystem would probably work best by fragmenting files like crazy but with every fragment the page size of the SSD itself; this would perform terribly on an HDD. An HDD-optimized filesystem tries to keep files as unfragmented as possible and even puts frequently-used files in the "hot" area on the faster-spinning outer portion of the platter; these characteristics wouldn't help SSD read speeds at all, and puts a huge encumbrance on how they are written to.

I think a very important fact is missing. Most of the time programmers tend to think that their way of doing something is superior to all other ways and therefore they look at a File System design and come up with problems and solutions that seem to be more general, elegant, fast, right... And if this feeling is strong enough they are free to build their own file system.

This leeds to competition, fragmentation, confustion and I hope in the end better solutions and more options to choose a matching solution for you.

Here's another concrete example of why you would need a separate file system or extend the functionality of an existing FS.

Say you are a database vendor and would like to reduce the complexity of managing striping / mirroring of data to improve IO. You will feel the need to extend the functionality of basic file systems as Oracle did with ASM (Automated Storage Manager) which is like a Logical Volume Manager.

In your list you mention an old filesystem used because a better one is no available however it is fast.

There are other filesystems. I heard google filesystem is mostly for fast duplication/redundancy if one harddrive or server goes down. I remember hearing of another filesystem made for many small files and to be used on a system for lots of request on small file (thumbnails).

Essentially they have difference goals and or may be propriety vs open source.