Metadata Performance of Four Linux File Systems

Each test was run 10 times with the four combinations of the file systems (ext3, ext4, btrfs, nilfs). The test system used for these tests was a stock CentOS 5.3 distribution but with a 2.6.30 kernel and e3fsprogs was upgraded to the latest version as of the writing of this article, 1.41.9. The tests were run on the following system:

GigaByte MAA78GM-US2H motherboard

An AMD Phenom II X4 920 CPU

8GB of memory

Linux 2.6.30 kernel

The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)

/home is on a Seagate ST1360827AS

There are two drives for testing. They are Seagate ST3500641AS-RK with 16 MB cache each. These are /dev/sdb and /dev/sdc.

Only the first Seagate drive was used, /dev/sdb for all of the tests.

For all 4 file systems, the defaults were used in building the file systems. For btrfs, version btrfs-progs v0.18. For nilfs2, nilfs-utils-2.0.14 was used. Both ext3 and ext4 were mounted with “data=ordered” since this is recommended practice to prevent data loss.

Benchmark Results

This section presents the results of the testing (exploration). The results are presented in tables listing the average value and just below it, in red, is the standard deviation. This is done for each of the four combinations and each of the four file systems.

The first combination tested was for small files (4 KiB) with a shallow directory structure. Table 1 below lists the results with an average value and just below it, in red, is the standard deviation.

The first test, directory creates, had an average run time of 12 seconds for all four file systems, so the results may not be that meaningful. In addition, the directory remove test ran in about 1 second or less. Consequently, this test may not have much value.

Table 2 below lists the performance results with an average value and just below it, in red, is the standard deviation.

The second combination tested was for small files (4 KiB) with a deep directory structure. Table 3 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.

For these tests, the first test, directory creates, took about 40 seconds for ext3 (the fastest). This time is fairly small and, consequently, the results may not be as applicable the other tests which had a much longer run time. The last test, directory removes, took 11 seconds for ext3 (the fastest). Again, this time is fairly quick so it may not be as useful because the time is so short.

Table 4 below lists the performance results with an average value and just below it, in red, is the standard deviation.

The third combination tested was for medium files (4 MiB) with a shallow directory structure. Table 5 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.

For these tests, the first test, directory creates, took less than 1 second. This time is very small and, consequently, the results are not as applicable as some of the other tests. The file removes test took about 10-15 seconds. Again this is a very short time and the results may not be as applicable. The last test, directory removes, took 0-1.4 seconds. This time too, is very short.

Table 6 below lists the performance results with an average value and just below it, in red, is the standard deviation.

The fourth and final combination tested was for medium files (4 MiB) with a deep directory structure. Table 7 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.

The first test, directory creates, took 2-3 seconds, which is very short. The time for the third test, file removal, was also fairly short at 11-19 seconds. The last test, directory removes, was extremely fast at less than 2 seconds. These three results are somewhat suspect because of short run time.

Table 8 below lists the performance results with an average value and just below it, in red, is the standard deviation.

Table 8 – Results of Medium Files (4 MiB) – Deep Directory Structure

File System

Directory Create(Dirs/sec)

File Create(Files/sec)

File Create(KiB/sec)

File Remove(Files/sec)

Directory Remove(Dirs/sec)

ext3

818.30213.10

16.200.60

66,053.102,515.72

225.6035.42

1,518.00658.48

ext4

671.70147.98

18.100.30

74,607.50380.54

331.50112.06

1,842.20409.60

btrfs

886.60167.06

25.200.40

102,807.40917.56

253.2017.72

1,944.60307.20

nilfs2

852.50170.50

13.700.64

56,998.602,122,26

356.5015.50

1,637.40501.66

Discussion of Results

There are four different combinations that were tested for the four file systems. Comparing the file systems is interesting but also comparing the same file system for the different tests is interesting as well.

Firstly, let’s examine the shallow directory structure results (Tables 2 and 6). For small files (4 KiB) the four file systems performed about the same for the file create and file removal tests (the directory create and remove tests ran too quickly to be really useful). All four file systems achieved about 1,000 file creates per second or about 4,000 KiB per second (see Table 2). But medium files (4 MiB) produced very different results. For this case, btrfs was almost twice as fast as nilfs and 50% faster than ext3 or ext3 with respect to the number of files creates per second or throughput (KiB/s) (see Table 6).

However, the small file case produced 109X more files, 27X more directories, but only produced about 1/10-th the total amount of data. This points out the extreme pressure that is put on metadata performance of file systems for small files.

Second, we can perform the same comparison for deep directory structure (Tables 4 and 8). Small files (1 KiB) put extreme pressure on file system metadata performance as with the shallow directory structure.

For small files, a shallow or deep directory structure did not appreciably impact metadata performance

For larger files, a shallow or deep directory structure also did not appreciably impact metadata performance

For small files, btrfs has good file creation performance but file removal performance is not as good as ext3 and ext4 at this time

For larger files, btrfs has both excellent file creation and removal performance relative to the other 3 file systems

Log-base file systems such as nilfs2 should work well with metadata tests. But the developers are evolving the garbage collection (gc) algorithm which should improve performance.

The standard deviation, or the spread in the data was much greater for ext3 than the other three file systems. The reason(s) for this are not known.

This is the first attempt at useful benchmarks and analysis for Linux file systems using the approach outlined in a previous article around benchmarking. There will be future articles that use the same basic tenants. But be ready because this approach introduces a great deal of data into the article. But overall it does give much more information that a quick table and a conclusion “file system X is better” (usually followed by a run for cover). Please let me know in the forums if this type of article is useful.

Comments on "Metadata Performance of Four Linux File Systems"

Greetings from Colorado! I’m bored to death at work so I decided to browse your site on my iphone during lunch break.
I love the knowledge you present here and can’t wait to take a look when I
get home. I’m shocked at how quick your blog loaded on my
cell phone .. I’m not even using WIFI, just 3G ..
Anyhow, very good site!

What i do not understood is in fact how you are not really much more
neatly-appreciated than you might be now. You’re so intelligent.
You know thus significantly in the case of this subject, produced me
for my part consider it from numerous various angles. Its like women and
men aren’t involved except it is one thing to do with Lady gaga!
Your individual stuffs great. All the time take care of it up!

Hello! I could have sworn I’ve visited this
site before but after looking at many of the articles I realized it’s
new to me. Regardless, I’m certainly happy I discovered it and I’ll
be bookmarking it and checking back regularly!