File System Benchmarks: UDF, Btrfs, NILFS, ext4

Florian Berger, 2016-05-10

Benchmarking UDF, Btrfs, NILFS, ext4 on Linux sees ext4 ahead, and yields some insights regarding UDF and NILFS.

A couple of weeks ago I found out about NILFS, a "continuous snapshotting filesystem". As I had recently started to toy with Btrfs and also been using UDF for USB drives for a while, I thought it was about time to put all of these to a test.

Benchmark Setup

To put a possible disappointment first: I was not at all interested in file system usage patterns by databases, mail servers and the like. I wanted to find out how these file systems perform in day-to-day /home usage, and also on removable drives.

Also, I was lazy. :-)

So I took some USB drive that was at hand, it happend to be a Toshiba 16 GB. I created three partitions ~4 GB each, using my 4.3.6-ck desktop Gentoo Linux system.

Then I wrote a shell script which would perform some really simple write-read-remove-write-remove tests:

write_files

This test would write a set of files, ranging from 256 MB to 512 kB. The files are created by reading from /dev/urandom. The total amount written is about 1 GB.

hash_files

This uses OpenSSL to compute a SHA512 hash of all files, reading every bit of every file in the process.

fragment

This test creates a couple of directories and fills them using a modified write_files routine. It then repeatedly removes and re-creates such directories, in a first-in-first-out manner. The routine will fill about 75% of the partition in this process.

removing

This test simply clears out the partition by rm'ing everything.

A partition would be mounted before each test, and unmounted after.

The 'time' command was used to time execution and unmounting times.

I planned to do some more advanced tests, such as HD image access, parallel reading and writing, but see "being lazy" above.

Having tested the three file systems, I was curious how ext4 performed alongside them, so I tested it as well.

All tests were performed 9 times, with erroneous runs discarded.

Results

The results will be reported as the mean time taken for the operation, the mean time for the unmounting after the operation, the sum of these times, and the standard deviations for operation times and unmount times.

Regarding operation and unmount times, smaller is better.

The standard deviations (SD) give an idea how consistent and predictable the file system performs. The closer to zero, the better.

The absolute times are not significant, as a different drive or connection may yield far better results. Therefore percentages relative to the respective loser are given for each measurement.

write_files

In raw operation time, ext4 leads this one, closely followed by UDF and Btrfs. NILFS is way behind. Then again, it shines with an impressive unmount time. But also constistency-wise NILFS is really bad here, with an SD about 15 times larger than the winner. UDF, overall second in this test, also has a nice predictability. Btrfs performs so-so.

hash_files

This is a much closer call, with a total SD of only 6 seconds. Btrfs wins by about 3 seconds, while UDF and ext4 rival each other by split seconds. Of these two, UDF leads in terms of consistency. NILFS, producing one large spike of an outlier in the data, clearly loses this one.

fragment

Of all these very simple tests, this one is certainly the one most closely resembling actual day to day file system usage, and thus the most interesting one. The winner is ext4 due to its fast operation and reasonable unmount time. Btrfs is almost a minute slower, NILFS two minutes, and UDF performs worst. NILFS has to be credited for the quickest unmount time, and an surprising overall consistency; no one of the others comes even close. UDF, overall loser, at least is as consistent as ext4. In that respect, Btrfs proves to be the least predictable candidate.

removing

This one had not been planned as a test in the beginning, but I decided to record the times anyway. The results are intriguing. ext4 wins with a combination of operating time, unmount time and operation consistency that the others can't even dream of. Btrfs follows up close, and is still blazingly fast. NILFS comes far behind, with about 300 times the operation time of the winner. UDF, while not too bad at raw operation time, very consistently takes about two and a half minutes—let that sink in—to unmount the partition afterwards.

Verdict

Overall

ext4 is the clear overall winner for the operations tested. Taking consistency into account, it generally takes the shortest time to operate and has competetive unmount times.

Btrfs comes out second, with an overall fast operation making up for less optimal unmount times.

UDF and NILFS follow far behind, with UDF a bit ahead. NILFS' impressive unmount times, which in total rival the winner ext4, can not compensate for its slow and unpredictable operation.

Now, as one wouldn't pick UDF or NILFS as general-purpose file systems, some remarks taking into account the special character of each.

UDF

UDF has been designed as a file system for optical media, and is famously deployed on DVD and BluRay discs. It was not designed as a server or desktop file system. Then again, being an open standard, natively cross-platform, and supporting large files and long filenames, it is also a convenient format for USB drives. Handing a hard disc image over from an MS Windows PC to Linux? Archiving files from a Mac? UDF is your friend.

In the tests, UDF had okay write times, and reading times that rival ext4. It does not like heavy delete and re-write operations, and it definitely is not fond of wiping your drive by rm'ing all files. Re-formatting may be a much quicker choice in that case. Regarding predictability, it has pretty consistent operation and unmount times. Sounds ideal for USB drives to me, where one occasionally adds another large file, for archiving or passing files on to others.

Btrfs

The shimmering hope at the Linux file system horizon scores very well in all tests. Its raw reading speed is impressive, and most of the time it is far ahead of UDF and NILFS. Its weakness is its comparatively bad predictability, especially its unmounting times may vary considerably.

Then again, it takes a solid second place in the tests, and when one picks Btrfs, it will be for its support for very large files, writeable snapshots, subvolumes, online defragmentation etc. A very reasonable choice for a general-purpose file system.

NILFS

There is one thing that makes NILFS extremely sexy: continuous snapshotting of the entire file system, with a dead-simple way to remount past file system state. If losing data is of any concern, the idea of NILFS provides a very appealing solution. Also, more or less as a side effect, it offers very fast crash recovery—orders of magnitudes faster by design than any other conventional file system.

A drawback of this design is that, given limited storage space, the file system regularly needs to delete old data, deploying a 'nilfs_cleanerd' garbage collector process. Putting the file system under load, user operations can interfere with the cleanup.

In the tests, NILFS plainly failed in 6 of 9 runs. Most of the times the garbage collector had not started or finished its work, resulting in "no space left on device" errors. Only the 3 error-free runs have been used in the stats above.

So, in a non-tuned scenario, the continuous snapshotting comes at a price. In raw operation times, NILFS comes out last in 3 of 4 tests. Worse, that is also true for the predictability of said operation times. The one thing that NILFS made shine were its sometimes extremely quick unmounting times, and their consistency. That is probably because NILFS performs most of its work online, instead of caching it and delaying it until the device is being unmounted.

On a large storage device, with less frequent operations and careful tuning, NILFS might work wonders—more so if one is after its special features. Out of the box I can not recommend it, given its performance in the tests.

ext4

I included the rocker granddaddy of file systems very late in the tests, simply to have a baseline comparison with a file system that, supposedly, has been getting a bit long in the tooth.

To my surprise, it wiped all other file systems off the carpet. It won 3 of 4 tests by total time, it sported highly predictable operation and unmounting times, and when it was ahead of its rival Btrfs, it was so by considerable amounts.

So, when I'm not after snapshots and subvolumes, when cross-platform does not matter, will I format ext4 everything that moves? Hell, will I swap out my 130 GB /home to reformat the drive ext4?

Absolutely. ♥