Re: file systems
- From: "Boyd Stephen Smith Jr." <bss@xxxxxxxxxxxxxxxxx>
- Date: Mon, 2 May 2011 16:02:44 -0500
In <4DBE75DD.80900@xxxxxxxxxxxxxxxxx>, Stan Hoeppner wrote:
On 5/1/2011 10:40 AM, Boyd Stephen Smith Jr. wrote:
In<4DBD0D23.1080903@xxxxxxxxxxxxxxxxx>, Stan Hoeppner wrote:
Independent Linux filesystem tests performed by an IBM engineer to track
BTRFS performance during development. XFS trounces the others in most
tests:These results are interesting and useful, but I think "trounces" is a poor
description for what XFS does.
Not using barriers undermines data consistency guarantees, I think it is
best to ignore the 2.6.35-rc5-autokern1-ext3-*-ext3,
2.6.35-rc5-autokern1-ext4-*- ext4-nobarrier, and
It would be best for you to state something like "which results are
relevant to you depend on your workload and hardware environment".
Generally, anyone running a server with persistent storage is interested
in the nobarrier results. Those without persistent storage should be
interested in the barrier results. There are exceptions to this general
guideline and I mention at least one below.
Barriers are used to flush storage device cache to avoid data loss due
to a system crash or power loss.
That's not all they are used for. They are also used to makes sure that
writes occur in the correct order.
Barriers are great for low end systems
lacking persistent storage.
They are also essential for any journaled filesystem to have correct behavior
in the face of sudden pwoer loss. Barriers ensure, (e.g.) that the journal
entry creating a file is flushed to the backing store before the journal
entry. Or, that the journal entry
However, barriers should never be used with
high performance persistent storage, i.e. RAID/SAN controllers w/
battery or flash backed write cache.
Even with a a battery-packed RAID cache, like I have in my desktop, executing
without barrier can result in extra data loss that executing with a barrier
Never run without barriers if you value your data.
Of course, even with out barriers a properly journaled or log-structed
filesystem should be able to immediately and silently recover.
Using barriers with such storage
arrays simply costs you between 50-90% of your random write performance,
especially with shared SAN storage and many hosts, as issuing a single
barrier flushes the entire BBWC in the array controller. May enterprise
arrays contain 32GB or more of BBWC.
Barriers do have a cost, however, on modern file systems it is not anwhere
near 50% even in the random write case, since a barrier doesn't have to occur
between every journal entry.
rc5_Large_file_random_writes._num_threads=128.html> shows less than a 10%
difference between XFS performance with barriers and without.
From this LWN article <http://lwn.net/Articles/283161/>: "The filesystem code
must, before writing the [journaling] commit record, be absolutely sure that
all of the transaction's information has made it to the journal. Just doing
the writes in the proper order is insufficient; contemporary drives maintain
large internal caches and will reorder operations for better performance. So
the filesystem must explicitly instruct the disk to get all of the journal
data onto the media before writing the commit record; if the commit record
gets written first, the journal may be corrupted. The kernel's block I/O
subsystem makes this capability available through the use of barriers; in
essence, a barrier forbids the writing of any blocks after the barrier until
all blocks written before the barrier are committed to the media. By using
barriers, filesystems can make sure that their on-disk structures remain
consistent at all times."
The nobarrier results are far more relevant than the barrier results,
especially the 16 and 128 thread results, for those SAs with high
performance persistent storage.
I disagree entirely. You should be looking at the threaded results, probably
128 threads (depending on what the server does), but you should also be using
For desktop users the single thread
barrier results are probably the most relevant.
I wouldn't say that is a "trouncing", since it doesn't even win in many
In those test where XFS has the best 'score' it typically beats most
others by a wide margin.
No, it doesn't. On a few tests, it does amazingly outpace other file systems.
On most of the others, even when it is first, one of the other filesystems is
a close second.
If you could interpret a graph correctly Boyd maybe you'd see ithttp://btrfs.boxacle.net/repository/raid/2.6.35-rc5/2.6.35-rc5/2.6.35-rc5
4th, 2nd, 4th, 4th, 2nd
differently. :) Just taking the last one above as an example, you
listed XFS as 4th place when it's clearly in first place, by a huge
margin over all but JFS. XFS 32k, JFS 30k, EXT4 20k.
XFS without barriers is first -- but I prefaced my statements with the fact
that I was ignoring -nobarrier variants and the -nocow variant of btrfs.
JFS (default): 30k
BtrFS (default): 10.8k
Ext3 (barrier): 6.1k
XFS (default, barrier): 5k
Ext4 (default, barrier): 3.5k
XFS = 4th.
Boyd Stephen Smith Jr. ,= ,-_-. =.
bss@xxxxxxxxxxxxxxxxx ((_/)o o(\_))
ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-'
Description: This is a digitally signed message part.