Barriers and journaling filesystems

So, for those that wish to enable them, barriers apparently are turned on by giving
"barrier=1" as an option to the mount(8) command, either on the command line or in /etc/fstab:

mount -t ext3 -o barrier=1 <device> <mount point>

(along with whatever other options you desire)

If this is in error, please feel free to correct.

so is there any work going on towards making the barriers work on lvm volumes? 

regards  robert

This is all well and good but it would be I think most effective if drives had their own
battery 
backed write-back cache which can be pinned by the OS to the journal.  If the feature can be
made 
persistent, by committing the data to flash or something on drive shutdown, then the data
doesn't 
even need to be committed to the disk.  It seems to me that this kind of design could remove 
inherent penalties associated with journaling using proper write barriers.  This kind of
technology 
could use used for transactional databases as well by making the transaction record work at 
memory and interface speed rather than be limited by the rotational speed of the drives.  This

should reduce contention on the platters by removing one of the more constant sources of
activity.

> There is also some potential for improvement by carefully ordering operations so that
barriers (which are normally implemented as "flush all outstanding operations to media"
requests) do not force the writing of blocks which do not have specific ordering requirements.

SCSI actually provides a convenient mechanism for doing this. You can declare a command as
being "Ordered" which will prevent it from being reordered relative to any other commands
which have also been declared as such. Unfortunately AFAIK ATA provides no such niceties. I
also don't know how widely supported this feature is by the various SCSI HBA drivers (I
suspect not very) but it is an option to explore.

XFS has barriers enabled by default since 2.6.17, and it still seems to perform well (on the
last server where I put it on without disabling barriers).

^1 http://lkml.org/lkml/2006/5/22/278

On a system with ext3 filesystems and barrier=0 (because of LVM usage), am I at risk in the
following situations?

a. The system's storage controller has a battery
   backed cache.

b. The system uses plain local disks, but 
   'hdparm -W 0 <device>'
   is run at boot time.

Of course, md also doesn't support barriers for raid5/raid6, which means 
that using RAID actually *reduces* reliability in some respects. This is 
somewhat irritating, but also hard to fix: barriering all the constituent 
devices when a barrier comes in from the higher level isn't good enough on 
raid5, because if one drive loses power before the other you still have 
data corruption.

(In practice this too is rare enough that it takes explicit torture tests 
to trip it unless you're very unlucky.)

I manage transactional systems for a living (first IBM MQ, now relational databases including
Sybase, DB2 and Oracle).  I am sorry to say the current filesystem approach is still somewhat
optimistic, even in the presence of barriers; and that commercial transational systems do
additional work that ext3/ext4 do not yet do, but that can be required for some disk failures.
Sadly, implementing this correctly may make the filesystems even slower.

One issue has to do with disk failures (say on a power failure) and a partial write.  If
transaction A is committed and uses a partial disk block, then transaction B cannot safely be
written to the same disk block - a power failure could lead to the block being partially
written and losing the data from committed transaction B.  On the other hand, always using a
new disk block or each transaction would eat up log space really quickly.  One solution is to
use ping-pong blocks (three are required of transactions can span a block).

Another issue has to do with when transactions are written to disk.  As the article indicates,
delaying writes may help if multiple concurrent and independent transactions can be written to
disk at the same time; but waiting to write data to disk is bad if one application is doing
most of the writes, as it would not run as quickly as possible.  Some databases allow this to
be tuned on the fly (e.g. mincommit in DB2), but that is not desirable for a filesystem; the
kernel should use heuristics to figure this out as a workload is running.

Next, you want to carefully tune: the size of the log buffer in memory and the  total size of
the transaction log on disk (i.e. when do we wrap round).  In databases you also care about
how much log space a single transaction can use and how much other concurrent log activity
(from other transactions) may occur between activity and commit, but until we expose
filesystem transactions to userspace we can safely ignore this.

Summary: filesystem implementors ought to talk to database implementors. I'm sure both groups
can teach each other a lot, but in this area databases are still quote a bit ahead of
ext3/ext4.

"if the commit record gets written first, the journal may be corrupted."

Would placing the Journal into its own mirrored device suffice?
(Assuming of course that only one mirror could fail at a time.)

Wouldn't a mirrored Journal perform faster on (re)mount, especially if journal_data was in
use? 

For LVM an LV has the option to be mirrored (as well as striped), could a separate mirrored LV
be used just for the Journal to avoid concerns about the lack of barriers on LVM?

Why not insert a barrier *only* when the commit record is inserted before the journaled data
on the disk? I can't imagine that happens too often, so it should cause few barriers will be
inserted. It shouldn't have a terrible impact on performance, but it *would* help deal with
the edge case of a journal write wrapping on the commit record causing the file system to be
inconsistent.

I gotta say, I'm dubious of the notion that most of the time things out just fine with the
ordering of journal writes. If that were true, enabling barriers wouldn't impose a 30%
performance penalty (unless they really are being generated far more often than they need be,
in which case, you have two problems ;-).

This article has generated a lot of comments, and I for one would like to see a follow on
article that takes many of these comments into account and filters the real facts out of them.

Thanks for the great work, keep it coming.