Responding to ext4 journal corruption

Ted Ts'o was doing some ext4 testing when he noticed a problem with how the journal checksum is handled. The journal will normally contain several transactions which have not yet been fully played into the filesystem. Each one of those transactions includes a commit record which contains, among other things, a checksum for the transaction. If the checksum matches the actual transaction data in the journal, the system knows that the transaction was written completely and without errors; it should thus be safe to replay the transaction into the filesystem.

The problem that Ted noticed was this: if a transaction in the middle of the series failed to match its checksum, the playback of the journal would stop - but only after writing the corrupted transaction into the filesystem. This is a sort of worst-of-all-worlds scenario: the kernel will dump data which is known to be corrupt into the filesystem, then silently throw away the (presumably good) transactions after the bad one. The ext4 developers quickly arrived at a consensus that this behavior is a bug which should be fixed.

But what should really done is not as clear as one might think. Ted's suggestion was this:

A bit of background might help in understanding the problem that Ted is trying to solve here. In the default data=ordered mode, ext3 and ext4 do not write all data to the journal before it goes to the filesystem itself. Instead, only filesystem metadata goes to the journal; data blocks are written directly to the filesystem. The "ordered" part means that all of the data blocks will be written before the filesystem code will start writing the metadata; in this way, the metadata will always describe a complete and correct filesystem.

Now imagine a journal which contains a set of transactions similar to these (in this order):

1. A file is created, with its associated metadata.

2. That file is then deleted, and its metadata blocks are released.

3. Some other file is extended, with the newly-freed metadata blocks being reused as data blocks.

Imagine further that the system crashes with those transactions in the journal, but transaction 2 is corrupt. Simply skipping the bad transaction and replaying transaction 3 would lead to the filesystem being most confused about the status of the reused blocks. But just stopping at the corrupt transaction also has a problem: the data blocks created in transaction 3 may have already been written, but, as of transaction 1, the filesystem thinks those are metadata blocks. That, too, leads to a corrupt filesystem. By replaying the entire journal, Ted hopes to catch situations like that and leave the filesystem in an overall better shape.

It is, perhaps, not surprising that there was some disagreement with this approach. Andreas Dilger argued:

The next proposal was to make a change to the on-disk journal format ("one more time") turning the per-transaction checksum into a per-block checksum. Then it would be possible to get a handle on just how bad any corruption is, and even corrupt transactions could be mostly replayed. As of this writing, that looks like the approach which will be taken.

Arguably, the real conclusion to take from this discussion was best expressed by Arjan van de Ven in an entirely different context: "having a journal is soooo 1999". The Btrfs filesystem, which has a good chance of replacing ext3 and ext4 a few years from now, does not have a journal; instead, it uses its fast snapshot mechanism to keep transactions consistent. Btrfs may, thus, avoid some of the problems that come with journaling - though, perhaps, at the cost of introducing a set of interesting new problems.

Writing garbage into the journal is quite easy, too. All it takes is for 
the disk to forget a single seek after a legitimate journal write, and 
it'll write something into the journal which was supposed to go elsewhere. 
(I've seen this on disks running live systems on ext3 for huge banks. The 
banks were not very happy, because the sysadmins simply unplugged the disk 
array after the disk errors: so the filesystem was unclean, the journal 
was replayed... and oops, that's sprayed quite a lot of garbage into the 
fs, because a multimegabyte logfile write had landed in the journal, and 
all of that was misinterpreted as metadata. That specific case, in which 
the blocks look nothing like journal blocks at all, was plugged in 
e2fsprogs 1.40, but the bank was using a version of RHEL that was still on 
1.35...)

Reiserfsv3 and jbd both use write ahead logging schemes, and so they solve very similar
problems.  Reiserfs keeps tracks in ram of which blocks are pinned and not available for
allocations, while jbd uses these revoke records.

Keeping track in ram has performance implications, but it is certainly possible.

That's probably one of the big reasons I've found reiserfs (3) so stable 
here, at least after ordered-by-default hit the tree.  I ran a system for 
some time with an annoying memory bus error issue (generic memory rated a 
speed notch higher than it should have been, a BIOS update eventually let 
me limit memory speed by a notch, after which it was absolutely stable) 
that would crash the system with MCE errors relatively frequently.  100% 
reiserfs, no UPS, no problems after ordered-by-default, tho I certainly 
had some previous to that.

I'm running the same system but with a memory and CPU upgrade now, and 
with reiserfs on mdp/kernel RAID-6, system directly on one RAID-6 
partition (with a second for a backup system image), everything else on 
LVM2 on another one.  Despite the lack of barriers on the stack as 
explained in last week's barrier article, and despite continuing to run 
without a UPS and having occasional power outages that often trigger a 
RAID-6 rebuild, I've been VERY happy with system integrity.

Duncan

Another way, which doesn't pin blocks and prevent their reallocation, is to keep track of
dependencies in the journal: transaction 3 _depends_ on transaction 2, because it uses blocks
which are repurposed in transaction 2.  So there should be a note in transaction 3 saying "I
depend on T2".

During replay, if transaction 2 fails due to bad checksum, transaction 3 will be rejected due
to the dependency.  Transaction 4 may be fine, etc.

(The same dependencies can be converted to finer-grained barriers too - e.g. to optimise ext4
on software RAID.)

Some RAM is needed to keep track of the dependencies, until commits are known to have hit the
platters.  If it's a problem, this can be bounded with some hashed approximation akin to a
Bloom filter.

This is what first came to my mind, but if data has been written, but metadata saying what
this data is gets discarded, the new data could be misinterpreted as what the previous
metadata said it was (such as believing it to be more metadata pointing to blocks on the disk,
but it's actually an image). I guess the solution here would be to zero any pointers to
metadata first (or settings a 'corrupt' or 'deleted' flag on the metadata sector itself) and
making sure that's reached the disk before writing the data. Of course this can slow things
down as you have to write to the metadata block an extra time per update.

I think the snapshotting way is the only way forward; if you never get rid of something until
certain the new one works (ie, has completely reached the disc) then it doesn't matter what
you do or when... you'll always have at least one working version. Large writes would start
failing when your disc is nearing full, but with todays drive sizes, we're more concerned with
losing 500G of data than filling it.