| |||||||||||||
Improving ext4: bigalloc, inline data, and metadata checksumsImproving ext4: bigalloc, inline data, and metadata checksumsPosted Nov 30, 2011 0:57 UTC (Wed) by yokem_55 (subscriber, #10498)In reply to: Improving ext4: bigalloc, inline data, and metadata checksums by pr1268 Parent article: Improving ext4: bigalloc, inline data, and metadata checksums
I would also run smartctl -l error to see if your hard drives are bugging out and maybe even run memtest86+ overnight to see if you are having memory errors. Wierd, widespread data (with metadata intact) corruption in my experience tends to be more hardware related than anything else.
(Log in to post comments)
ext4 experience Posted Nov 30, 2011 2:11 UTC (Wed) by dskoll (subscriber, #1630) [Link] I also had a very nasty experience with ext4. A server I built using ext4 suffered a power failure and the file system was completely toast after it powered back up. fsck threw hundreds of errors and I ended up rebuilding from scratch. I have no idea if ext4 was the cause of the problem, but I've never seen that on an ext3 system. I am very nervous... possibly irrationally so, but I think I'll stick to ext3 for now.
ext4 experience Posted Nov 30, 2011 4:52 UTC (Wed) by ringerc (subscriber, #3071) [Link]
The usual culprit in those sorts of severe corruption or loss cases is aggressive write-back caching without battery backup. Some cheap RAID controllers will let you enable write-back caching without a BBU, and some HDDs support it too.
Write-back caching on volatile storage without careful use of write barriers and forced flushes *will* cause severe data corruption if the storage is cleared due to (eg) unexpected power loss.
ext4 experience Posted Nov 30, 2011 9:00 UTC (Wed) by Cato (subscriber, #7643) [Link]
You are right about battery backup. Every modern hard disk uses writeback caching, and some of them make it hard to ensure that the cache is flushed when the kernel wants to ensure a write barrier is implemented. The size of hard disk caches (32 MB typically) and the use of journalling filesystems (concentrating key metadata writes in journal blocks) can mean that a power loss or hard crash loses a large amount of filesystem metadata.
ext4 experience Posted Nov 30, 2011 12:40 UTC (Wed) by dskoll (subscriber, #1630) [Link] My system was using Linux Software RAID, so there wasn't a cheap RAID controller in the mix. You could be correct about the hard drives doing caching, but it seems odd that I've never seen this with ext3 but did with ext4. I am still hoping it was simply bad luck, bad timing, and writeback caching... but I'm also still pretty nervous.
ext4 experience Posted Nov 30, 2011 12:50 UTC (Wed) by dskoll (subscriber, #1630) [Link] Ah... reading http://serverfault.com/questions/279571/lvm-dangers-and-caveats makes me think I was a victim of LVM and no write barriers. I've followed the suggestions in that article. So maybe I'll give ext4 another try.
ext4 experience Posted Nov 30, 2011 20:20 UTC (Wed) by walex (subscriber, #69836) [Link]
You have been wishing for O_PONIES!
It is a very well known issue usually involving unaware sysadms and cheating developers.
Improving ext4: bigalloc, inline data, and metadata checksums Posted Nov 30, 2011 2:13 UTC (Wed) by nix (subscriber, #2304) [Link]
Quite so. I've been using ext4 atop LVM (atop raid1 md, raid5 md, and Areca hardware RAID) for many years, and have never encountered a single instance of fs corruption which fsck could not repair -- and only one severe enough to prevent mounting which was not attributable to abrupt powerdowns, and *that* was caused by a panic at the end of a suspend, and e2fsck fixed it.
I'm quite willing to believe that bad RAM and the like can cause data corruption, but even when I was running ext4 on a machine with RAM so bad that you couldn't md5sum a 10Mb file three times and get the same answer thrice, I had no serious corruption (though it is true that I didn't engage in major file writing while the RAM was that bad, and I did get the occasional instances of bitflips in the page cache, and oopses every day or so).
bitflips Posted Nov 30, 2011 12:49 UTC (Wed) by tialaramex (subscriber, #21167) [Link]
"occasional instances of bitflips in the page cache"
To someone who isn't looking for RAM/ cache issues as the root cause, those often look just like filesystem corruption of whatever kind. They try to open a file, get an error saying it's corrupted. Or they run a program and it mysteriously crashes.
If you _already know_ you have bad RAM, then you say "Ha, bitflip in page cache" and maybe you flush a cache and try again. But if you've already begun to harbour doubts about Seagate disks, or Dell RAID controllers, or XFS then of course that's what you will tend to blame for the problem.
bitflips Posted Dec 1, 2011 19:23 UTC (Thu) by nix (subscriber, #2304) [Link]
This does depend on how bad the RAM was. The RAM on this machine was so bad that the fs was not the only thing misbehaving by any means.
Rare bitflips are normally going to be harmless or fixed up by e2fsck, one would hope. There may be places where a single bitflip, written back, toasts the fs, but I'd hope not. (The various fs fuzzing tools would probably have helped comb those out.)
|
|||||||||||||