Runtime filesystem consistency checking [LWN.net]

Runtime filesystem consistency checking

Posted Apr 3, 2012 17:33 UTC (Tue) by jzbiciak (guest, #5246)
Parent article: Runtime filesystem consistency checking

It seems like this could also catch hardware problems, such as bit flips in main memory (for folks that lack ECC), and bugs in non-filesystem code, if it results in memory corruption.

to post comments

Runtime filesystem consistency checking

Posted Apr 4, 2012 12:07 UTC (Wed) by nix (subscriber, #2304) [Link] (5 responses)

There's a bigger problem: it is possible to tell a disk to write something somewhere, and it reports success, but has actually either written something different, lost it entirely, or overwritten something different. Local consistency checks on read would detect the first two cases, but no local check can spot the third case, as far as I can see. (This case is certainly not common, though: I have seen only one disk problem that was possibly attributable to this class of failure. I've been trying to get info on the frequency of this sort of missed-seek or mis-seek error, but all I can find is rumours. The secrecy of disk manufacturers regarding info critical to the integrity of one's data is quite distressing at times.)

Runtime filesystem consistency checking

Posted Apr 4, 2012 13:16 UTC (Wed) by ashvin (guest, #83894) [Link]

Recon can catch bit flips that occur in memory and that corrupt file system metadata buffers before the Recon code is run. Catching bit flips after the Recon checks require checksumming and replication of file system metadata buffers. Recon will also not detect lost and misdirected writes if they occur after the Recon checks. However, they can be caught using version numbers, but the file system needs to support that, e.g., generation numbers in btrfs.

Runtime filesystem consistency checking

Posted Apr 12, 2012 12:47 UTC (Thu) by nye (guest, #51576) [Link] (3 responses)

>There's a bigger problem: it is possible to tell a disk to write something somewhere, and it reports success, but has actually either written something different, lost it entirely, or overwritten something different

My ZFS experience on a ~5TB pool consisting of six commodity HDs under fairly light load (ie. it's a home file server) is that every couple of months scrub detects checksum errors in a block or a small handful of blocks, without any corresponding read/write errors being given by the device.

Not sure if that's the situation you're talking about.

(Also, the same experience has taught me that Western Digital should be avoided like ebola. I actually wonder if their green series might be drives that have failed QC and been re-badged for the unsuspecting consumer.)

Runtime filesystem consistency checking

Posted Apr 13, 2012 10:33 UTC (Fri) by etienne (guest, #25256) [Link] (2 responses)

> every couple of months scrub detects checksum errors in a block or a small handful of blocks, without any corresponding read/write errors being given by the device.

I am not sure to interpret exactly what is happening on my own PC, but I suspect something like:
- one block of sectors develop a bit fault in the magnetic data
- the ECC correct it each times the PC reads the sector resulting in a *very long* delay of few seconds
- the Linux driver do not noticed there was long ECC correction and do not decide to rewrite the sector identically to get the magnetic data corrected
- long term a second error will appear on the magnetic data and the ECC will no more be sufficient.

I do not know why the sector is not rewritten by the Linux driver, I know that I did solve same problem on another PC by touching a file in a directory, forcing the sector containing the directory entry to be rewritten.
I never noticed the problem when the "old" ATA/IDE driver was used, but I am not sure I interpret correctly what happens on my PC during the last few days...

Runtime filesystem consistency checking

Posted Apr 13, 2012 12:34 UTC (Fri) by james (guest, #1325) [Link]

I understand that these days, it's the disk firmware's job to do this. It knows that there's a read error (which Linux doesn't, although it can sometimes guess ¹); it knows that sector is dodgy (which Linux doesn't, because it doesn't get told when the hardware remaps sectors), it can try re-writing the sector, and if that fails, it can remap in a spare sector (which Linux can't do, because it doesn't have access to the disk's spare sectors) if (and only if) it knows what that sector's contents should be, either because it's managed to read something, or the computer is overwriting that sector.

And it can do all of that without having to worry about which operating system is running, or it's a database using raw access, or if it's a light layer using BIOS calls but no filesystem. It can preserve this information across reformats.

In your case, by causing the sector containing the directory entry to be rewritten, the disk probably decided that this was a great time to remap in a spare sector, so it actually went to a different part of the disk. (Unless the filesystem you were using put the new directory entry somewhere else anyway.)

And ECC correction doesn't take seconds; re-reading the same sector repeatedly in the hope that you can get a last good read does.

¹ If you've got command queueing turned on, several requests outstanding, and there's a delay, which sector caused the problem?

Runtime filesystem consistency checking

Posted Apr 13, 2012 14:51 UTC (Fri) by jzbiciak (guest, #5246) [Link]

Adding to what James said above me, I think you might want to look into installing the SMART tools to query your drive. The "smartctl" tool will read out and display the various statistics the drive's been collecting about corrected errors and remapped sectors.

It will also let you fire up background health checks (these can take quite a long time to complete -- as long as a day, as I recall) that may help turn up other problems.