Quote of the week

All of my disks now are in mirrored pairs. Upgrading them to larger space is just a matter of replacing one of the pair with a larger disk, resilvering, and replacing the other disk. (Or something fancier, such as striping). I haven't used snapshots much yet, but that looks very good for having some off-site backup as well.

As far as I understand it, most HDD failures are physical failures in mechanical parts that are not repairable by in-built error-correction. Recovering data from a head HDD is expensive (probably impractical for a home user? Not sure). After a HDD failure a few years ago where I lost a lot of photos that didn't have a backup, I've now got all my critical stuff in 'the cloud', but it is also nice to know that my desktop itself has some resilience to hardware failures.

Because the real and true value is not so much about checking whether your media is lying to you about the data it's reading back to you you but really about knowing what you are reading and got from the whole chain (eg: media -> firmware -> transport -> controller -> memory) is exactly what you wanted to write in the first place.

With other non checksumming filesystems, when an IO goes from the kernel to the disk controller to end up on a disk, you have no idea if it was corrupted in flight or not.

Maybe you're just writing random crap that was corrupted in memory, during transfer, by your controller, by your disk array, by your FC infrastructure and so on. In that case when you read it back, your hard drive, array or whatever will happily say it's all good since it's what it got in the first place. Or equally, maybe what it read was correct but it was corrupted on its way to you: you just don't know.

And that's what a checksumming filesystems is good at.

You're thinking filesystem checksums are about catching media corruption while you should be looking at the big picture: it's an end-to-end checksum.

Because it is a *real life problem* that block addresses get corrupted during a write operation. Which means that data block your system wrote went to the *wrong block*. Now the new data is not where it is expected *and* it just wrote over some unknown block of data.

ZFS or btrfs can help with that because by having multiple copies of the data when it goes to read it or scrub it, it will notice that data is either missing or overwritten and can recover from another copy.

That paper, even though it is 12 years old by now, gives very accurate models for those errors, and even if you tweak the percentages down to compensate for "new developments" in HDD engineering, you still get non-negligible chances for visible errors during the lifetime of the harddisk.

Yes, there are error checks and corrections, and yes, these might drop the BER for another order of magnitude or two. But there are also software bugs, race conditions, etc. in the drive firmware that counter these improvements.

And these issues actually happen in practice. We have thousands of harddisks deployed in the field inside storage systems, and we see the blocks that just so happen to not make it to disk (or back) correctly.

So while the actual numbers might be considered "fear-mongering", and you might argue that it should be one or two orders of magnitude higher or lower, if you have hundreds or thousand disks deployed it will have an impact.

So, no, there really are errors coming off the platter, that the controller can, most of the time, tease back to the correct bits--when it can tell they were wrong. But some errors produce the right checksum, and the controller is none the wiser. The designer has a target error rate, and producing fewer errors than the budget allows means the bits are not packed in as tightly as they could have been, and the disk advertises less capacity, for less money. If you care about errors, you will have made arrangements to tolerate some.

Those are in addition to any errors resulting from noise on the bus, which is, ultimately another analog affair, albeit with better margins than on the platter. The bus doesn't get to re-try, and anyway can't tell if it should have.