The Next3 filesystem

Posted Jun 14, 2010 14:01 UTC (Mon) by Cato (guest, #7643)
In reply to: The Next3 filesystem by cmccabe
Parent article: The Next3 filesystem

You are lucky - the user of one Linux PC that I manage had an unfortunate habit of nudging the reset button when sitting down ... This is rather a pathological case but it resulted in serious data loss on two separate filesystems on different physical disks (one PATA and one SATA), despite no actual power loss. I was using an ext3 plus LVM setup and I'm convinced that the use of hard disk write caching was the problem.

http://lwn.net/Articles/343425/ has more details. Haven't had any more problems since stopping use of write caching and making some other changes such as ext3 data=journal. I also stopped using LVM but I don't think that's a factor - on other PCs I now just use data=journal and turn off hard disk write caching, and still use LVM.

The Next3 filesystem

Posted Jun 15, 2010 10:07 UTC (Tue) by etienne (guest, #25256) [Link] (1 responses)

That is strange: the reset button is not (and has never) been connected to the hard disk, i.e. the hard disk is only aware of the reset button been pressed when the BIOS re-initialises the ATA/SATA interface (so a long time after). There isn't any "reset wire" on ATA or SATA connector.
Historically, that has lead to strange bugs - like LILO was able to start after running Windows 3.1 but not from a cold boot (or the opposite), because the hard disk was reconfigured with a different number of heads and sector per track (BIOS only times, no LBA).
For all what I can see, it is exactly the same nowadays.
The reason has always been to give time to the hard disk to finish and write back its cache.

The Next3 filesystem

Posted Jun 15, 2010 20:33 UTC (Tue) by Cato (guest, #7643) [Link]

My mistake - it was actually the power button not reset that was pressed repeatedly. So there was power loss which is probably what caused the problem due to hard disk write caching.

The Next3 filesystem

Posted Jun 23, 2010 1:41 UTC (Wed) by cmccabe (guest, #60281) [Link] (3 responses)

Up until Linux 2.6.31, write barriers were always disabled while using LVM. So it's unlikely that journaling would have been very effective at preventing data loss in a system using something like ext3 + LVM + Linux 2.6.28.

Personally, I use rsync for monthly backup, every month, and hope for the best. And when you see that first I/O error come out of /dev/sda... throw that thing in the trash. I've never seen a disk "get better" after starting to give I/O timeouts and errors.

The Next3 filesystem

Posted Jun 23, 2010 1:49 UTC (Wed) by cmccabe (guest, #60281) [Link]

Just to clarify. I've never had to restore my data from backup drives. I have had hard disks go "funny" on me. This happened on two disks. Files started becoming unreadable some of the time (but not all) and I/O timeouts started happening. In both cases, I copied over my data from the affected disk to a new disk.

I haven't ever lost data as a result of a power outage, partly because I'm a compulsive user of the save button / command. I also didn't get bitten by the ext4 rename bug / controversy because I was using ext3 at the time. I don't have a UPS at home or work.

The Next3 filesystem

Posted Jun 23, 2010 12:16 UTC (Wed) by Cato (guest, #7643) [Link] (1 responses)

Since I turn off disk write caching that bypasses the problem of write barriers being disabled in such kernels. For backups, I use DAR (like tar but with granular checksums for easier recovery from corruption) and rsnapshot, which is rsync-based, but a true backup system as it saves multiple versions and runs very fast, like rsync - works very well as long as you don't have very large files that change frequently.

The Next3 filesystem

Posted Jun 23, 2010 13:30 UTC (Wed) by nix (subscriber, #2304) [Link]

DAR (like tar but with granular checksums for easier recovery from corruption)

Actually par2 provides that feature. What dar gives you is multi-storage-medium support via running arbitrary scripts to change medium. tar has nothing like it.

works very well as long as you don't have very large files that change frequently

Like, uh, VM images? I hear they're quite common these days.

(If you've got a lot of those, try rdiff-backup. It's slower than rsnapshot, but when a file changes it stores rdiff-format compressed deltas from the new file to the old one, rather than resaving the entire old file all over again.)