Now that I've rebuilt the PC... I actually lost the contents of two filesystems (root and backup), each on a separate physical disk, and hosted on LVM logical volumes within separate volume groups. There weren't any physical disk errors, or noticeable errors relating to PATA/SATA cables, memory, etc. The only common factor is that both LVM VGs were, well, handled by LVM. It's also suspicious that only one LVM FS was uncorrupted, plus all of the non-LVM FSs.
I suspect that some combination of disk write caching plus LVM and possibly ext3 caused these problems. At least some of the problem was purely at the LVM level, since I couldn't even access the VGs on the backup disk, and got LVM errors.
In the hope that it helps someone else:
- To help avoid integrity problems in future, I used the ext3 'data=journal,barriers=1' options in fstab, and also used tune2fs to set the journal_data option on the root FS (only way that worked for root). I also disabled disk level write caching with "hdparm -W0 /dev/sdX' on both hard disks. This will have some performance cost but this PC is ridiculously fast for light email and web surfing anyway.
- I've dropped SpiderOak for online backup - it didn't back up most of the files (on two PCs, in different ways), generated a corrupt ZIP file on recovering some files via web interface, and the GUI client got stuck recovering files, and generally makes it hard to track backups/restores.
- I have implemented local backups with rsnapshot, which is really outstanding for multi-verson rsync based, and will extend this for online backups, possibly using DAR to encrypt and compress for remote backups.
- Sbackup (Simple Backup) is great for really quick backup setup (literally 2 minutes to install, configure and have first backup running), but I wouldn't rely on that alone.
Also, if you haven't used etckeeper before, it's worth a try - version control for the whole of /etc using git, hg, bzr, or darcs, and also tracks APT package installs that generate /etc changes. Great if you need to replicate some or all of the setup at a later date.