When and why to deprecate filesystems [LWN.net]

When and why to deprecate filesystems

Posted Mar 8, 2022 12:29 UTC (Tue) by Wol (subscriber, #4433)
In reply to: When and why to deprecate filesystems by calumapplepie
Parent article: When and why to deprecate filesystems

Well, I learnt the lesson when I installed a new program-in-testing, and it killed my integritysetup.service. All three disks of my raid-5 failed to appear. Cue panic stations.

So I now have an 8TB backup drive. I've still to automate backups with systemd timers, but as I run ext4 over lvm (over raid over dm-integrity for my live system) over spinning rust, I just take lvm snapshots as my first stage backup, followed by rsync --in-place to lvm snapshots of my backups.

Just buy the biggest, cheapest (and yes SMR is fine for this) drive you can get your hands on, and set up rsync backups.

Cheers,
Wol

to post comments

When and why to deprecate filesystems

Posted Mar 8, 2022 13:01 UTC (Tue) by geert (subscriber, #98403) [Link] (12 responses)

Using rsync to a single backup drive is not a real backup solution.

When and why to deprecate filesystems

Posted Mar 8, 2022 15:34 UTC (Tue) by Wol (subscriber, #4433) [Link] (11 responses)

It's a lot better than having only one copy of the data (or all copies on the same drive).

Yes I ought to think about having multiple backups, but one backup is a major advance on none.

Cheers,
Wol

When and why to deprecate filesystems

Posted Mar 8, 2022 16:04 UTC (Tue) by nix (subscriber, #2304) [Link] (10 responses)

Nothing that is not automated is a real backup solution either. At the *very least* you need a backup and an offsite backup that's being kept up to date (and an offsite backup that's actually offsite and disconnected, and the two swapped occasionally). Without that, one electrical storm and you might well lose everything, all at once.

Personally I'd be lost without historical backups as well -- ever so often I find I need something "as it was before Sep 15 last year" or something like that, and with proper backups you can get that sort of thing in minutes. This is not even hard. All the tools exist: all you need is a bit of scripting and a cronjob.

When and why to deprecate filesystems

Posted Mar 8, 2022 16:54 UTC (Tue) by Wol (subscriber, #4433) [Link] (9 responses)

Offsite? Where? :-) For a home system that could be a problem (unless I put a bare drive in the garage...)

As for historical, that's why I use lvm. Take a snapshot, do the inplace rsync, and I get a full backup for the price of an incremental.

I don't plan to clean out my incrementals, unless I'm running short of disk space.

Cheers,
Wol

When and why to deprecate filesystems

Posted Mar 8, 2022 17:11 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

Offsite? Where? :-) For a home system that could be a problem (unless I put a bare drive in the garage...)

If you're limiting yourself to critical files- or if you're willing to spend more- you could back up to the cloud. It's definitely offsite, and it has its own layers of protection. If you're serious, you'd want some solution that would let you encrypt your files locally, so anyone who gets a copy of your cloud files won't be able to read all your secrets. Of course you probably want that for any off-site backup.

When and why to deprecate filesystems

Posted Mar 8, 2022 17:21 UTC (Tue) by geert (subscriber, #98403) [Link] (1 responses)

Yep, dropping off at relatives LUKS-encrypted USB hard drives containing Amanda vtapes.

When and why to deprecate filesystems

Posted Mar 8, 2022 17:36 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link]

I keep an off-site backup in a locked desk drawer at work, but I still keep that encrypted. Transparent encryption is easy enough that it's silly not to use it on any USB hard drive whose contents you care at all about keeping secret.

When and why to deprecate filesystems

Posted Mar 8, 2022 18:16 UTC (Tue) by smurf (subscriber, #17840) [Link] (4 responses)

You can do the same thing with btrfs snapshots. Way faster than rsync because it doesn't have to scan a complete hierarchy only to discover that nothing has changed.

When and why to deprecate filesystems

Posted Mar 8, 2022 23:10 UTC (Tue) by Wol (subscriber, #4433) [Link] (3 responses)

???

How does btrfs snapshot magic everything onto a different hard drive?

And to the best of my knowledge, btrfs raid-5 is still in experimental eat-your-data status ...

I use lvm on my live raid-5 array to take live snapshots, and lvm/rsync on my backup drive to take full/incremental backups.

Cheers,
Wol

When and why to deprecate filesystems

Posted Mar 9, 2022 6:08 UTC (Wed) by calumapplepie (guest, #143655) [Link]

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

When and why to deprecate filesystems

Posted Mar 10, 2022 2:31 UTC (Thu) by bartoc (guest, #124262) [Link] (1 responses)

btrfs raid5 is arguably less "eat your data" than lvm raid5, since it does actually checksum and can, in fact, detect when drives are doing bogus stuff. You need to be quite careful with scrubbing regularly (and per drive) and with what you do if you need to rebuild the array. I don't really recommend running raid5/6 in any system using SATA drives, they are too willing to lie to you and try and cover up impending failures.

Tbh ZFS' raid5 also leaves something to be desired since it avoids the write hole by doing things that can have pretty bad performance consequences down the line (esp with how unwilling ZFS is to muck around with stuff on disk, in general). I'm hoping bcachefs' approach pays off (it does raid5 by initially writing the data in raid10 (or at least mirrored), committing that, then later rewriting that mirror to raid5 and atomically updating the filesystem metadata once it's done.

When and why to deprecate filesystems

Posted Mar 14, 2022 15:10 UTC (Mon) by hkario (subscriber, #94864) [Link]

btrfs raid5 is still marked as experimental, while yes, HDDs lying about their ability to read valid data is an issue, that can be fixed by placing dm-integrtiy below MD-raid and then RAID-5 works fine

also, HDDs lying about bad sectors is not only a RAID-5 issue, it's just as likely to impact RAID-1 setups

When and why to deprecate filesystems

Posted Mar 9, 2022 21:27 UTC (Wed) by nix (subscriber, #2304) [Link]

> Offsite? Where? :-) For a home system that could be a problem (unless I put a bare drive in the garage...)

You give a drive to a friend you see sometimes and swap it. (I do this with several friends and relatives living at different distances from me.)

You really don't want to use lvm snapshots to do long-term historical backups: IIRC, writes slow down proportional to the number of snapshots in place. For comparison and to see what you can do with an actual automated backup system, I have 9103 backup snapshots currently accessible in my onsite backup (mostly once-every-three-hourly backups of /home). Most of the backups took under a minute to run (and, obviously, happened with no human intervention at all). LVM would grind completely to a halt long before you reached *that* sort of scale.

> I don't plan to clean out my incrementals, unless I'm running short of disk space.

Quite so -- and since bup uses rsync-style deduplication and also compresses the backups, that's going to be a long time.