When and why to deprecate filesystems

Posted Mar 7, 2022 18:36 UTC (Mon) by linuxrocks123 (subscriber, #34648)
In reply to: When and why to deprecate filesystems by post-factum
Parent article: When and why to deprecate filesystems

Yes. I'm using it on a MythTV system since I read somewhere it's supposedly good for lots of very large files.

I don't use it on my newer MythTV systems since I later discovered that ext4-without-journal works fine for MythTV. ext4-without-journal is my goto filesystem for everything now, although I really wish ext4 would get COW support.

When and why to deprecate filesystems

Posted Mar 7, 2022 19:15 UTC (Mon) by calumapplepie (guest, #143655) [Link] (21 responses)

I think it's highly unlikely that anyone will add COW to the EXT filesystems.
These days, the EXT filesystem is the 'ol reliable' of linux filesystems. It's stable, easy to recover from crashes, simple enough in structure, and an excellent compromise between keeping data local and spreading things out. It's very low overhead: at the end of the day, you just find the block and change it. COW, even just of individual files rather than all metadata, requires a lot of additional work for every operation. For instance, deleting a file doesn't just mean removing the extents, but it also means checking other files to see if they're using those extents rather than just marking them as free. You also lose the nice, low-fragmentation behavior on hard disks where seeking is costly: file blocks stop tending to be in nice ordered segments, and start tending to be chaotically placed across the disk. These aren't properties you want in a filesystem that is trying its very best to be as stable, reliable, lightweight, and boring as it can be.

Ext4 is so reliable that even if the first 600 MB of your workhorse laptop's disk are overwritten with, say, the Debian Testing installation ISO because you wrote of=/dev/sda instead of of=/dev/sdb, you can recover almost all your data and have a fully functional system again with a bit of effort, some abuse of fsck, and a bit of quality time with debsums. I don't think I can say the same about BTRFS, though I can't say I've tested that theory.

When and why to deprecate filesystems

Posted Mar 7, 2022 21:01 UTC (Mon) by Paf (subscriber, #91811) [Link] (3 responses)

"but it also means checking other files to see if they're using those extents rather than just marking them as free."

No, it means checking the extent reference count, no need to know who's using it. It's not expensive. The other reasons you gave are real downsides, though.

But I do agree that EXT4 is not going to suddenly become a COW file system. For one thing, a COW file system is a different design point that is very far from globally superior to an extent based file system. It is different, with advantages and disadvantages.

It could *conceivably* gain some limited COW capabilities like I believe XFS has, where it can be done optionally for specific files.

When and why to deprecate filesystems

Posted Mar 8, 2022 6:16 UTC (Tue) by linuxrocks123 (subscriber, #34648) [Link] (1 responses)

The XFS thing is what I'm hoping to have eventually. I want just enough COW to be able to do cp --reflink=always and run rmlint.

When and why to deprecate filesystems

Posted Mar 16, 2022 3:14 UTC (Wed) by nevyn (guest, #33129) [Link]

Probably want to explicitly ask for reflink support, as the COW bits are implied and has some chance of being done.

When and why to deprecate filesystems

Posted Mar 20, 2022 6:50 UTC (Sun) by oldtomas (guest, #72579) [Link]

"No, it means checking the extent reference count"

Yes, but. If you look at the garbage collector community you soon realize that /maintaining/ those refcounts is expensive. It means one write access for each change in refcount.

That's why when GC performance starts to matter, most drop the first (naive) refcount implementation.

For file systems things won't be that different.

When and why to deprecate filesystems

Posted Mar 7, 2022 22:08 UTC (Mon) by amarao (guest, #87073) [Link] (16 responses)

We've used btrfs on jbod for this exact reason. If we loose one (not the first) drive, btrfs can recover data from other drives without dark magic. Just mount and copy.

When and why to deprecate filesystems

Posted Mar 8, 2022 6:03 UTC (Tue) by calumapplepie (guest, #143655) [Link] (14 responses)

Well, I followed the rigorous backup protocol of "nah, I don't need that". Unfortunately, I have not learned my lesson.

Also, I object to the implication that any dark magic was required. Magic, yes: but I didn't need anything too dark and evil. Of course, it would've been nice to know that BEFORE I sacrificed those lambs.

When and why to deprecate filesystems

Posted Mar 8, 2022 12:29 UTC (Tue) by Wol (subscriber, #4433) [Link] (13 responses)

Well, I learnt the lesson when I installed a new program-in-testing, and it killed my integritysetup.service. All three disks of my raid-5 failed to appear. Cue panic stations.

So I now have an 8TB backup drive. I've still to automate backups with systemd timers, but as I run ext4 over lvm (over raid over dm-integrity for my live system) over spinning rust, I just take lvm snapshots as my first stage backup, followed by rsync --in-place to lvm snapshots of my backups.

Just buy the biggest, cheapest (and yes SMR is fine for this) drive you can get your hands on, and set up rsync backups.

Cheers,
Wol

When and why to deprecate filesystems

Posted Mar 8, 2022 13:01 UTC (Tue) by geert (subscriber, #98403) [Link] (12 responses)

Using rsync to a single backup drive is not a real backup solution.

When and why to deprecate filesystems

Posted Mar 8, 2022 15:34 UTC (Tue) by Wol (subscriber, #4433) [Link] (11 responses)

It's a lot better than having only one copy of the data (or all copies on the same drive).

Yes I ought to think about having multiple backups, but one backup is a major advance on none.

Cheers,
Wol

When and why to deprecate filesystems

Posted Mar 8, 2022 16:04 UTC (Tue) by nix (subscriber, #2304) [Link] (10 responses)

Nothing that is not automated is a real backup solution either. At the *very least* you need a backup and an offsite backup that's being kept up to date (and an offsite backup that's actually offsite and disconnected, and the two swapped occasionally). Without that, one electrical storm and you might well lose everything, all at once.

Personally I'd be lost without historical backups as well -- ever so often I find I need something "as it was before Sep 15 last year" or something like that, and with proper backups you can get that sort of thing in minutes. This is not even hard. All the tools exist: all you need is a bit of scripting and a cronjob.

When and why to deprecate filesystems

Posted Mar 8, 2022 16:54 UTC (Tue) by Wol (subscriber, #4433) [Link] (9 responses)

Offsite? Where? :-) For a home system that could be a problem (unless I put a bare drive in the garage...)

As for historical, that's why I use lvm. Take a snapshot, do the inplace rsync, and I get a full backup for the price of an incremental.

I don't plan to clean out my incrementals, unless I'm running short of disk space.

Cheers,
Wol

When and why to deprecate filesystems

Posted Mar 8, 2022 17:11 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

Offsite? Where? :-) For a home system that could be a problem (unless I put a bare drive in the garage...)

If you're limiting yourself to critical files- or if you're willing to spend more- you could back up to the cloud. It's definitely offsite, and it has its own layers of protection. If you're serious, you'd want some solution that would let you encrypt your files locally, so anyone who gets a copy of your cloud files won't be able to read all your secrets. Of course you probably want that for any off-site backup.

When and why to deprecate filesystems

Posted Mar 8, 2022 17:21 UTC (Tue) by geert (subscriber, #98403) [Link] (1 responses)

Yep, dropping off at relatives LUKS-encrypted USB hard drives containing Amanda vtapes.

When and why to deprecate filesystems

Posted Mar 8, 2022 17:36 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link]

I keep an off-site backup in a locked desk drawer at work, but I still keep that encrypted. Transparent encryption is easy enough that it's silly not to use it on any USB hard drive whose contents you care at all about keeping secret.

When and why to deprecate filesystems

Posted Mar 8, 2022 18:16 UTC (Tue) by smurf (subscriber, #17840) [Link] (4 responses)

You can do the same thing with btrfs snapshots. Way faster than rsync because it doesn't have to scan a complete hierarchy only to discover that nothing has changed.

When and why to deprecate filesystems

Posted Mar 8, 2022 23:10 UTC (Tue) by Wol (subscriber, #4433) [Link] (3 responses)

???

How does btrfs snapshot magic everything onto a different hard drive?

And to the best of my knowledge, btrfs raid-5 is still in experimental eat-your-data status ...

I use lvm on my live raid-5 array to take live snapshots, and lvm/rsync on my backup drive to take full/incremental backups.

Cheers,
Wol

When and why to deprecate filesystems

Posted Mar 9, 2022 6:08 UTC (Wed) by calumapplepie (guest, #143655) [Link]

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

When and why to deprecate filesystems

Posted Mar 10, 2022 2:31 UTC (Thu) by bartoc (guest, #124262) [Link] (1 responses)

btrfs raid5 is arguably less "eat your data" than lvm raid5, since it does actually checksum and can, in fact, detect when drives are doing bogus stuff. You need to be quite careful with scrubbing regularly (and per drive) and with what you do if you need to rebuild the array. I don't really recommend running raid5/6 in any system using SATA drives, they are too willing to lie to you and try and cover up impending failures.

Tbh ZFS' raid5 also leaves something to be desired since it avoids the write hole by doing things that can have pretty bad performance consequences down the line (esp with how unwilling ZFS is to muck around with stuff on disk, in general). I'm hoping bcachefs' approach pays off (it does raid5 by initially writing the data in raid10 (or at least mirrored), committing that, then later rewriting that mirror to raid5 and atomically updating the filesystem metadata once it's done.

When and why to deprecate filesystems

Posted Mar 14, 2022 15:10 UTC (Mon) by hkario (subscriber, #94864) [Link]

btrfs raid5 is still marked as experimental, while yes, HDDs lying about their ability to read valid data is an issue, that can be fixed by placing dm-integrtiy below MD-raid and then RAID-5 works fine

also, HDDs lying about bad sectors is not only a RAID-5 issue, it's just as likely to impact RAID-1 setups

When and why to deprecate filesystems

Posted Mar 9, 2022 21:27 UTC (Wed) by nix (subscriber, #2304) [Link]

> Offsite? Where? :-) For a home system that could be a problem (unless I put a bare drive in the garage...)

You give a drive to a friend you see sometimes and swap it. (I do this with several friends and relatives living at different distances from me.)

You really don't want to use lvm snapshots to do long-term historical backups: IIRC, writes slow down proportional to the number of snapshots in place. For comparison and to see what you can do with an actual automated backup system, I have 9103 backup snapshots currently accessible in my onsite backup (mostly once-every-three-hourly backups of /home). Most of the backups took under a minute to run (and, obviously, happened with no human intervention at all). LVM would grind completely to a halt long before you reached *that* sort of scale.

> I don't plan to clean out my incrementals, unless I'm running short of disk space.

Quite so -- and since bup uses rsync-style deduplication and also compresses the backups, that's going to be a long time.

When and why to deprecate filesystems

Posted Mar 8, 2022 7:42 UTC (Tue) by ibukanov (subscriber, #3942) [Link]

For me the biggest attraction of BTRFS is that it check-sums everything. So checking for validity of all data is just one command that runs close to max read speed of the drive. I periodically run it and if the errors will be reported I will restore data from the backup.

When and why to deprecate filesystems

Posted Mar 8, 2022 22:20 UTC (Tue) by clump (subscriber, #27801) [Link]

There used to be the "ext3cow" project. I absolutely think the ext* filesystems should keep progressing. Of course ext4 is still being developed, but why not create an ext5 with features like COW?