Fedora and LVM

The schroot tool uses LVM snapshots to create and destroy transient scratch build environments. We use them on the Debian autobuilders. If you kick off 24 parallel builds on a 24 core system, the system will be dead as a doornail in a few tens of seconds. Entirely due to lvcreate/lvremote triggering what looks like some kernel locking bug. Even on systems only running a single build, we still regularly have lvcreate and lvremove failures. There's some fundamental bugs in LVM which really need fixing, and which I'm surprised haven't been addressed given that they are easily reproducible. schroot is admittedly a special case--most people don't churn through as many LVs as we do--rebuilding the whole archive is ~14 hours with 24 parallel builds, and ~18000 transient LVs (though it always died in under 5 mins, less than 100 LVs in). But it should certainly work without killing your system.

We now also support btrfs snapshots, and while the filesystem itself is still not perfect, I've not yet run into a single issue doing heavy parallelised snapshotting.

But my main beef with LVM is and has been since years that it's assembly is just broken and wrong. They assume that during boot there was a time where all devices have shown up, and which point they can invoke vgchange -ay, and that's the only time where they need to enumerate devices. But that's really not how things work these days, and haven't been working in the last decade or so. Hardware devices show up all the time, and we do not know when all connected devices have been detected, so in the boot process we don't actually know when we could invoke "vgchange -ay". Also, harddisks in times of USB and iSCSI show up at any time, depending on network status and their own initialization time and there are no general rules about when initialization has to be complete. The way distributions hack around the fact that they don't know when to invoke vgchange -ay is by pulling in udev-settle and the atrocity that is scsi-wait-scan. These hacks make things work for the "majority" of runs, and as long as you only have SATA disks and other pretty standard stuff. But the question is whether the "majority" is good enough where reliability is required, and whether limiting stuff to SATA and friends is such a good choice. But on top of that, it's just slow, since it basically is little more than just delaying the boot arbitrarily in the hope that all possible devices might have shown up when some time passes. Of course, if you are unlucky the delay is not sufficient, but still everybody has to endure the delay.

This issue has been known by the LVM folks since many years, and pointed out to them again and again. But nothing ever happened. Now, they say they'll soon have the "option" to make LVM work like any other hw daemon and actually wait on its own for precisely the devices it needs and not longer, thus not delaying the boot any bit longer than necessary. And they'll investigate if they can make that "option" default one day... At that speed I am sure that LVM well get discovery/assembly right only after the time it already has been replaced by btrfs... (btrfs in turn does get all this right: assembly is based of device plug events and a btrfs raid will delay the boot only exactly until the point where all devices it actually needs have shown up. btrfs raid is hence hotplug-safe, snappy at boot and absolutely reliable).

Anyway, the take-away is: LVM is the one major slowness in Fedora's boot. It also adds fragility where it shouldn't. And this hasn't changed in years. LVM is written they way hardware was in the 80's and 90's, but it is not how hardware has beeen working in the past 15 years or so.