Debian still having trouble with merged /usr

Posted Apr 6, 2022 2:06 UTC (Wed) by josh (subscriber, #17465)
In reply to: Debian still having trouble with merged /usr by KJ7RRV
Parent article: Debian still having trouble with merged /usr

Multiple reasons.

First, technical challenges: HURD tried to move from /usr to / and had quite a bit more trouble with it, and that trouble warned against trying it on a larger scale. It's much easier to move from / to /usr than from /usr to / ; fewer things break.

Second, consolidation: moving *into* /usr gives us a single directory containing all files managed by the distro / package manager. This has quite a few advantages. It makes sharing /usr among several chroots or containers feasible (one /usr, different /etc and /var). It makes versioning and A/B upgrades (upgrade to new image, fall back to old image if new image doesn't boot) simpler and easier. It makes it easy to lock down /usr with something like fs-verity.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 15:33 UTC (Wed) by juliank (guest, #45896) [Link] (1 responses)

Second argument doesn't really hold at the moment, as /etc and /var are managed by the package manager too.

Debian still having trouble with merged /usr

Posted Apr 6, 2022 16:00 UTC (Wed) by bluca (subscriber, #118303) [Link]

It can actually work, and it's getting better all the time as more and more software is updated to avoid a hard-reliance on a pre-populated /etc. Fresh from the press: https://0pointer.net/blog/running-an-container-off-the-ho...

Debian still having trouble with merged /usr

Posted Apr 7, 2022 5:51 UTC (Thu) by smurf (subscriber, #17840) [Link] (14 responses)

… except for the fact that dpkg's database itself doesn't live in /usr, so snapshotting it breaks your system.

The easiest remedy for this is to move /var/lib/dpkg to /usr/state/ (or wherever), and bind-mount /var/lib/dpkg to that. No, a symlink isn't sufficient.

Debian still having trouble with merged /usr

Posted Apr 7, 2022 6:10 UTC (Thu) by josh (subscriber, #17465) [Link] (13 responses)

If you're working with a snapshot of a system, and you're updating it as a snapshot rather than via individual packages, you don't typically need the tools used to put that snapshot together. You can (and often do) omit the dpkg database from a container image based on Debian.

Debian still having trouble with merged /usr

Posted Apr 7, 2022 8:14 UTC (Thu) by bluca (subscriber, #118303) [Link] (12 responses)

That said, moving the database to /usr makes a ton of sense, and Fedora is doing just that with the rpm database - of course we are talking about dpkg, so no chance of that ever happening

Debian still having trouble with merged /usr

Posted Apr 8, 2022 4:25 UTC (Fri) by pabs (subscriber, #43278) [Link] (11 responses)

There are things in the dpkg database that aren't read-only outside of dpkg runs, like alternatives. So some of it could move but not all of it.

Debian still having trouble with merged /usr

Posted Apr 8, 2022 11:09 UTC (Fri) by smurf (subscriber, #17840) [Link] (10 responses)

Managing /etc is another of those thorny problems that dpkg should have tackled 10 years ago but didn't.

Specifically you want a 3-way merge between the old /etc, the new one as updated by dpkg, and your current /etc contents. Surprise, there's etckeeper and git, so this is a solved problem.

Solution: teach dpkg to redirect any files in /etc to /usr/state/etc instead, use etckeeper to check those into git, pull that from your system's /etc, ask the user to fix any merge conflicts, done. Maybe do the manual merge in a scratch /run/etc directory so the system isn't in a broken state while you do this. Ditto for updating config file formats to newer versions, and other shenanigans some packages get into.

The chances of this actually happening while dpkg's current maintainer guards their fiefdom are somewhat less than zero, however. Otherwise I'd've started implementing this scheme years ago.

Debian still having trouble with merged /usr

Posted Apr 8, 2022 11:36 UTC (Fri) by farnz (subscriber, #17727) [Link] (9 responses)

Ultimately, the deep bug is outside dpkg - programs should read and merge multiple config locations, so that you can have /usr/etc containing the packaged config, and /etc containing the user config.

Debian still having trouble with merged /usr

Posted Apr 9, 2022 1:39 UTC (Sat) by josh (subscriber, #17465) [Link] (8 responses)

Exactly. Ultimately, we'll hopefully end up with a near-empty /etc containing only the divergence from the defaults.

Debian still having trouble with merged /usr

Posted Apr 9, 2022 2:22 UTC (Sat) by pabs (subscriber, #43278) [Link] (7 responses)

The amount of configuration diversity in /etc is extremely high. I think we need a systemd-esq initiative to solve that. Perhaps the next thing systemd needs is a library for configuration that everything using /etc could switch to.

Debian still having trouble with merged /usr

Posted Apr 9, 2022 9:37 UTC (Sat) by bluca (subscriber, #118303) [Link]

There's already a library that implements this pattern, built by suse, libeconf

Debian still having trouble with merged /usr

Posted Apr 9, 2022 9:39 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

libelectra exists, but is ignored by everyone.

Debian still having trouble with merged /usr

Posted Apr 9, 2022 9:51 UTC (Sat) by smurf (subscriber, #17840) [Link] (4 responses)

The problem with this is that there's no easy way to get backward compatibility unless you do something file system-ish.

OpenWRT tries to do something along these lines. Rather than hack the individual services they invented their own data structure (still text based, but a common format – though personally I'd use YAML instead of their scheme) and then generate the service-specific config files from that. Requires a generator script for each service supported that way, of course.

Debian still having trouble with merged /usr

Posted Apr 12, 2022 0:03 UTC (Tue) by pabs (subscriber, #43278) [Link]

Having just had to deal under time pressure with the OpenWRT config vs generated upstream config, where the OpenWRT wiki didn't adequately document the OpenWRT config and the upstream wiki documented the upstream config not the OpenWRT config, I have to say that this is an incredibly bad idea. I essentially had to read through the init script that generates the upstream config, run it under shell tracing and try and guess how to structure the OpenWRT config and what the options were. Since the upstream config is generated by a script containing hard-coded options, it is also possible the OpenWRT generatable upstream config isn't as full-featured as the upstream config, or could be outdated compared to upstream.

OTOH, the OpenWRT approach is *great* in the situation where there is a reasonably complete web app for the config in the Luci web interface. A discoverable graphical interface is often way easier to use than config file formats with a wide range of options, some of which might not be well documented.

Debian still having trouble with merged /usr

Posted Apr 12, 2022 10:05 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

One other very interesting approach is https://augeas.net/ - instead of translating a custom text/YAML into configuration files, it provides a uniform interface to most configs. I'm honestly stumped why it's not used more widely.

Debian still having trouble with merged /usr

Posted Apr 12, 2022 11:04 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

Augeas is a tool I have used before including via Puppet. It’s error messages can be cryptic and it seems unmaintained now.

Debian still having trouble with merged /usr

Posted Apr 12, 2022 19:56 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

Looking at their Github history, there seems to be a fair amount of activity. I guess there's just not that much stuff that needs changing, apart from lense updates.