LWN: Comments on "Seeking the endgame for Debian's /usr merge"

Erm, bind mounts?

nix — Tue, 05 Sep 2023 15:17:33 +0000

> dpkg knows nothing about /bin and /usr/bin being the same thing. So it installs package B, which puts files in /usr/bin and /usr/lib (which happen to also be /bin and /lib but it doesn't know that). Then it deletes the old, obsolete files from package A, which package B replaced. But the files in package A are the same as the files in package B, through a symlink, so dpkg has just deleted the files that were moved from A to B and then from /bin and /lib to /usr/bin and /usr/lib, but has no idea that it did this and appears to succeed.

So maybe I'm just blinded by boredom (stuck in a hospital waiting room, typing on a phone) but there seems to be a trivial one-liner fix for this in dpkg: when considering whether an old file F should be removed due to package B replacing obsolete files in package A, see if the (dev, ino) pair of F is used by any of the files in package B (trivial to determine, just remember them as you unpack them in a hashtable or something). Only delete files for which their (dev, ino) is not also the (dev, ino) of any files in the new package.

Bingo, handles not only /bin -> /usr/bin replacements in a generic fashion but also people doing weird things like bind-mounting bits of the system into unusual places across a d-u. mount --bind/--move etc do exist and people will use them... it *does* rely on (dev, ino) reliably identifying file uniqueness within a single boot, but all sorts of things break if that's not true so Linux even makes it work, more or less, on FAT...

Benefits

farnz — Mon, 19 Jun 2023 13:45:07 +0000

Having the root directory be completely ephemeral (could be a tmpfs set up by the initramfs) is useful, because it makes it trivial to do quite major surgery on your installation.

If you have the virtual filesystems (/proc, /dev, /sys etc), a /tmp for "temporary files, may be deleted any time", a /system (could be named /usr) for the installed system, and a /user-data (could be named /home) for user data, you now have some nice properties. First, you can unmount the user data at any time, in order to protect it somewhat from mistakes. Second, you can do things like take a snapshot of the current system area, do major surgery (e.g. try to convert in place from x86-32 to x86-64), and just roll back cleanly if your major surgery goes wrong. Thirdly, you now don't treat any partition specially in a rescue situation - because / is an ephemeral tmpfs, you just create one, mount in all the FSes, and now you have your rescue layout to examine.

With 20/20 hindsight, putting things directly on the root FS was a bad decision. But it's often better to make a bad decision and get something working than to never get anything to run at all for fear of making a mistake.

Benefits

anselm — Mon, 19 Jun 2023 12:57:39 +0000

Yes, but all of these also apply to a post-/usr-merge system.

The root/usr split is an accident of history that is now finally getting fixed for good (in Debian GNU/Linux, anyway – as with. e.g., the init system, most other Unix-like systems including many other Linux distributions have addressed this ages ago). There's really no more to be said other than let's get it over with already.

Seeking the endgame for Debian's /usr merge

farnz — Mon, 19 Jun 2023 11:33:50 +0000

That's roughly the same level of support as you get from Debian, though - because you'll get the same response from the Debian volunteers, too (as they don't have unlimited time to spend fixing your systems for you).

Benefits

geert — Mon, 19 Jun 2023 09:25:42 +0000

- You can mount /usr read-only most of the time, which means
- less chance of it becoming corrupted, either due to manual errors or due to bugs,
- no need to wait for fsck to finish checking your huge /usr file system on every boot.
- You can share/clone /usr among multiple machines (compared to /var, which is machine-specific).

Seeking the endgame for Debian's /usr merge

mathstuf — Sun, 18 Jun 2023 22:41:01 +0000

While I'm sure the RHEL upgrade is *possible*, I believe the gripe is that if something breaks, Red Hat's support for such attempts is probably something along the lines of "you have backups and fresh install media, right?".

Benefits

birdie — Sun, 18 Jun 2023 18:55:49 +0000

> Why should the structure of today's systems be beholden to the fact that 50 years ago some guys couldn't fit all of their software on one hard disk?

Exactly.

I wonder what benefits the original divided system had. I cannot think of any. Yeah, 30 year ago or so /usr was too big and /bin /lib /sbin were necessary to boot the system. We've long migrated to initrd which made this division redundant. Modern disk are crazy spacious, so again this division is redundant.

Seeking the endgame for Debian's /usr merge

birdie — Sun, 18 Jun 2023 18:55:30 +0000

> IIRC other distros have done it by requiring a full fresh install.

I've been upgrading Fedora continuously for almost two decades on one of my PCs.

I'm sorry but your assertion at least for Fedora/RHEL is patently false.

I'm quite sure you can upgrade RHEL 2.1 up to 9.0 without any issues as well.

Erm, bind mounts?

fest3er — Thu, 15 Jun 2023 19:42:48 +0000

Would it perhaps be better to handle pkg upgrades differently?
- add /upg-root/{bin,lib,etal} to PATH (earlier in PATH so old-pkg is used until new-pkg is ready for use)
- move old-pkg's files to /upg-root/...
- install new-pkg
- upgrade old-pkg's config/etc. files to /....
- clear /upg-root

The old pkg is moved aside, but still usable. The new pkg is installed, prepared and configured. The old pkg is erased. This would eliminate the cornfusion of which files are old and which are new, which are gnarled and which are true. And would work after pkgs are split.

I encountered related problems as I attempted to build an initramfs for a small distro I manage. I found that if I made /bin a symlink to /usr/bin, at least some pkgs would, for example, replace the /bin symlink with a real dir. I finally gave up, let stuff install where it wanted to, then moved everything from /{bin,lib,etal} to /usr/..., and replaced those dirs in / with symlinks to those in /usr. (Or maybe I moved stuff from /usr to /; don't remember now, but it's the same problem.)

Whilst I only have about 200 pkgs to handle, many of them make it easy to change install locations, but enough of them seem to obfuscate how to change where they install as to make it more of a bother than it's worth.

Debian's /{bin,lib,etal} scheme is workable, but may still cause problems when someone builds and installs a pkg only to discover that the pkg replaces the /bin and /lib symlinks with real dirs, and then spends hours figuring out what went wrong. (Not everyone is an experienced admin or developer. And even if they are, the process of creating a .deb from scratch is so poorly documented that most people will just give up and install directly.)

Merge /bin and /sbin

fest3er — Thu, 15 Jun 2023 19:01:11 +0000

My tuppence. Would 'admin space' or 'boot space' describe it better? Granted it's all still 'kernel space' and 'user space' (or 'non-kernel space'), but it seems reasonable to me that categorizing things as 'needed to control/prepare the hardware' (kernel space), 'needed to boot the system' (boot space), needed to administer the system (admin space), and 'needed by normal users of the system' (user space) would be helpful in more easily identifying things that are needed for specific purposes.

Benefits

anselm — Sun, 11 Jun 2023 11:07:15 +0000

The main observation here is that once you've done the /usr merge, there are various advantages that result from this, but the main argument against it seems to be “it's not what we're used to, and we don't like it”. (Now where have we heard that before?) It's also not really a systemd thing; systemd doesn't care whether a system has been /usr-merged or not.

Why should the structure of today's systems be beholden to the fact that 50 years ago some guys couldn't fit all of their software on one hard disk?

Fedora in-place upgrades

DemiMarie — Sat, 10 Jun 2023 20:57:53 +0000

Interestingly, upgrading in-place from Fedora 25 to 32 worked well enough that Qubes OS’s in-place upgrade tool did exactly that in dom0. It probably helped that most Qubes OS users had roughly the same set of packages installed in dom0.

Merge /bin and /sbin

mezcalero — Thu, 08 Jun 2023 08:00:01 +0000

I think merging those two would make sense too. Many distros (such as Fedora) have both in $PATH anyway, so the distinction is pointless these days. But unlike the /usr/ merge it's not that much of an enabler of new things, hence I guess it has not found the big champion yet.

But yeah, if Fedora would decide to merge the two I'd absolutely support that.

(Traditionally there were some ill-designed tools that installed into both dirs, and exposed different behaviour in the two dirs. Not sure how much that still matters)

Lennart

Erm, bind mounts?

rra — Wed, 07 Jun 2023 18:01:30 +0000

There has been a lot of musing and I think at least some writing in one of the working branches in conjunction with a fix in progress, but I don't know how far it's gotten. I personally haven't had the time to meaningfully contribute and therefore have been trying to stay out of people's way, so I haven't been following the implementation details that closely.

A lot of the work so far has been finding new edge cases that break and then estimating how prevalent they are in the archive.

Erm, bind mounts?

mathstuf — Wed, 07 Jun 2023 17:35:56 +0000

> But the most important part will be thorough testing of the edge cases to ensure that we didn't create new problems while trying to fix the old problem.

Not that I'm volunteering, but are these edge cases at least captured as test cases in the relevant project(s)? Or just mused about on mailing lists?

Erm, bind mounts?

rra — Wed, 07 Jun 2023 16:02:25 +0000

> I believe that a generic solution would be hard. But a one-off that does a hardcoded check for /bin and /usr/bin and /sbin and /usr/sbin/ and /lib and /usr/lib and the few other prefixes that matter could be implemented without any problem.

You would think that, and of course we did think that, and various people tried, and failed. This turns out to create quite a lot of problems, even if you're willing to hard-code those specific exceptions, because there are complex interactions between package installation, diversions, alternatives, dpkg's existing file databases, existing features allowing directories to be symlinks to non-managed directories elsewhere on the system, and so forth.

Certainly if you think it's easy and we're all missing something, we'd love the help. But the most important part will be thorough testing of the edge cases to ensure that we didn't create new problems while trying to fix the old problem.

>> The dpkg maintainer also believes the way /usr-merge was done is completely wrong from a technical perspective
> This is the real problem. If there was will to solve the issue, dpkg could have done this approx. 10 years ago.

There was a time when I was wondering if that was the case, but the subsequent detailed analyses by people who are not the dpkg maintainer have convinced me that this is not true and the dpkg maintainer is indeed correct that this is a very invasive and non-trivial change.

Erm, bind mounts?

zuki — Wed, 07 Jun 2023 15:26:01 +0000

> dpkg doesn't know that /bin has been aliased to /usr/bin. It has no mechanism to know this fact, and its internal architecture is fairly adverse to the whole idea, so it's proving quite difficult to add.

Meh. I believe that a generic solution would be hard. But a one-off that does a hardcoded check for /bin and /usr/bin and /sbin and /usr/sbin/ and /lib and /usr/lib and the few other prefixes that matter could be implemented without any problem.

> The dpkg maintainer also believes the way /usr-merge was done is completely wrong from a technical perspective

This is the real problem. If there was will to solve the issue, dpkg could have done this approx. 10 years ago.

Seeking the endgame for Debian's /usr merge

bluca — Tue, 06 Jun 2023 14:18:35 +0000

It is not cleaner, it's doomed to fail. We know this because we've seen it happen.

Seeking the endgame for Debian's /usr merge

KaiRo — Tue, 06 Jun 2023 12:50:02 +0000

Oh, I had misremembered then from what was discussed back then. I would have found that other approach cleaner but it is what it is. At the time, i was on my phone and couldn't actually check on my own system.

Benefits

jccleaver — Mon, 05 Jun 2023 22:10:16 +0000

>All distributions decided that "/usr merge" should be done, even Debian. What this article is about is that Debian is being Debian so it discusses how that should be done for about a decade.

Few if any distributions decided that "'/usr merge' should be done;" they decided to stop fighting Lennart and the systemd "cabal" and relented. Big difference.

Erm, bind mounts?

pmatilai — Mon, 05 Jun 2023 11:20:03 +0000

Interesting. That does make /usrmove in Debian quite different to rpm based distros. Rpm tracks files via a mechanism called fingerprinting pretty much since the beginning, which allows it to identify and handle cases where directory symlinks make a file appear at multiple places. Not just for conflict management but dependency resolution as well.

Seeking the endgame for Debian's /usr merge

patrakov — Mon, 05 Jun 2023 06:41:31 +0000

Back in 2011, when I worked for Google, I saw a colleague removing a test that broke because it tested an implementation detail that was no longer applicable and should have never been covered by a test. Is this still permitted?

Seeking the endgame for Debian's /usr merge

mirabilos — Sun, 04 Jun 2023 14:24:52 +0000

Why would you even move the files if the directory symlinks are still in place?

And the directory symlinks *have* to stay in place for things like #!/bin/sh precisely (the FHS even demands that, if ed(1) is installed, it’s reachable as /bin/ed, and I successfully defended this against a distro trying to move it to /usr/bin/ early).

Anyway, this systemdification of Debian is creepy.

Seeking the endgame for Debian's /usr merge

pabs — Sun, 04 Jun 2023 07:21:05 +0000

Wiki pages related to this:

https://wiki.debian.org/DebianUpgrade
https://wiki.debian.org/AutomatedUpgrade

Seeking the endgame for Debian's /usr merge

rjones — Sun, 04 Jun 2023 00:44:54 +0000

The difference boils down to "who the organization is in service of".

In businesses, in a sane situation (which not all situations are sane), the goal is to serve their customers. The customers have what the business wants and to get it they need to convince the customer that it's worth it to give it to them. So they try to identify the needs of customers, which is not a easy task. Once they have some needs identified then they need to figure out what it is worth to the customers... how much customers are willing to spend on fulfilling that need. Which is even harder. It's nearly impossible to get that right every time. Lots of products fail because of this.

Once the business leaders have those things sussed out as well as possible then it is up to the engineers to try to make it a reality. They have to figure out a solution that costs low enough to be profitable. Not just in terms of money or capital investment, but in terms of time. Sometimes it's not realistic and sometimes it is. But that is how "in service of a customer" should work in most situations. Profit/loss is how you determine how good of a job you are doing.

In the case of The Debian Project their "customer" is The Debian Project. People join and work on the project for the sake of the project and themselves. The cost is mostly individual/personal investment of time and effort into the project. And profit is determined by how much individual/personal benefit/fulfillment they derive from the project. What second and third parties derive from the project is almost incidental in comparison to what the maintainers get out of it.

Since the costs and profits are so "squishy" in nature.. it makes proper accounting mostly impossible. So it is very difficult to figure out how well the project is doing at serving itself. It makes for a tight balancing act between how much the project needs to keep people happy versus how much work they impose on other people in the project all the while fulfilling the technical requirements.

It is a difficult thing to get right. And it is a testimony to the governance model of Debian that they get it close enough to being right to survive so long. It is quite remarkable.

Merge /bin and /sbin

geert — Sat, 03 Jun 2023 11:34:25 +0000

Both /bin and /sbin (after it was introduced) used to be on the root partition. /sbin contains the binaries needed only for managing the system (e.g. mount), while /bin contains binaries used by both ordinary users and system administrators (e.g. ls).
/usr/bin (and /usr/sbin, and /usr/local/... , opt/...) contains the binaries that are not needed to bring the system into working state.

Benefits

vasvir — Sat, 03 Jun 2023 05:53:55 +0000

I think the benefits listed in the message are substantial even though the poster thinks that while cool may be the cost is not justifiable.

Path portability is not a cosmetic change and I have been bitten many time by it.

The second one to have the system under or directory /usr is tidier than the alternative. Everything can be done both ways albeit with minor difficulties. The question is if we never clean our home at which point it becomes intolerable? Maybe we get used to its quirks and its little foot guns here and there and even somebody depends on it xkcd style. But is this reason to never clean up? What are newcomers supposed to feel when everything feels so crufty and overloaded with history?

I understand that I am not the one that does the cleaning and tide up so my opinion doesn't carry any value here.

Seeking the endgame for Debian's /usr merge

NYKevin — Sat, 03 Jun 2023 03:37:21 +0000

It depends on both the workplace and the specific change being made. At Google (my workplace), large-scale changes are typically coordinated in a centralized fashion, and the most important rule is that you can't break existing tests or functionality. This is possible because Google uses a monorepo, so we have the ability to find all of the tests you potentially affected and re-run them. It would be illegal to break a bunch of tests and then tell the affected teams "fix it yourself." The person making the breaking change would be asked (told) to back it out, fix the tests, and try again.

More complicated changes are sometimes made in a top-down fashion, but this tends to be much slower and less efficient, and is usually only done for things that require a lot of subject-matter expertise and individualized judgment (often to the point where each individual team will be writing a separate design document about their transition process and all the weird pain points they ran into). There is also a degree of managerial discretion in terms of exactly who works on which projects, and in what order, so these mandates may not always be the top priority.

Erm, bind mounts?

rra — Sat, 03 Jun 2023 00:11:48 +0000

This is the root of the entire problem.

dpkg doesn't know that /bin has been aliased to /usr/bin. It has no mechanism to know this fact, and its internal architecture is fairly adverse to the whole idea, so it's proving quite difficult to add. The dpkg maintainer also believes the way /usr-merge was done is completely wrong from a technical perspective, as noted in the article, but even if they didn't feel that way, adding this understanding to dpkg is non-trivial.

There are a bunch of reasons why one cannot just ship those symlinks in a Debian package right now, but one of them is the transition from an unmerged to a merged system. This isn't something dpkg understands how to do.

As a result, /usr was merged in Debian via a separate package that manipulates the file system directly outside of the dpkg package installation process. This was an expedient approach to get the transition to happen (and is also what was done in Ubuntu), but now we're working through the consequences of creating this sort of fundamental inconsistency between the file system and dpkg's understanding of the file system.

Erm, bind mounts?

mjg59 — Sat, 03 Jun 2023 00:08:07 +0000

If /bin is a symlink to /usr/bin, and package A installs a file as /bin/foo, it will actually be installed to /usr/bin/foo. However, dpkg's database will still track the file path as /bin/foo. If package B then installs a file as /usr/bin/foo, this will overwrite the file belonging to package A (because they're actually the same), but dpkg will track the file belonging to package B as /usr/bin/foo. We now have to references to the same file - one belonging to package A, and one to package B. If we now uninstall package A, dpkg will remove /bin/foo, except that means it's actually removing /usr/bin/foo, and now package B has a file missing.

Erm, bind mounts?

pebolle — Fri, 02 Jun 2023 23:26:42 +0000

> dpkg knows nothing about /bin and /usr/bin being the same thing.

Come again?

Benefits

pebolle — Fri, 02 Jun 2023 23:20:24 +0000

You're beating a dead horse.

All distributions decided that "/usr merge" should be done, even Debian. What this article is about is that Debian is being Debian so it discusses how that should be done for about a decade.

(It sometimes feels there's a tendency among the LWN readers to think they're more rational than people not into software. "MBAs", lawyers, "beancounters", politicians, whatever. They're too dumb to understand software. Ha, ha!

Guess what. The fact that LWN is still reporting on this /usr nonsense indicates that people writing software are prone to make the same mistakes that those that do not write software make. We're all the same! Or worse, perhaps the hubris of software developers makes them more likely to make mistakes.)

Erm, bind mounts?

rra — Fri, 02 Jun 2023 23:13:43 +0000

Apart from the issue that bind mounts don't fix the dpkg aliasing problems, I'm not seeing how this would work conceptually. The pre-/usr-merge starting point was that, e.g., bash was in /bin/bash. There is no /usr/bin/bash. If you bind-mount /bin to /usr/bin, now there's no visible bash on the system. This seems bad?

Seeking the endgame for Debian's /usr merge

bluca — Fri, 02 Jun 2023 23:05:13 +0000

There weren't many IIRC, and most of those were buggy and fixing them was a good thing regardless.

Merge /bin and /sbin

Wol — Fri, 02 Jun 2023 22:58:19 +0000

But yes it does. /sbin is binaries required by the system, so they have to be on the root partition, while /bin is binaries required by the users, so they can have their own partition to avoid filling up the (back then) small system disk.

Cheers,
Wol

Erm, bind mounts?

rra — Fri, 02 Jun 2023 22:46:56 +0000

There are a few different problems, some of which involve complex interactions, but the simplest case looks like this:

1. Package A contains files in /bin or /lib
2. Those files move to a different package B, staying in the same location
3. Package B then moves those files into /usr/bin or /usr/lib

User upgrades package A to a step 3 package B.

dpkg knows nothing about /bin and /usr/bin being the same thing. So it installs package B, which puts files in /usr/bin and /usr/lib (which happen to also be /bin and /lib but it doesn't know that). Then it deletes the old, obsolete files from package A, which package B replaced. But the files in package A are the same as the files in package B, through a symlink, so dpkg has just deleted the files that were moved from A to B and then from /bin and /lib to /usr/bin and /usr/lib, but has no idea that it did this and appears to succeed.

I hope I have the details right. The precise scenarios are subjects of many-message threads and there are a lot of variations. But I think this is the basic idea, if not precisely correct in all the particulars.

Erm, bind mounts?

p2mate — Fri, 02 Jun 2023 22:36:03 +0000

I think dpkg also deletes files which were part of the previous version of a package and not part of the new to be installed version, presumably to prevent old files from sticking around forever. This obviously causes problems if version n - 1 had its files in /bin and version n has them /usr/bin and /bin is symlinked to /usr/bin, unless dpkg realizes this, but apparently it doesn't.

Benefits

ballombe — Fri, 02 Jun 2023 22:29:34 +0000

So Sam lists only two benefits, the first being cosmetic and the second entirely speculative, using as examples OS that did not do the merge. This is far from compelling.

Seeking the endgame for Debian's /usr merge

vstinner — Fri, 02 Jun 2023 22:13:41 +0000

I didn't know that Debian doesn't have an unique and dedicated tool to upgrade Debian. That's interesting.

In Fedora, "dnf system-upgrade" is the command line too for that. I suppose that GNOME Software runs it internally for a system upgrade.

Erm, bind mounts?

pebolle — Fri, 02 Jun 2023 22:12:41 +0000

> The problems caused by the mismatch between dpkg's view of the world and the file system are due to the creation of any aliases for files that dpkg is managing.

This I don't get.

rpm packages declare which files they want to drop where. The rpm tool drops those files at those locations and updates its database. Any difference between what rpm registers and what is actually happening would be a severe bug.

Doesn't dpkg work like that?