Seeking the endgame for Debian's /usr merge

At its core, the /usr merge is the project to move files out of root-level directories like /bin and /lib into the equivalent directories under /usr. Most distributors took a "rip the bandage off quickly" approach to this change, causing it to simply happen over the course of a single major-version upgrade. It was an abrupt change and not without problems but, for the most part, distributors got through it quickly.

Debian has taken a slower approach, spreading the transition over a few releases. If one looks closely at a fresh Debian install, either the current Debian 11 ("bullseye") or the upcoming "bookworm" release, one will see that directories like /bin and /lib exist only as symbolic links into their equivalent directories under /usr. It would seem that the merge process is complete.

There are some rough edges, though. If one looks into the packages installed on even a bookworm release, one will see that many of them are still installing into the top-level /bin and /lib directories. If the release is called anything but "Debian" (if it is a derivative distribution, for example), the dpkg package manager will emit a scary warning about how the system is broken. And, if one looks at the project's mailing lists, one will find ongoing discussions about the remaining problems with the transition, along with comments like this one from Bdale Garbee, who has in the past served as both Debian project leader and Technical Committee chair:

Merged-/usr seems to me to have brought great pain with no discernible benefit to Debian so far, and I at least have completely lost the thread on what the point of doing it was supposed to be.

All of this reflects a couple of fundamental problems that the project has not, as yet, figured out how to resolve.

dpkg and aliases

One of these problems is that much of this transition has been effected outside of dpkg, with the result that the Debian package manager lacks a complete understanding of how the system is structured. Specifically, the use of symbolic links to replace the top-level directories creates "aliases" — multiple paths to files — that dpkg does not properly understand. As a result, there is a scenario — involving renaming a file within a package, then moving that file to a different package — that can cause dpkg to delete files belonging to an installed package, thus corrupting the system when those packages are installed.

Problems like this are the inspiration behind the warning that dpkg prints on merged-/usr systems. The specific failure mode may seem obscure, but it is aggravated by a couple of factors. The first of those is that the end game for this whole transition involves moving files into their expected locations; rather than installing bash into /bin, the package would be changed to put it into /usr/bin instead. That is a lot of file moves; add to that the fact that moving files between packages is reasonably common in Debian, and the chances of steering users into this bug look fairly high.

Corrupted systems after an upgrade is just not part of the image that the Debian project has for itself, so this bug is a significant problem. It is the reason why the project has maintained a moratorium on the movement of files to their new locations under /usr — a moratorium that was recently renewed by the Technical Committee.

It is not uncommon to see project members blaming dpkg maintainer Guillem Jover for this problem. Jover has not hidden his disdain for how the transition has been handled, and is seen by many as preferring the addition of scary warnings over actually addressing the problem. But a careful reading of the lists can yield a more nuanced view. An attempt to have the Technical Committee force the complete removal of the warning from dpkg (for derivatives as well as for Debian itself) has not gone anywhere, and a number of developers feel that the substance (if not the wording) of the warning is appropriate. As Helmut Grohne put it:

The present discussion clearly shows that dpkg's support for how Debian deals with merged /usr is lacking. We are dealing with multiple file-loss scenarios (something we otherwise consider grave) and issuing a warning about such behaviour seems fine to me.

Meanwhile, developers continue to seek a way to fix this problem. Grohne recently posted an extended proposal on how dpkg could gain some new options to inform it of the existence of directories. There has been a variety of responses. Luca Boccassi said that he intended to simply force-move everything after the bookworm release, ending the aliasing as seen by dpkg and thus avoiding the problem; "That should bring the matter to an end, without needing to modify dpkg". Simon McVittie, though, worried that this sort of move was particularly likely to trigger the lost-file bug rather than avoid it. Grohne tried to adapt Boccassi's plan to avoid a number of problems that he found, but eventually decided that his approach, too, had too many problems. "The amount of complexity we are piling up here becomes non-trivial".

Jover, meanwhile, responded to the proposal while avoiding the main development list; he described it as "conceptually wrong" and provided his reasoning in detail. Among other things, informing dpkg of these links using separate commands adds a "source of truth" about the installed system that is not found in any of the packages installed there. Jover did put some effort, in that response, into thinking about how the situation might be improved, but did not contribute thereafter.

The end result of all this is that the problems with dpkg remain, with more likely to be discovered, and there is still no clear picture of how they will be addressed. Until that issue is resolved, this transition will continue to loom over the Debian project.

Bootstrapping and interpreters

Most readers will be familiar with the "shebang" lines at the beginning of scripts that specify which interpreter a script should be fed to:

    #!/bin/sh

It may be a bit more surprising to learn that natively compiled programs also specify an interpreter — specifically, the program that will load the executable into memory, link in dynamic libraries, and prepare it to run. This interpreter is stashed into the ELF program header as the PT_INTERP variable, and will normally be set to: /lib64/ld-linux-x86-64.so.2 on 64-bit Linux systems.

Regardless of whether one is trying to run a script or a compiled binary, little joy will result if the specified interpreter cannot be found. This creates a bit of a problem given that, in a merged-/usr system, these interpreters should be moved into /usr/bin and /usr/lib64, turning all of those interpreter strings into dangling references. Most of the time, the symbolic links added to the root directory paper over this problem, everything works, and nobody notices. But there is an exception.

The creation of a new root disk image containing a Debian installation is done with a bootstrap tool, usually either debootstrap or mmdebstrap. The bootstrap tools install the "essential set" of packages that every Debian system needs, then run their configuration scripts (in a chroot setting) to set up a basic, functioning installation. If, however, the packages in that essential set are changed to install their files under /usr, the programs needed to finish the setup will not find their interpreters and the installation will fail; that tends to lead to another round of calls to give up on this transition entirely.

In theory, all that one needs to do is to ensure that the appropriate symbolic links are in place early in the bootstrap process. In practice, it is far from so simple; among other things, the presence of those links can, once again, confuse dpkg. At one point, Boccassi suggested that Debian could be built with a modified PT_INTERP value and changed shebang lines to sidestep the problem, but that led to a flurry of complaints about breaking the Linux ABI, breaking compatibility with older Debian releases, and losing the ability to compile programs that would run on different distributions. As Russ Allbery put it: "breaking the ABI is a Really Big Deal". So that idea did not get far.

Grohne took the time to put together a detailed look at the problem that should be required reading for anybody who is interested in the bootstrapping challenges. He outlined four different ways in which bootstrapping could be fixed and discussed the difficulties with each. His recommendation is to leave a handful of files in their old location for now while, in the long term, changing the way bootstrapping works to ensure that the requisite symbolic links are present from the beginning, and that they do not complicate the installation thereafter.

Now what?

As of this writing, neither problem has a definitive solution identified for it. A combination of avoiding file moves and keeping the interpreters at their old locations may be enough to get the bookworm release out the door successfully, but leaves a number of problems in place. The transition cannot be said, by any stretch, to be complete.

Some of the participants in the discussion evidently feel that the remaining problems are easily solved and that this transition is essentially done. Others clearly worry that, not only are the problems difficult, but that problems are still being discovered and that there are more to come. The latter view seems likely to be the more accurate one. As Technical Committee chair Sean Whitten said:

I am far from being an expert on the details of merged-/usr. But one thing I've noticed is that among the people who have spent the most time looking into it, the majority think that simple fixes are not going to be sufficient. Only a few people who have spent a lot of time on it still think that the fixes that are required are relatively simple ones.

LWN last looked at this transition just over one year ago; at that time, many of those involved were describing it as a social failure, and possibly a technical failure as well. Since then, progress has been made, and many Debian users are running on systems that are, for all practical purposes, fully merged. But the technical and social challenges remain as intransigent as ever.

Former Technical Committee member Gunnar Wolf described the /usr merge as "the point we most came back to" during his time there. Perhaps a new spirit of cooperation and new technical insights will bring this episode to a close during the upcoming "trixie" development cycle. But it also seems plausible that the Technical Committee will find itself hearing arguments about the /usr merge for a while yet; the Debian community may well have more opportunities to wonder whether it would have been better to just rip the bandage off and be done with it years ago.