Debian still having trouble with merged /usr

The addition of the "/usr merge" feature has been something of longstanding mess in the Debian world. It seems like a relatively innocuous change, which is in keeping with the practice of most other distributions at this point; it effectively eliminates the top-level /bin, /sbin, and /lib* directories in order to move their contents to the corresponding locations under /usr. But ever since we first covered the feature introduction for Debian—more than six years ago—it has a been a recurring series of headaches within that community. Recent events have seemingly simply prolonged the pain, though perhaps the end is in sight.

Background

The /usr merge idea was first raised in "The Case for the /usr Merge" by Lennart Poettering in 2012. It came out of the systemd community, but was not meant to be systemd-specific, though, as the comment thread on that LWN post indicates, the idea was controversial at least in part because of its association with systemd. At its core, it was a straightforward plan to move the files in /bin to /usr/bin, then make a symbolic link so that /bin would resolve to /usr/bin; the same would be done for the other top-level directories of interest (/sbin and /lib*).

The main reason behind the merge was for compatibility with other Unix systems and to make building packages for Linux distributions easier; upstream software, build tools, and the like would no longer have to treat Linux specially. Anything that relied on /bin/foo or /usr/bin/foo would find what they were looking for, but everything would live under /usr. That would permit things like an immutable, distribution-supplied /usr, snapshots of the system state would be simpler, and so on.

Fedora, as is often the case, took the lead and implemented merged /usr, for Fedora 17 in 2012. Other distributions followed suit, with most of the mainstream distributions following the Fedora approach of having a "flag day" switch; new or upgraded versions of the distribution would either install a merged /usr or switch the system to use that as part of an upgrade. Supporting both the old and the new layout simultaneously was not something those distributions chose to do.

Debian took a more incremental approach, in part because it strives not to make wholesale changes to users' systems like those required by a flag-day upgrade to a merged /usr. In 2016, the ability to voluntarily switch to that scheme was added, then some attempts were made for newer versions of the distribution to be installed with a merged /usr by default. In 2018, the Debian build systems were switched to use that layout, but problems cropped up because packages built under a merged /usr would sometimes fail to work on non-merged systems. The location of some files was being resolved at build time to point into /usr/bin (for example), but those files only existed in /bin on the non-merged systems.

The build system changes were reverted, so that both types of systems could be supported. But the problem led Ian Jackson to ask the Debian technical committee (often abbreviated as CTTE or ctte), which is the ultimate arbiter of technical disputes within the project, to override the decision by the debootstrap developers to make merged /usr be the default. Debootstrap is used to create a base system into a directory that can then be used as a chroot environment for building additional packages, though it has other uses as well.

The committee declined to overrule the debootstrap maintainers, but did give a lengthy summary of the cases for and against /usr merge. The committee also used its advisory role to note that the then-under-development Debian 10 ("buster") would support both system types in a "weak" form, but suggested that it was desirable that Debian 11 ("bullseye") move toward a "middle" ground with better support for merged /usr.

Debian, being Debian, discussed the issue multiple times along the way, at varying levels of intensity and rancor. Supporting both types of systems was difficult in various ways, leading some to say that either /usr merge should be made mandatory—or scrapped entirely. At the end of 2020, Ansgar Burchardt asked the technical committee to decide whether Debian 12 ("bookworm") should only support merged /usr systems. At the end of January 2021, the question was put to a vote, and the result was unanimous in favor of doing so:

The Technical Committee resolves that Debian 'bookworm' should support only the merged-usr root filesystem layout, dropping support for the non-merged-usr layout.

Until after the release of 'bullseye', any implementation of this resolution must be done in the 'experimental' distribution, or otherwise kept out of the critical paths for the release of 'bullseye'.

Symlink farms

That would seem to resolve the situation, in favor of a flag day for Debian 12, but, as might be guessed, things did not quite work out that way. During the discussion on the vote, committee member Simon McVittie expressed concern that there were two competing visions of what a merged /usr meant. He wondered: "Should we be more specific than this in what we vote on, to avoid later having to adjudicate between developers who say that a particular implementation is or isn't merged-usr?" That comment would foreshadow the events to come.

The "standard" way to support /usr merge, as described above, is sometimes called "merged /usr via aliased directories". It is what the usrmerge tool, which got the ball rolling for Debian in 2016, does; that mechanism is tracked on the UsrMerge page on the Debian wiki. An alternative, which is championed by dpkg maintainer Guillem Jover, uses "symlink farms", which McVittie described as follows:

an arrangement where all regular files that have traditionally been in /bin, /sbin, /lib and /lib64 are physically located in /usr, with /bin etc. becoming "symlink farms" containing symlinks like /bin/sh -> /usr/bin/sh, /lib/ld-linux.so.2 -> /usr/lib/ld-linux.so.2 and so on

There was some discussion in the bug entry about the advantages of the two different arrangements, but most of the participants seemed to think that the symlink-farm approach was not desirable, viable, or both. Along the way, McVittie realized that the earlier decision on debootstrap seemed to indicate that the committee thought that the aliased-directories mechanism was the singular approach to solving the /usr merge problem. His vote reflected that understanding.

The discussion moved to the mailing lists, where a certain amount of consensus was found, at least in some areas. McVittie opened a bug asking the committee to resolve the aliased directories versus symlink farm question along with others swirling around the upcoming /usr merge transition for bookworm. He was asking the committee to use its formal advisory role to offer guidance on how that transition should proceed.

A lengthy statement was put out for a vote by the committee; once again, it passed unanimously. The advice clearly rejected the symlink-farm approach, in favor of aliased directories, and made it clear that packages could not assume that systems have a merged /usr until Debian 13 ("trixie") development starts. Beyond that, the committee advised that upgrading from either type of system to Debian 12 should work, packages should build and function on both types of systems (until after Debian 12), and a few other items. The bug was marked as "done" on October 18 and, once again, it seemed that the situation was resolved.

Dpkg resistance

Meanwhile, though, the dpkg maintainer was adamantly opposed to the aliased-directories approach; his complaints were described in a MergedUsr page in the dpkg team's section of the wiki. It presented a long list of problems with that approach because the mechanism is handled outside of dpkg itself:

This approach goes behind dpkg's back, and has caused and [does] now cause problems due to the aliased directories, as multiple pathnames canonicalize into different pathnames that point to the same dentry, which can mess up anything that handles pathnames in databases and similar. Note: dpkg has supported for a long time symlinked directories as a way to allow local admins to manage filesystem size constraints, by moving directories contents into other partitions, but has never supported aliased directories via symlinks pointing to other directories already tracked by dpkg. This approach makes it impossible to know what is the canonical (from dpkg's PoV) pathname for an object just by looking at the filesystem, so it is trivial for users use the wrong one.

Many of the entries in the list were marked with a symbol indicating: "This approach is considered broken by design and is unsupported by dpkg. dpkg-buildinfo marks packages built on these systems as tainted in the .buildinfo file." The symlink-farm approach is superior, the page says, even though it still has the aliasing problem for the symbolic links that get installed:

This still suffers mild aliasing problems, but only for pathnames that will end up in both locations due to backwards compatibility symlinks, which should keep decreasing, and any such issue will self-heal over time, eventually ending up being just a handful of them. The big difference is that in the long term this is a tiny and decreasing bounded set of pathnames that might end up causing problems, while the other layout is an unbounded set affecting all pathnames permanently. Another thing that makes a difference is that the object is a symlink, and not the same object being accessed from different pathnames.

Both that page and the dpkg FAQ entry recommend using the dpkg-fsys-usrunmess utility, which "undoes the merged-/usr-via-aliased-dirs mess". So it was perhaps unsurprising when Josh Triplett reported in mid-March that dpkg had started to emit a warning when it was installed or upgraded:

Setting up dpkg (1.21.2) ...
dpkg: warning: System unsupported due to merged-usr-via-aliased-dirs.
dpkg: warning: See <https://wiki.debian.org/Teams/Dpkg/FAQ#broken-usrmerge>.

This escalation seems in direct contradiction to the tech-ctte decision in 994388. Moreover, this seems to effectively use package maintainer scripts as a means of directing a complaint at Debian users that has not gotten traction in other forums, and then directing such users at a wiki page that contradicts a prior project decision.

His report was aimed at the committee bug that resulted in the decision back in October. As Russ Allbery noted, that kind of a warning "will be perceived by users as an official declaration from Debian as a project that their system configuration is unsupported, while simultaneously this is the default installation mode for new systems and something that we have elsewhere said is a correct system configuration". Burchardt said that the warning was already causing confusion and asked the committee to decide quickly on how to handle it "to avoid this becoming yet another energy drain (we had several sufficiently long enough threads about this topic already)".

Bugs and fixes

There is a question of what "supported" means in the context of /usr merge, Helmut Grohne said; he was added to the committee at the beginning of the year after two members reached their term limit and retired from it. He said:

At this point, neither merged-/usr nor unmerged-/usr is supported well. Both are broken in one way or another and nobody steps up to fix the mess. In particular, the dpkg maintainer does not support merged-/usr in dpkg (which is his constitutional right as long as he does not block reasonable patches), but neither does anyone else. As such I find it difficult to disagree with the content of the warning. I do see how it confuses people. It definitely does not reach people who could do something about [it]. Rather it takes users as hostages.

As had been mentioned in some of the (many) threads surrounding this issue, Ubuntu has successfully made the move to a merged /usr; as Luca Boccassi put it: "And on top of new installations, old installations of Ubuntu upgrading to 21.10 and/or the soon-to-be-released 22.04 have been [forcefully] migrated too. They are not blocked, unsupported, or broken." There have been few problems with that transition, at least according to /usr merge proponents, so whatever bugs exist are not show-stopping. But Allbery cautioned against taking that too far:

We know that merged-/usr is buggy, in that one can construct a set of package operations that leave the system in an invalid state. We have a project disagreement over how serious those bugs are. No one is stepping forward to fix those bugs, which is indeed quite unfortunate. I personally strongly disagree with the belief that simply because Ubuntu hasn't seen many instances of this class of bugs while using a package set where people have not moved files between packages and out of /lib and /bin very much if at all, it is acceptable to leave dpkg in that buggy state.

[...] I personally am disappointed that the folks who have been pushing merged-/usr forward are willing to leave dpkg in a known-buggy state without attempting to patch it to fix the remaining issues. I realize that there are various obstacles in successfully doing that, not all of which are technical, but I want to believe that Debian is the sort of project that will do the hard work (both technical and social) to fix edge cases and maintain a high level of consistency and correctness.

But Triplett disagreed with that characterization:

It does not seem at all obvious that such patches would have been accepted, given the repeated vehement objections from the dpkg maintainer about the chosen approach. Those objections did not invite contribution; at every point, the assertion was that usrmerge was broken, not that dpkg needed help supporting it.

Boccassi said that looking at Ubuntu's experience is legitimate; the problems identified are relatively small and "it's believed to be impossible to get them fixed" over the maintainer's objections. He asked what else would be a reasonable way forward. Allbery replied that proponents of the change should create a patch to fix the problems that have been identified in dpkg even though it might be rejected. In part, it comes down to finding something that is actionable under the powers of the technical committee, he said in a follow-up: "It's difficult, procedurally, for the TC to do anything about a theoretical patch that someone could write but hasn't written."

Committee chair Sean Whitton agreed with that analysis, though he did say that he thought the committee could take action to get the warning changed or removed; other members concurred with that view. Shortly thereafter, Triplett pointed out that the warning message had changed, though it was not really any better in his view.

Grohne also wanted to see fixes for dpkg to better support merged /usr. He even mentioned the possibility of reverting the change:

If the resulting bugs do not get fixed, we may need to consider other means for limiting their impact. The most obvious method here is revisiting the decision and considering whether the /usr-merge may have failed. On a process level, it certainly has failed. At some point, we may need to look at a bigger picture than the technical one. If the people driving the change are not able to do it, then maybe we should not have that change in the first place and revert back to the known working state. Of course that route is not without cost.

A patch arrives

But as the thread progressed, Boccassi noted that a link to a dpkg patch had just been posted to IRC by a user named "uau". Allbery (among others) was elated to see the patch and immediately had some feedback on ways to improve it. Committee member Gunnar Wolf thought that the "patch seems sane from a first, very much 10000m-point-of-view"; he wondered if it had "been shared with Guillem, or included in any relevant bug report".

Boccassi followed up with some more information from the author of the patch, again from IRC. The author said "that some time ago the patch was presented to the dpkg maintainer, who rejected it with an answer along the lines of the usual 'usrmerge is broken by design', with no further comment". Boccassi wondered if those who had been looking for a patch of this nature would "try and take it forward themselves". Based on that conversation with the author, the license of the patch is the same as for dpkg, so that should not be a barrier.

Grohne said that he remembered some more concrete criticism of the patch and that the patch itself says that it is incomplete, so it cannot go into dpkg in its current form. He lamented the state of the transition, and once again raised the specter of reverting the whole thing:

The more and more I have to deal with the /usr-merge the more I get disappointed by how badly this transition is planned and carried out. In principle, the technical merits seem solvable to me, but the total failure on the process level leaves me wish for a revert. I am really surprised that instead of improving the process, you carry on with that destructive attitude. Given this, it seems unsurprising that Guillem does not want to interact with you. Of course that's not an excuse for implementing the recent changes to dpkg. The communication is clearly failing on both sides, which is why we're here at the ctte again.

Burchardt asked that Grohne stop suggesting a revert unless he wanted to start the process of asking the committee (or the developers by way of a general resolution) to do so. There are lots of problems with making that change at this point, so: "If you don't want that solution, please don't suggest it repeatedly: it's non-motivating to spend any further time."

Things went further off the rails on March 29, with a non-maintainer upload (NMU) for dpkg made by Bastian Blank that removed the warning and stopped installing dpkg-fsys-usrunmess. An NMU is something of a hostile act for a package with an active maintainer. As reported by Triplett, the NMU was quickly overridden by Jover with a comment: "This also clears a bullying NMU." As Triplett put it: "we're now even further into full-blown 'fights in the archive' territory".

In his most recent message (as of this writing), Grohne said that Boccassi was not being constructive and that he was not surprised Jover did not want to communicate with him about the problems. He reiterated some of the problems that have occurred with the feature, noting that the current problem was largely not technical:

It has a history of its proponents not fixing the resulting bugs, but deferring them to others and/or denying/downgrading them. I've definitely spent more than a week on fixing /usr-merge breakage excluding the time discussing it. It is not working fine at all. Possibly, it is fixable on a technical level, but it is totally broken socially. Please stop this unconstructive behaviour.

While this may sound single-sided, Guillem's behaviour wrt /usr-merge cannot be described as constructive either. Rest assured, that side of the picture is not being ignored. That should also be evident from dpkg.git at this time.

The dpkg Git repository does show a recent change by Jover that removes the offending warning, but only on Debian systems. The commit message could certainly be interpreted as non-constructive, as well, however. For example: "On Debian some people seem so offended by a (factual) warning, that the obvious recourse has been for them to bully and abuse." The communication between proponents of /usr merge and the maintainer of the critical package-manager for the distribution seems utterly broken at this point—probably irretrievably so.

It is not clear where things go from here. The technical committee has repeatedly made its feelings known on the path it wants to see the project take, but it cannot order anyone to do anything, exactly. Debian developers are volunteers and its packages are the personal fiefdoms of their maintainers. The Debian Constitution does allow the committee to override developer decisions—to not accept a patch, for example—but someone has to voluntarily do the work to bring the late-breaking dpkg patch up to snuff.

It seems like some project elder (in terms of experience, not necessarily age) could steer this kind of change through the rocks. While it would appear to be vanishingly unlikely that the committee would backtrack on its three earlier decisions, even though there are two new members since they were made, it is not impossible, at least in theory. The bugs that exist in dpkg with the aliased-directories version of /usr merge may simply live on as irritations for a time, since they do not really seem to rise to the level of a release-blocking bug, at least so far. We will all have to wait and see how it turns out in the next episode of "As the Debian Turns" ...