Unexpected fallout from /usr merge in Debian [LWN.net]

December 4, 2018

This article was contributed by Alexander E. Patrakov

Back in 2011, Harald Hoyer and Kay Sievers came up with a proposal for Fedora to merge much of the operating system into /usr; former top-level directories, /bin, /lib, and /sbin, would then become symbolic links pointing into the corresponding subdirectories of /usr. Left out of the merge would be things like configuration files in /etc, data in /var, and user home directories. This change was aimed at features like atomic upgrades and easy snapshots. The switch to a merged /usr was successful for Fedora 17; many other distributions (Arch, OpenSUSE, Mageia, just to name a few) have followed suit. More recently, Debian has been working toward a merged /usr, but it ran into some surprising problems that are unique to the distribution.

Debian and its derivatives are definitely late to the /usr merge party. Systems running Debian testing that were initially installed before June 2018 still have /bin, /sbin, and /lib as normal directories, not as symbolic links. The same applies to Ubuntu 18.10. But both Debian and Ubuntu want to make the switch to a merged /usr. Debian tried, but it hit something completely unexpected.

The Debian /usr merge history started in 2016, when Marco d'Itri got the usrmerge package into Debian unstable. This package contains a Perl script that converts an existing system into the state with a merged /usr. Also, a change was made to the debootstrap program (which installs a Debian system into a chroot), so that it could create the needed symbolic links by itself before installing any packages. The end result is the same in both cases.

The plan was to default to a merged /usr when it is ready (and the initial testing revealed only three broken packages), but it was hoped that both merged and non-merged setups would be supported. In other words, the expectation was that there would be no flag day when everyone must switch.

The decision on whether /usr merge would be done by default has changed multiple times during the debootstrap development timeline. The initial support was coded in September 2016, and released in debootstrap version 1.0.83, in a disabled-by-default state. In October 2016 there was an attempt to enable it by default, but this was reverted in November, because the dpkg-shlibdeps program (which is used during package builds to automatically generate dependencies on packages that provide the needed shared libraries) broke. Therefore, Debian 9.0 (with the code name "Stretch") was released in June 2017 without this feature.

A second attempt to re-enable /usr merge by default happened in June 2018, with debootstrap version 1.0.102. Since June 25, 2018, new default installations of Debian testing from the "daily" builds of the install CD have a merged /usr, and that has not been reverted so far.

Happy ending? Not so fast ...

Problems with R

On November 10, a new version of debootstrap was uploaded to the "backports" repository for the Debian stable distribution. This repository is not enabled on end-user systems by default, but it is enabled on the build daemon (buildd) machines for the purpose of getting a few updated packages including debootstrap. Therefore, shortly thereafter, Debian build daemons started using a merged /usr for their chroots. One of them attempted to automatically rebuild the r-base package and produced an R binary that did not work on non-merged systems. As the bug report says:

    /usr/bin/R: line 193: /usr/bin/sed: No such file or directory

Traditionally, sed was always placed in /bin. The Debian package sed also has /bin/sed, not /usr/bin/sed. In the bug report, the problem is treated like a one-off issue, to be solved by a rebuild. However, on the debian-devel mailing list, Ian Jackson quickly pointed out that the problem is, in fact, due to /usr merge on the build daemons. He suggested that the change should be reverted. Dirk Eddelbuettel seconded that suggestion, and noted that he expects "much more breakage to follow". Indeed, similar problems were triggered in sympow, pari, and monitoring-plugins. Other bugs of this nature can be found by searching the Debian bug tracking system for a special tag (but this search also finds other kinds of issues).

Jackson provided a good explanation of the mechanics of what has happened, quoted below.

R's autoconfery is autodetecting the location of (say) sed at build time by searching the PATH. R then bakes the discovered path into the built binaries.
With usrmerge, /bin is a symlink to /usr/bin [...].
Consequently the R autoconfery always detects (say) sed in the first place out of /usr/bin and /bin it looks, and bakes /usr/bin/sed into its binaries.
On a system without usrmerge, /usr/bin/sed does not exist, because sed is in /bin.

Matthias Klumpp suggested that a sensible build system would automatically detect the correct paths even on a /usr-merged system; he also wondered why the issue with incorrect paths has not been encountered by other distributions. The crucial difference was found to be that all other distributions had a flag day, before which merged /usr was unsupported, and after which it was mandatory. That is, on other distributions there was no requirement that a package built on a /usr-merged system should work correctly on a system without a merged /usr.

D'Itri said that there is only a handful of packages affected and provided an example of what a fix would look like. Namely, the proper locations of programs should be provided to the configure script or its equivalent, or the software has to be modified so that it always looks for programs using $PATH at runtime. In fact, Debian maintainer scripts (that is, scripts written by Debian maintainers that run before or after the package installation, upgrade, or removal) are already required, by Debian policy, to use $PATH instead of hard-coding the paths to the tools they use. There is, however, no such requirement in the policy for programs other than maintainer scripts.

The argument that there is only a handful of affected packages was not compelling enough, because the number of packages affected is, in fact, unknown without someone trying to rebuild them all. So Simon McVittie added a new check to the reproducible builds infrastructure, specifically for differences caused by /usr merge. This check caught problems in perl, quilt, and systemd.

On November 20, the change was reverted — build daemon chroots were recreated without a merged /usr.

Plan B?

Due to the perceived damage to the distribution caused by /usr merge, Adam Borowski started a new discussion thread on the debian-devel list, with the subject "usrmerge — plan B?". He proposed to scrap the usrmerge package, and, instead, move binaries and libraries to /usr one by one. "If it takes 10 years, so what?" McVittie pointed out that moving binaries on a package-by-package basis would not solve anything. Indeed, any move of a binary would break all packages that hard-coded its expected location, which is exactly the same problem that was triggered by /usr merge on the build daemons.

The arguments usually presented by /usr-merge proponents have also been questioned, for example on the basis that atomic upgrade is worthless if it leaves the system with mismatching /etc or /var. McVittie wrote a long email in response to the complaint about a lack of detailed justification of the change. The criticism regarding /var and /etc mismatch was answered by showing how existing systems with immutable /usr (like OSTree and systemd-based stateless systems) are dealing with it.

McVittie explained the /usr merge failure mode again and mentioned the one-way compatibility property: packages built on a system without a merged /usr would work fine on a merged-/usr system, but not the other way around. Therefore, he concluded that the transition simply happened, by accident, in the wrong order: the build daemons should have transitioned last.

The rest of the email presents the benefits of /usr merge. In particular, "it's a significant simplification for reliable special-purpose systems" such as consumer appliances. It brings benefits for end-user systems, too, because of increased portability of scripts from other systems. This is important because some upstream authors work on merged-/usr systems and don't know (and don't need to know) the "proper" location of binaries. In addition, it was noted that the change would simplify the security-relevant code of sandboxing systems like bubblewrap.

But Borowski's objection was not to /usr merge itself, but of the need to support systems both with and without a merged /usr. In other words, according to him, in order to be supportable, /usr merge must either be mandatory or not happen at all.

The discussion also touched upon the topic of the transition freeze that will happen in January 2019 and limits the time available for planning and testing any changes. So far, there is no consensus, but the most popular opinion seems to be that a mandatory /usr merge has to be postponed until after the release. As Russ Allbery put it: "That's going to be a disappointing delay for a lot of people, I'm sure, but it's still better for them than never doing this at all."

Ubuntu also plans to go through /usr merge. Dimitri John Ledkov described the situation this way: in the upcoming Ubuntu 19.04 release, new installations will use merged /usr, but existing ones will not have their /usr merged on upgrade. Of course, the official Ubuntu build daemons will not have merged /usr.

The status quo in Debian is that debootstrap (and, thus, the Debian installer) installs a system with /usr merged, and the build daemons explicitly pass a command-line switch to avoid this. This is still dangerous, because initial uploads of new packages in Debian (but not in Ubuntu) require a binary package built by the developer on his system. As long as a single uploader has a system with merged /usr, there is a potential for the bug to resurface in his Debian package. On the other hand, as d'Itri noted, there are already too many possibilities for a Debian maintainer to misbuild a package by not using a chroot, so there is nothing new in this danger. Anyway, support for marking packages as "tainted" by the build environment has been implemented in dpkg in response to this concern, and dpkg taints the package if it detects the symbolic links related to /usr merge.

The discussion is still in progress, though; no consensus has been reached. A bug was filed against debootstrap by Jackson to revert the change to merge by default for the next release of Debian. Due to the disagreement of the debootstrap maintainer to the proposed change, Jackson reassigned the bug to the Debian Technical Committee, which is the ultimate authority for resolving otherwise unresolvable technical disputes within Debian. There is also a request from the Debian backports FTP master that the default should be the same in Debian stable backports and in Debian testing. Emilio Pozuelo Monfort, a member of the release team, also spoke in favor of reverting to non-merged /usr in new installations.

It is impossible to predict now how the Technical Committee will rule. In the worst case for /usr-merge proponents, proper introduction of a merged /usr into Debian may be delayed by a few more years. But, if it votes for keeping the status quo, new end-user systems in the next stable release of Debian will have merged /usr, old but upgraded ones won't, and the build daemons will reliably build packages suitable for both cases, just like what's planned for Ubuntu 19.04. No flag day is needed in this scenario, so it would follow the best Debian traditions of not forcing transitions onto users.

Index entries for this article
GuestArticles	Patrakov, Alexander E.

to post comments

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 19:14 UTC (Tue) by rweikusat2 (subscriber, #117920) [Link] (6 responses)

"If it ain't broken, fix that!"

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 20:53 UTC (Tue) by quotemstr (subscriber, #45331) [Link] (5 responses)

It is sort of broken though. The distinction between / and /usr no longer has a useful purpose, and its continued presence subtly complicates everything. It doesn't seem all that bad because you've grown accustomed to the situation, but if you take a step back and look at it objectively, it's nonsense.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 21:05 UTC (Tue) by rweikusat2 (subscriber, #117920) [Link] (4 responses)

It subtly complicates using binaries with hardcoded paths on systems not installed/ configured by the person who wrote the code with the hardcoded paths. Hence, life of people who may want to chose their own FS layout for whatever reasons "must be made difficult".

Considering that you've already managed to come up with a boilerplate (and uncalled for) conjecture about "my hidden personality defects", there's obviously no point in discussing this.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 21:13 UTC (Tue) by quotemstr (subscriber, #45331) [Link] (3 responses)

I said nothing about any personality defect and meant the "you" as a collective "we all", as is idiomatic in English. This hostile interpretation of my post is totally uncalled for.

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 6:02 UTC (Thu) by k8to (guest, #15413) [Link] (2 responses)

This escalated quickly, but your comment doesn't read as referencing the general you at all. It can be read that way for sure, but I don't think most people would. Something to read back next time, perhaps.

The generic you is often part of a conjecture.

"Say you want ...."

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 15:59 UTC (Thu) by quotemstr (subscriber, #45331) [Link] (1 responses)

Even if one does read the "you" as specific and not general, there's nothing wrong with saying "X seems okay because it's familiar but if you take a step back, one can see that it's actually pretty bad". This kind of argument says nothing about one's "personality", much less "defects".

I maintain that the explosive reaction to my original post was uncharitable and hostile, and I have zero desire to interact with someone who's demonstrated that he'll adopt a creative interpretation of an innocious post and treat it, in public, as some kind of ridiculous offense.

Unexpected fallout from /usr merge in Debian

Posted Dec 7, 2018 2:17 UTC (Fri) by k8to (guest, #15413) [Link]

Not engaging on that axis.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 19:21 UTC (Tue) by mbiebl (subscriber, #41876) [Link] (33 responses)

That article is very opinionated and not what I expect from LWN

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 20:40 UTC (Tue) by clugstj (subscriber, #4020) [Link] (28 responses)

To me, an outsider, this article does not appear to contain a significant amount of opinion. Please elaborate on why you believe it to be "very opinionated".

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 22:48 UTC (Tue) by nix (subscriber, #2304) [Link] (27 responses)

Because it is obviously sane and sensible to support forever a filesystem layout implemented to allow Ken Thompson et al to install a pair of RK05 disk packs in 1971 (rather than the original one disk pack). This capacity limitation on Unix Installation #1 is *obviously* not a ludicrous historial wart in 2018 and anyone suggesting that eventual migration is a sensible idea is *obviously* a biased crazy person.

(Right?)

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 23:12 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (21 responses)

To be fair, /usr<->/bin split over time evolved into being able to host a small-ish recovery or initializer system without a /usr partition.

Then it evolved again, initializer systems were kinda made obsolete by initrd/initramfs and Busybox took over the task of the recovery system.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 23:57 UTC (Tue) by nix (subscriber, #2304) [Link] (20 responses)

Indeed so -- and a bunch of other evolution happened too, e.g. none of us have /usr/home any more. But nobody would ever have implemented the /usr split just for that: it was exapted for those other purposes, and now those too are obsolete, I think that bidding farewell is the right thing to do. (It's just annoying that compatibility concerns prevent us from doing the /usr -> / thing the Hurd did. We still *have* /usr in paths everywhere, and it will in time become pure noise.)

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 0:02 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> It's just annoying that compatibility concerns prevent us from doing the /usr -> / thing the Hurd did. We still *have* /usr in paths everywhere, and it will in time become pure noise.
No. /usr serves a clear purpose - it indicates that this part of the filesystem contains immutable executable code and its supporting data. It makes it easy to reason about it and do atomic operations on it.

Unexpected fallout from /usr merge in Debian

Posted Dec 11, 2018 15:56 UTC (Tue) by nix (subscriber, #2304) [Link]

Disproof: /usr/src. :)

(Immutable executable code, really? I know people with /usr/backup, /usr/finance, /usr/god-only-knows-what-stuff. Sure, it's not in the FHS, but /usr/src *is*, and is neither immutable nor executable.)

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 8:31 UTC (Wed) by epa (subscriber, #39769) [Link] (16 responses)

Wasn’t /usr the original place for individual users’ home directories — hence the name?

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 9:52 UTC (Wed) by mgedmin (guest, #34497) [Link] (1 responses)

Yes. And when the original Unix people ran out of disk space in /bin, they created a user called 'bin' and put some binaries in its home directory, /usr/bin.

(There's still a 'bin' user in my /etc/passwd on my Ubuntu 18.10, but its home directory is just /bin now, and it doesn't own any files.)

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 6:20 UTC (Thu) by draco (subscriber, #1792) [Link]

Thank you. I've long wondered what the bin user was for...

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 17:53 UTC (Wed) by jezuch (subscriber, #52988) [Link] (12 responses)

From what I heard it comes from Unix System Resources. Not sure if it's more or less trustworthy than the "user -> usr" one (though I don't really buy that one).

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 19:20 UTC (Wed) by mpr22 (subscriber, #60784) [Link] (2 responses)

user -> usr sms prfctly plsbl frm th ppl wh spld 'remove' as 'rm'.

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 14:39 UTC (Thu) by andrewsh (subscriber, #71043) [Link] (1 responses)

And create as creat.

Unexpected fallout from /usr merge in Debian

Posted Dec 10, 2018 14:52 UTC (Mon) by Thomas (subscriber, #39963) [Link]

And unmount as umount.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 19:52 UTC (Wed) by karkhaz (subscriber, #99844) [Link]

Here's an image from the Wikipedia article on Ken Thompson:

https://en.wikipedia.org/wiki/Ken_Thompson#/media/File:Ve...

It's Version 6 Unix running on an emulator; it shows a directory called /usr/ken.

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 9:35 UTC (Thu) by epa (subscriber, #39769) [Link] (7 responses)

An unwieldy acronym like "Unix System Resources" doesn't ring true to me. Not from the people who came up with the name 'Unix' as a pun on 'Multics', and gave short, pithy names to everything else. (Apart from the kernel, does any file on a Unix system have the word 'unix' in its name?) Only large organizations can come up with cumbersome names like "Program Files".

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 10:12 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

“Program Files” name was intentional, to force every program to care about spaces in file names.

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 13:29 UTC (Thu) by pbonzini (subscriber, #60935) [Link]

Interesting. Some localized versions not have the space though.

Unexpected fallout from /usr merge in Debian

Posted Dec 7, 2018 10:17 UTC (Fri) by k8to (guest, #15413) [Link] (3 responses)

Sad that adding the word "files" offers no value to the user. What else would be in there?

I suppose directories. Though that isn't in the name. So I guess it's just wrong.

Unexpected fallout from /usr merge in Debian

Posted Dec 7, 2018 10:43 UTC (Fri) by excors (subscriber, #95769) [Link] (2 responses)

See https://blogs.msdn.microsoft.com/oldnewthing/20131119-00/...

> Program Files are not the same as Programs. Programs are things like Calc, Notepad, Excel, Photoshop. They are things you run.
>
> Program Files are things like ACRORD32.DLL and TWCUTCHR.DLL. They are files that make programs run.
>
> If the directory were named Programs, then people who wanted to run a program would start digging into that directory and seeing a page full of weird DLL names and wonder "What the heck kind of programs are these?" And eventually they might figure out that if they want to run PowerPoint, they need to double-click on the icon named POWERPNT. "Computers are so hard to use."

Unexpected fallout from /usr merge in Debian

Posted Dec 7, 2018 21:34 UTC (Fri) by epa (subscriber, #39769) [Link] (1 responses)

Which is why application directories as used by NextStep (later Mac OS) and RISC OS (inspiring Rox Desktop) are such a good idea. It is strange that Microsoft did not adopt them for Windows 95. Too late now.

Unexpected fallout from /usr merge in Debian

Posted Dec 10, 2018 14:53 UTC (Mon) by cortana (subscriber, #24596) [Link]

I don't think anything in Windows 95 prevented XCOPY deployment. It's just that by then the Windows world was used to the insanity of running an installer that could do _anything_ to your system...

Unexpected fallout from /usr merge in Debian

Posted Dec 11, 2018 16:01 UTC (Tue) by nix (subscriber, #2304) [Link]

Quite. It's not like Unix has the term 'system resources' anyway. Heck it doesn't even use the term 'system' to mean anything like that, not if the system() libc function is any indication. (It's a rather strange choice of name that shows the historically close connections between the shell and system calls. They really were meant to be the same sort of thing at different layers.)

As for 'resources', Unix never uses that term at all. *Windows* calls things 'resources', but Unix? The most you get is the relatively recent /usr/share, which emphasises the shareability of data, not whether or not it is a 'resource', whatever sort of thing that's supposed to be.

If /usr really did mean 'Unix system resources', it would contain the shell, the kernel source, libc, and probably not a lot else. :)

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 1:47 UTC (Thu) by djbon2112 (guest, #101854) [Link]

According to a common history, yes. /usr originally had homedirs on disk #2, then applications that wouldn't fit in /bin got put in /usr/bin, then eventually the homedirs themselves moved to /home leaving the "user" programs in /usr. It makes sense in an abstract way - /bin is for OS-critical binaries, while /usr/bin are "things the user installs", with 45+ years of growth on top of it. At least that's how I've always thought of it.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 12:03 UTC (Wed) by smcv (subscriber, #53363) [Link]

> It's just annoying that compatibility concerns prevent us from doing the /usr -> / thing the Hurd did.

I think this was briefly considered for Fedora, but it's actually less good than the /usr merge. In the layout the Hurd tried, the static-data directories (/bin, /sbin, /lib* and /share) are siblings of /etc (which must be on the root filesystem because it's where /etc/fstab is located). However, the static-data directories are more similar than they are different (for instance they can be mounted read-only during normal operation, if you have a strategy for making them briefly read/write during upgrades), so it makes sense to group them together.

The name /usr for that group is historical accident, sure, but the fact that they're a group does seem good.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 14:05 UTC (Wed) by nivedita76 (subscriber, #121790) [Link]

What does that have to do with the article being opinionated? It seems pretty factual, and I don't see it expressing anywhere any opinion as to whether this merge is undesirable?

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 22:41 UTC (Wed) by wahern (subscriber, #37304) [Link] (1 responses)

A few releases ago OpenBSD made /var/tmp a symlink to /tmp. Like many people (and per OpenBSD's recommendations) I put /var and /tmp on different partitions or devices. Among other things, /tmp can be cleaned on startup by simply initializing the superblock or encrypting with an ephemeral key. But I also generally make /tmp rather small. /var/tmp was useful for the rare times where I needed alot of space for temporary files, such as when making backups. Making /var/tmp a symlink to /tmp has consistently been annoying.

I'm not taking sides, either wrt /bin -> /usr/bin or even /var/tmp -> /tmp, but there are good reasons for opposing those changes. The original motivation or design of something is not the sole determiner of its utility. It doesn't matter if a benefit is by accident or path dependency.

Unexpected fallout from /usr merge in Debian

Posted Dec 11, 2018 16:16 UTC (Tue) by nix (subscriber, #2304) [Link]

I too had a very split system once, /var, /usr, /usr/src, /var/spool/news, /home, /, all different filesystems. But while this did save my bacon in a certain massive filesystem corruption incident (that made its way to LWN :) ). I argued in favour of a mega-split layout here more than once. The eventual problem there is that the tools to manipulate filesystems are... cruder than the ones to manipulate the fs hierarchy, and in the end half these filesystems end up mis-sized and fixing it is a right drag and they're often in half a dozen noncontiguous fragments on rotating storage and LVM is not terribly good at defragmenting them (you have to do it by hand via repeated manual pvmoves).

In my most recent setups I've just been using one huge XFS filesystem where possible, perhaps with others hived off where encryption with different keys requires it, but otherwise just One Big FS. XFS slices the fs into ags anyway, so we don't see a performance loss from the fs putting things hugely far apart from each other: the only real risk is fs corruption causing the fs to not mount, and the fix there is a proper initramfs or a rescue disk, and good backups (of course I have *both*, a rescue partition rsynced at intervals from selected parts of the main system *and* an initramfs that recognizes boot failure and boots off it if need be).

You do need protection against disk corruption, but with modern titanic disk sizes the right approach is probably to store the core system more than once, so you can use *it* as a recovery tool, rather than slicing everything off the core system into smaller pieces so you only lose one bit to corruption at once. (And often corruption comes in the form of the whole disk dying or a bug somewhere roasting *all* your writeable filesystems, not just one. So it doesn't win you as much as it might, and its costs are substantial.)

As with all engineering, it's a tradeoff, and often the right answer is the simplest ("a few big filesystems and forget all about it, dammit").

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 6:04 UTC (Thu) by k8to (guest, #15413) [Link] (1 responses)

Confused. That doesn't seem to be in the article at all (either direction).

Unexpected fallout from /usr merge in Debian

Posted Dec 11, 2018 16:10 UTC (Tue) by nix (subscriber, #2304) [Link]

It's not in the article -- it's implied by the grandparent (well, great-great-grandparent now) comment's suggestion that this article is 'opinionated', thus that there is something sensible and natural and right about this split between /usr and / and that attempting to clean the mess up, or even *writing an article about it*, is somehow wrong.

The *article* is excellent. :)

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 8:11 UTC (Wed) by edeloget (subscriber, #88392) [Link]

> That article is very opinionated and not what I expect from LWN

I read it twice (once before I read your comment, and another time after) and I can't see how this article is opinionated. Care to explain?

Unexpected fallout from /usr merge in Debian

Posted Dec 6, 2018 15:59 UTC (Thu) by jschrod (subscriber, #1646) [Link] (2 responses)

1) Would you please explain why you think that the article is "very opinionated"?

2) Would you please disclose if you are involved in that discussion, and, if yes, what your viewpoint is? Maybe we can then better determine the background of your argument.

FTR: I'm not involved in that discussion. I have lived with separate /bin and /usr/bin for several decades. (I still remember the time when my home directory was *directly in* /usr/ and not in some new-fangled-thing named /home or /usr/home. Oh yes, and I have a grey beard.) I also use some systems with merged /usr since a few years, doesn't make a difference.

Unexpected fallout from /usr merge in Debian

Posted Dec 8, 2018 5:13 UTC (Sat) by dirtyepic (guest, #30178) [Link] (1 responses)

What you are witnessing is what happens when someone who can't possibly be mistaken encounters something that doesn't support their personal world view. Facts become opinions, objective truth become subjective interpretation, and neutral retelling of events becomes biased slander intended to further some agenda. It's not unusual. We all do it, in some way or another. Some people just do it on the internet.

Unexpected fallout from /usr merge in Debian

Posted Dec 14, 2018 0:39 UTC (Fri) by jschrod (subscriber, #1646) [Link]

For sure, but: I experienced personally the introduction of the account "bin" -- introduced to hold binaries that could not be stored in /bin anymore because root disk space ran out. As standard at this time, the home directory of bin was /usr/bin.

/usr/lib came later, btw -- we didn't have shared libraries yet. I don't remember exactly, but maybe that's why we don't have a pointless "lib" account on our Linux systems, while "bin" still is there, without any rationale.

When I read nowadays that /bin vs. /usr/bin was a great architectural decision that should not be removed because it ain't the Unix Way(tm) -- well, I scratch my grey beard and try to avoid the worshipping discussion of the "Unix fathers". I had occasion to have several dinners with Dennis (Ritchie) and Ken (Thompson), and from that personal experience I'm sure that they would laugh at the thought that this might have been a technical decision and not some technical coincidence.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 19:50 UTC (Tue) by ballombe (subscriber, #9523) [Link] (4 responses)

(first thanks for linking to my package! much appreciated!)

There is one common use case where you have to use the full path:
#! path.
It is not uncommon to do
#! @PATH@
and have configure put the right absolute path.

Of course you can use env (even though it is discouraged by Debian policy)
but that does not help because env itself is affected.
Also usin genv prevnt you to pass an option since the kernel allows a single option.

> The Debian /usr merge history started in 2016
Actually it started in 2002 in Debian GNU/Hurd (see debian-devel) with a /usr ->/ symlink

What is missing from the article is what problem usrmerge is trying to solve.

Debian never followed the lib32/lib64 trend and it was for the best.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 20:41 UTC (Tue) by wahern (subscriber, #37304) [Link]

env should always be at /usr/bin/env. I think even Solaris and AIX have it there. (Can't confirm at the moment, but IIRC one of them was once an outlier but no longer.)

The other problem you mentioned, shebang word splitting semantics, remains.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 7:13 UTC (Wed) by eru (subscriber, #2753) [Link] (1 responses)

Also usin genv prevnt you to pass an option since the kernel allows a single option.

I have wondered about this. With scripting languages, passing more options on a #! line would sometimes be useful. Is there a technical reason, or just tradition? If an unlimited argument count is not feasible (and is probably not needed), even raising the limit to 10 arguments or so would help.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 16:50 UTC (Wed) by Liskni_si (subscriber, #91943) [Link]

env in FreeBSD and macOS has a -S option that makes passing additional options possible (https://ss64.com/osx/env.html, https://www.freebsd.org/cgi/man.cgi?env), and coreutils' env has it since July's release as well (https://github.com/coreutils/coreutils/commit/668306ed86c...).

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 13:55 UTC (Wed) by drag (guest, #31333) [Link]

> Of course you can use env (even though it is discouraged by Debian policy)

In a ideal world env should NOT be used for any OS-level software. So OS package managers, system configuration utilities, desktop management and display software, etc. However for applications and services that run on top of the OS then env should be used.

The idea being that a user can set up their own environments how they like with their own custom versions of python, ruby, etc and libraries and custom software for development purposes (or whatever) and have the software they use be able to take advantage of that. But still have a functional OS and desktop that doesn't break.

Unfortunately in Linux-land there is often very little distinction made between what is 'OS' and what is 'Applications'. They tend to get released all together as one big 'mush' of interconnected dependencies. So unfortunately there is no right way to do anything without resorting to sandboxing.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 20:51 UTC (Tue) by juliank (guest, #45896) [Link] (3 responses)

There would be no issue with NEW if we discarded binaries from NEW and just accepted the source.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 10:30 UTC (Wed) by smurf (subscriber, #17840) [Link] (1 responses)

Wasn't the plan to require binaries to be uploaded, just to make sure hat they're buildable in principle, but then to throw them away and re-build on all architectures?

Unexpected fallout from /usr merge in Debian

Posted Dec 7, 2018 2:28 UTC (Fri) by spwhitton (subscriber, #71678) [Link]

Also to enable their inspection by those reviewing the NEW queue, since the source couldn't be sent out to the buildd network until it's been reviewed.

Unexpected fallout from /usr merge in Debian

Posted Dec 7, 2018 2:29 UTC (Fri) by spwhitton (subscriber, #71678) [Link]

AIUI this is in the works.

Add symlinks in /usr

Posted Dec 4, 2018 21:13 UTC (Tue) by epa (subscriber, #39769) [Link] (8 responses)

Why not create /usr/bin/sed -> /bin/sed on existing systems? Once the current installed base of Debian systems has the necessary symlinks, R and other packages which hardcore paths (bad behaviour IMHO, but there you are) will still work when built with merged /usr and run on the old layout. The symlinks disappear when the merge happens, of course.

Add symlinks in /usr

Posted Dec 4, 2018 22:43 UTC (Tue) by patrakov (subscriber, #97174) [Link] (7 responses)

Won't work for the case when a package finds some binary via the /bin -> /usr/bin symlink (i.e. the opposite case to what was described in the article). See e.g. https://bugs.debian.org/914933 - the wrong path discussed there is /sbin/fdformat, but fdformat, officially, lives in /usr/sbin.

Anyway, as demonstrated by the team behind reproducible builds, there are only 80 or so packages to fix.

Add symlinks in /usr

Posted Dec 4, 2018 22:50 UTC (Tue) by patrakov (subscriber, #97174) [Link]

I take the last line back. That estimate does not take into account that not the whole archive has been rebuilt. The new estimate is that 0.3% of all packages are affected.

Add symlinks in /usr

Posted Dec 5, 2018 9:14 UTC (Wed) by epa (subscriber, #39769) [Link] (5 responses)

I don't understand how that would be a problem. If /usr contains symlinks to things in /bin, then either /bin/foo or /usr/bin/foo will work without problems. And once you get merged /usr, then either will still continue to work (since /bin is presumably a symlink to /usr/bin). Obviously you do the same for /sbin and /usr/sbin, so the fdformat bug you linked to would also be resolved.

I do agree with the general goal of fixing these packages that have their own strange ideas about PATH, but it needn't stand in the way of getting /usr merge done, because the packages can be made to work with symlinks both before and after the transition. The important part is to make sure the symlinks from /usr/bin/x to /bin/x exist on deployed systems before any rebuilt packages are pushed out. But while running the usrmerge script is quite a big change, creating a few symlinks is not.

Add symlinks in /usr

Posted Dec 5, 2018 12:09 UTC (Wed) by smcv (subscriber, #53363) [Link] (1 responses)

> I don't understand how that would be a problem. If /usr contains symlinks to things in /bin, then either /bin/foo or /usr/bin/foo will work without problems.

The problem case that you're missing here is partial upgrades. If (for example) ping versions < X have only /bin/ping, ping version X adds a /usr/bin/ping symlink, and gnome-nettool version Y was built with a ping version >= X installed and hard-codes /usr/bin/ping into its binaries, there is no mechanism for gnome-nettool version Y to get an automatic dependency on ping (>= X). A user could install gnome-nettool version Y with ping version < X, and it wouldn't find /usr/bin/ping.

Add symlinks in /usr

Posted Dec 6, 2018 7:59 UTC (Thu) by epa (subscriber, #39769) [Link]

I didn’t mean that the symlink would be owned by the ping package itself, but that it and the others would be created by a script run as a one-off. Like the usermerge script which all systems are going to run at some point in the future, but making the symlinks is a much simpler first step.

Add symlinks in /usr

Posted Dec 5, 2018 14:14 UTC (Wed) by nivedita76 (subscriber, #121790) [Link] (2 responses)

See the bug report. The issue is that because /sbin was symlinked to /usr/sbin, and the packages configure script is even more wonky, it located /sbin/fdformat and hard-coded that (using normal $PATH conventions, it should have found /usr/sbin/fdformat before /sbin/format). On an unmerged system, only /usr/sbin/fdformat exists. So you'd have to change unmerged systems to symlink both ways, at which point you might as well just merge them.

Add symlinks in /usr

Posted Dec 6, 2018 6:28 UTC (Thu) by lathiat (subscriber, #18567) [Link]

I had to re-read the article twice to get this point (that unmerged systems only have 1 working path), it would be a good clarification to add to the article.

Add symlinks in /usr

Posted Dec 6, 2018 7:57 UTC (Thu) by epa (subscriber, #39769) [Link]

Yes, I understand that. Which is why I suggested adding the symlinks on deployed systems as a first step — then packages like this will just work, whatever path they have baked in.

“At which point you might as well just merge them.”

Well, probably. But as something to do to all installed systems, creating symlinks seems like a much safer option as a first step, not requiring any reboot, and easy to undo if necessary. Then the much more elaborate operation of doing the merge could happen at a slower pace, with no flag day.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 21:21 UTC (Tue) by jezuch (subscriber, #52988) [Link] (3 responses)

> Therefore, he concluded that the transition simply happened, by accident, in the wrong order: the build daemons should have transitioned last.

Well, without a forced upgrade this will never happen. I installed my Debian in 2003 and had to reinstall just once: to switch from x86 to amd64 (this was before multiarch; with multiarch this could probably be done without a reinstall). (I may also have had one situation when I broke my system beyond recovery, but I don't even remember this so it may have not happened at all ;) ). My point is that there are plenty of users like me who regularly upgrade and have the new and shiny Debian without ever reinstalling. If there is no forced switch to merged /usr, these systems will remain forever with unmerged /usr.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 21:36 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

There’s already a script that does migration on an installed system.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 17:57 UTC (Wed) by jezuch (subscriber, #52988) [Link] (1 responses)

Are the users asked to install it? Is it enforced by some kind of Recommends: dependency?

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 22:17 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

No and no. But there's already a path for migration. Probably not for this release though.

Unexpected fallout from /usr merge in Debian

Posted Dec 4, 2018 23:56 UTC (Tue) by dkg (subscriber, #55359) [Link] (9 responses)

The problem here is baked-in paths in binaries that should know better. those binaries should generally rely on $PATH at runtime anyway, so i'm glad we're discovering this and getting it fixed, rather than punting it to a flag day. I ran into this with GnuPG recently, and ended up patching out this misbehavior in debian once it became clear that upstream wasn't going to fix it, due to abandoning the tool that was doing it.

(fwiw, i'm a proponent of the usrmerge, and would be happy if it happened sooner rather than later, but i still dislike having unrelated paths baked into the binaries)

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 0:47 UTC (Wed) by wahern (subscriber, #37304) [Link] (4 responses)

The utility discovered from $PATH isn't always the utility you expect. On Linux systems there's typically one tar program. On BSDs there's often tar and gtar, with most options beyond xvzf drastically different. On Solaris the POSIX-compliant utilities are *rarely* the first ones in $PATH.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 4:54 UTC (Wed) by dkg (subscriber, #55359) [Link] (3 responses)

Sure, but relying on $PATH at build time to learn the location of a tool (e.g. the use of AC_PATH_PROG during ./configure), and then expecting it to match at runtime also isn't always likely to yield the utility you expect, which is exactly the class of bugs we're uncovering here.

If a package needs a specific version of a utility, it needs to do one of three things:

rely on a well-documented, reliable interface that someone else will maintain, like a C library SONAME, or
ship the utility itself and point to it internally, or
test for the necessary functionality at runtime

Debian is showing that, given the possibility (but not hard requirement) of a /usrmerge, AC_PATH_PROG doesn't meet any of those standards.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 22:23 UTC (Wed) by wahern (subscriber, #37304) [Link]

It comes down to not letting perfect be the enemy of good. On the surface hardcoding the path seems like a bad idea, but when you're actually in a situation where you need to solve this issue and you consider the cost of those other options then it quickly becomes the least bad option.

Asking a C program (or any program) to implement autoconf-style utility detection is like begging for CVEs. Expecting people to use a non-existent libsed instead sed is... unreasonable. Shipping a wrapper of the utility is better, but then /usr/bin or [preferably] /usr/libexec would be littered with redundant wrappers.

The cost+benefit just isn't there for a "better" solution. The current solutions were manifestly good enough. It asks too much to expect packages to foresee *and* accommodate something like a wholesale shift of utilities from /bin to /usr/bin in addition to the myriad other issues they must contend with, and especially in light of the historic stability of core utility paths. For non-shell programs, the decision to rely on shell utility at runtime is already predicated on expediency.

Regarding the issue with relying on $PATH--it's far more reasonable to rely on a sane $PATH at compile-time than runtime. The reasons are self-evident.

FWIW, Ubuntu's r-base package's configure.ac uses `AC_PATH_PROGS(SED, sed, /bin/sed, [/usr/xpg4/bin:$PATH])`. We could nit-pick that endlessly, but why bother.

Unexpected fallout from /usr merge in Debian

Posted Dec 7, 2018 5:45 UTC (Fri) by zblaxell (subscriber, #26385) [Link] (1 responses)

> Sure, but relying on $PATH at build time to learn the location of a tool (e.g. the use of AC_PATH_PROG during ./configure), and then expecting it to match at runtime also isn't always likely to yield the utility you expect

autoconf (or its immediate predecessors) were designed for in-place self-hosted builds, where the runtime environment is fairly similar (if not entirely identical) to the build-time environment, and software is delivered to the end user as source code.

configure scripts can not only figure out where sed is, they can also test all the seds they can find, pick the one that has the right mix of extensions and bugs, and burn its complete path into the binary so the Wrong Sed will never be accidentally run.

On the other hand, that assumption is ~25 years old, give or take a decade...which is probably why no modern build systems look anything like autoconf.

Unexpected fallout from /usr merge in Debian

Posted Dec 7, 2018 8:59 UTC (Fri) by wahern (subscriber, #37304) [Link]

> which is probably why no modern build systems look anything like autoconf.

Rather than hardcoding paths to shared resources all the modern build systems (Docker, Go, pipenv, FlatPak, etc) literally duplicate multi-megabyte environments. They're increasingly hardcoding snapshots of the entire OS.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 1:25 UTC (Wed) by KaiRo (subscriber, #1987) [Link] (3 responses)

Juat that relying on $PATH is prone to security issues if anyone can manage to influence an executing user's environment.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 2:05 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

If an attacker has control over the environment then it's game over. There's also juicy LD_PRELOAD and LD_LIBRARY_PATH.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 14:21 UTC (Wed) by nivedita76 (subscriber, #121790) [Link] (1 responses)

That's why suid programs don't respect the LD_ environment variables. They need to avoid dependencies on PATH as well.

Unexpected fallout from /usr merge in Debian

Posted Dec 7, 2018 2:37 UTC (Fri) by k8to (guest, #15413) [Link]

Agreed, though the class of concern isn't entirely limited to suid. Any priveledged process or even a process running as another users which runs things via PATH could be potentialy subverted.

Of course compiling in paths and using fixed strings on exec as opposed to execvp etc aren't the same, but "just use the PATH' is often not the right approach from a defense in depth perspective.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 3:08 UTC (Wed) by flussence (guest, #85566) [Link] (4 responses)

The / <-> /usr split should go away, even though it will be unavoidably painful. But I think rather than throwing it away entirely, this should've been used as an opportunity to move to a better layout.

Wouldn't it be nice if there'd been separate /usr/*-pc-linux-gnu-c++0x and -c++1x directories instead of a flag day where everything had to be rebuilt/reinstalled in-place when GCC 5 came out? If CPU-dependent code was placed in separate directories, /usr/lib and /usr/bin could be kept arch-neutral.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 4:16 UTC (Wed) by sbaugh (guest, #103291) [Link]

If we're talking about taking this as an opportunity to make more radical changes to the filesystem layout, we'd be better off moving to a layout like GoboLinux or (my personal favorite) Nix/Guix, and solving all these problems at once.

The problem that Debian has run into (a resolved symlink being baked into the built binaries, resulting in the binaries not being portable to systems where that symlink points to something else or indeed doesn't exist at all) is a well known and long-solved problem from the perspective of Nix/Guix. Nix even refuses to work and requires overriding if the system is set up with the top-level package directory as a symlink - a bind mount is recommended instead. (Perhaps Debian could use the same solution?)

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 9:39 UTC (Wed) by matthias (subscriber, #94967) [Link] (2 responses)

> Wouldn't it be nice if there'd been separate /usr/*-pc-linux-gnu-c++0x and -c++1x directories instead of a flag day where everything had to be rebuilt/reinstalled in-place when GCC 5 came out? If CPU-dependent code was placed in separate directories, /usr/lib and /usr/bin could be kept arch-neutral.

Is there a chance that this would have worked without pain? How is it with programs pulling libs from both directories? These libs could then pull in the same lib from both directories and probably even try to pass around pointers across the barrier. This might be bad design, but if we would eliminate all software that uses some bad practices, there would not be a need for these directories anyway, as I would assume they would be empty ;)

Maybe you could build every library in two versions before starting to migrate applications. But this would also be a pain, as the build processes are not designed to build every library twice.

Unexpected fallout from /usr merge in Debian

Posted Dec 8, 2018 3:36 UTC (Sat) by wahern (subscriber, #37304) [Link] (1 responses)

I don't think that's possible without support from both the compilers and the system's run-time linker (e.g. ld-linux.so). And I don't think that support currently exists, or at least isn't sufficient to get the job done.[1] Probably not terribly difficult. Compilers already emit all manner of annotations, and both compilers and linkers (at least glibc's implementation) already have multiarch support. But that sort of toolchain work is the kind of thing that elicits very strong opinions. It might be why it doesn't already exist---nobody wanted to deal with the final gauntlet of coordinating and resolving the policy details.

As for mixing ABIs, the obvious solution is to mimic the existing multiarch support and build everything twice. Projects and packages can't normally build for different arches simultaneously, but packaging systems don't expect them to. It's why RPM and Debian packages do in-tree builds by default instead of out-of-tree builds: even most automake projects are subtly broken in that regard. It's a lost cause and not worthy of consideration.

[1] I mean, you can obviously do it by explicitly setting rpaths during compilation. But to do it in a manner consistent with default search paths rules and the application of global policy (i.e. ldconfg), I'd be surprised if there weren't more footwork required.

Unexpected fallout from /usr merge in Debian

Posted Dec 8, 2018 6:47 UTC (Sat) by matthias (subscriber, #94967) [Link]

Yes, of course. I thought of the problem of pulling in different so-names of the same library. But here, this should already fail at compilation time in any sane build environment.

OK. Then this would not require a flag day for switching from gcc4 to gcc5, but instead a flag day from when on we expect every library to exist twice. Or we should convert libraries for double build one by one according to the dependency graph (a tree would be to simple), which would probably take ages.

On Gentoo, I really liked the flag day approach. And it was more or less a day to recompile everything. But on Gentoo, every user could decide on his own when this flag day should happen. And after this day, everything was done.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 6:18 UTC (Wed) by abo (subscriber, #77288) [Link] (2 responses)

"initial uploads of new packages in Debian (but not in Ubuntu) require a binary package built by the developer on his system"

This seems dangerous to me. Why even allow uploads of binaries?

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 8:42 UTC (Wed) by laarmen (subscriber, #63948) [Link]

IIRC one of the results of this policy is that you were relatively sure that the package uploaded is buildable, at least on one architecture.

Of course, this reasoning is somewhat flawed since the maintainer's build environment might not be clean, and I don't think there's any proof that the .deb uploaded has been built with the accompanying source package. However, one other factor to consider is that when the policy was implemented, I'm guessing the Debian project had a lot less infrastructure available to build a package, thus this might also have been a sort of optimization.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 9:05 UTC (Wed) by plugwash (subscriber, #29694) [Link]

As usual the reason binary uploads are considered "normal"* in Debian comes down to history. The way Debian does things is probably not the way you would do things if you were designing a distro today from a clean sheet but changing stuff takes time and effort, especially when it cuts across multiple teams.

In the early days of Debian all binaries were built and uploaded manually alongside the source. When m68k support was added porters would build m68k binaries from the existing source and upload them. Gradually as Debian got bigger this process of building binaries for additional ports became automated but still the archive Demanded at least one binary with each source package and there was no mechanism for autobuilding architecture dependent packages.

Another quirk of Debian is that uploads of new packages (including adding of extra binary packages to existing source packages) go through extra checks by the ftpmasters, some of this checking is manual some is automated and some of it involves the binary packages.

The result if this is that while in 2014 the ftpmasters allowed source-only regular uploads and in 2015 the autobuilder system added support for building architecture independent packages**, binaries are still required to be part of the upload for "new" packages.

One option would be to make the archive discard the binaries from source+binary uploads unless a special flag was set (or discard them unconditionally and require two seperate uploads) but there are complications with doing that. For example the archive currently only allows each changes file name to be used once, but the normal changes files names for a source+binary upload and a binary only upload are the same, so if the archive discards the maintainer binaries for an architecture and then the autobuilder builds them the upload for the autobuilder is likely to be rejected.

None of this stuff is unfixable, but it takes time and effort to firstly persuade people that a solution is the correct one and then actually implement it.

* In ubuntu binary uploads that do not come from the autobuilders are considered highly exceptional and can only be performed by a handful of people, but they are occasionally necessary. Consider for example compilers that require themselves to build.

** architecture independent packages are always autobuilt on amd64, there are a handful of architecture independent packages that can only be built on a subset of architectures that does not include amd64 and so cannot be autobuilt.

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 9:09 UTC (Wed) by plugwash (subscriber, #29694) [Link]

> "This is still dangerous"

And it would still be dangerous even if Debian did not allow binary uploads because not all packages used on Debian systems come from Debian. Is it really acceptable for a binary package built on a freshly installed regular user system to fail on a regular user system of the same release that resulted from an upgrade?

Unexpected fallout from /usr merge in Debian

Posted Dec 5, 2018 17:23 UTC (Wed) by dbnichol (subscriber, #39622) [Link]

At Endless (which is Debian based), we wanted to have merged /usr to support atomic upgrades with OSTree where /usr is mounted read only. We hit exactly the issues described here. There were 3 things I ended up doing to avoid them. Importantly, we went with the flag day approach where all of our systems would use a merged /usr. That was possible since OSTree replaces the entire root filesystem on upgrades. Developers often manage their systems with dpkg (what we refer to as a converted system), but there we were willing to have breakage.

1. Some packages install to /bin and then add a symlink in /usr/bin (or vice versa) to support both paths. That would break in weird ways when installing on merged /usr systems. I just hacked all these packages (there were maybe 3 of the subset of Debian we use) to not create the symlink.

2. dpkg-shlibdeps would think a library wasn't in the expected location when /lib was a symlink. My approach (since we had no intention of supporting old lib64 style multiarch) was to just remove the relevant code - https://github.com/endlessm/dpkg/pull/2/commits/e08c3c1c0....

3. The package builder always uses non-merged /usr so that installs work correctly in either case. I didn't want to run into the case where package files suddenly started moving around on upgrades based only on the builder. The downside is that if you ask dpkg about a file it may not be the canonical path and dpkg will says it's unhandled. E.g., "dpkg -S /bin/sed" works, but "dpkg -S /usr/bin/sed" does not even though the latter is the real file on the system.

Not even Fedora has done this completely

Posted Dec 5, 2018 20:52 UTC (Wed) by jccleaver (guest, #127418) [Link]

Although not identical, there's a similar problem on the RedHat side with RPM dependencies hitting symlinks (or not) where the binary was usually expected to be in a different place.

The solution? Hard-code the historical locations (see below): https://fedoraproject.org/wiki/Packaging:Guidelines#Effect_of_the_UsrMove_Fedora_Feature

So even though it fixes the problem for all the atomic folks, paths never really went away and probably never really will. Surprised Debian hadn't prepared for this if they had decided they did eventually want to proceed after all this time.

[RPM] file dependencies don't work according to what's on the filesystem, they work according to the path specified in the rpm %files section. So an rpm which specified:
%files
/bin/sh
would be able to satisfy a file dependency for /bin/sh but not for /usr/bin/sh. As a packager you may need to pay attention to where other packages expect to find your files. Things that history has placed into /bin, /sbin, /lib, or /lib64 should be listed in the %files section as being in those directories. Things that history placed in /usr/bin, /usr/sbin, etc, should be listed in the %files section as being in %{_bindir}, %{_sbindir}, etc. If you feel that there is some historical confusion as to which directory a program is placed in, you can use a Virtual Provides to list the alternate path.

Simple fix?

Posted Dec 13, 2018 15:23 UTC (Thu) by mightypenguin (guest, #129211) [Link] (4 responses)

Would it work to make /usr/bin a symlink to /bin on unconverted systems?

Simple fix?

Posted Dec 14, 2018 10:41 UTC (Fri) by mattheww (guest, #108205) [Link] (3 responses)

On an unconverted system, both /bin and /usr/bin are real directories containing many binaries (that's what being unconverted means). So /usr/bin can't be a symlink.

Simple fix?

Posted Dec 15, 2018 6:28 UTC (Sat) by fest3er (guest, #60379) [Link] (2 responses)

Never mind that some versions of tar have a tendency to remove the /usr/bin symlink and replace it with a real dir. But it is kind of hard for tar to know what the sysadm wants it to do....

IMO, I would've thought it better to merge /usr/bin and /usr/lib into /bin and /lib, respectively. That is, return to the original UNIX tree and discard the hack that was required when hard drives were ' too small'.

Simple fix?

Posted Dec 15, 2018 6:39 UTC (Sat) by mjg59 (subscriber, #23239) [Link]

If you want an immutable environment with binaries then making /bin the canonical path means /etc and /var (and, if appropriate, /srv and friends) need to be separate mount points and that's more awkward than just making /usr immutable and having / as the thing that holds state.

Simple fix?

Posted Dec 15, 2018 16:54 UTC (Sat) by smurf (subscriber, #17840) [Link]

Nobody prevents you from marking the /bin symlink as immutable, assuming file system support. tar, rsync & co. will then no longer be able to annoy you.