Unexpected fallout from /usr merge in Debian
Back in 2011, Harald Hoyer and Kay Sievers came up with a proposal for Fedora to merge much of the operating system into /usr; former top-level directories, /bin, /lib, and /sbin, would then become symbolic links pointing into the corresponding subdirectories of /usr. Left out of the merge would be things like configuration files in /etc, data in /var, and user home directories. This change was aimed at features like atomic upgrades and easy snapshots. The switch to a merged /usr was successful for Fedora 17; many other distributions (Arch, OpenSUSE, Mageia, just to name a few) have followed suit. More recently, Debian has been working toward a merged /usr, but it ran into some surprising problems that are unique to the distribution.
Debian and its derivatives are definitely late to the /usr merge party. Systems running Debian testing that were initially installed before June 2018 still have /bin, /sbin, and /lib as normal directories, not as symbolic links. The same applies to Ubuntu 18.10. But both Debian and Ubuntu want to make the switch to a merged /usr. Debian tried, but it hit something completely unexpected.
The Debian /usr merge history started in 2016, when Marco d'Itri got the usrmerge package into Debian unstable. This package contains a Perl script that converts an existing system into the state with a merged /usr. Also, a change was made to the debootstrap program (which installs a Debian system into a chroot), so that it could create the needed symbolic links by itself before installing any packages. The end result is the same in both cases.
The plan was to default to a merged /usr when it is ready (and the initial testing revealed only three broken packages), but it was hoped that both merged and non-merged setups would be supported. In other words, the expectation was that there would be no flag day when everyone must switch.
The decision on whether /usr merge would be done by default has changed multiple times during the debootstrap development timeline. The initial support was coded in September 2016, and released in debootstrap version 1.0.83, in a disabled-by-default state. In October 2016 there was an attempt to enable it by default, but this was reverted in November, because the dpkg-shlibdeps program (which is used during package builds to automatically generate dependencies on packages that provide the needed shared libraries) broke. Therefore, Debian 9.0 (with the code name "Stretch") was released in June 2017 without this feature.
A second attempt to re-enable /usr merge by default happened in June 2018, with debootstrap version 1.0.102. Since June 25, 2018, new default installations of Debian testing from the "daily" builds of the install CD have a merged /usr, and that has not been reverted so far.
Happy ending? Not so fast ...
Problems with R
On November 10, a new version of debootstrap was uploaded to the "backports" repository for the Debian stable distribution. This repository is not enabled on end-user systems by default, but it is enabled on the build daemon (buildd) machines for the purpose of getting a few updated packages including debootstrap. Therefore, shortly thereafter, Debian build daemons started using a merged /usr for their chroots. One of them attempted to automatically rebuild the r-base package and produced an R binary that did not work on non-merged systems. As the bug report says:
/usr/bin/R: line 193: /usr/bin/sed: No such file or directory
Traditionally, sed was always placed in /bin. The
Debian package sed also has /bin/sed, not
/usr/bin/sed. In the bug report, the problem is treated like a
one-off issue, to be solved by a rebuild. However, on the debian-devel
mailing list, Ian Jackson quickly pointed
out that the problem is, in fact, due to /usr
merge on the build daemons. He suggested that the change should be
reverted. Dirk Eddelbuettel seconded
that suggestion, and noted that he
expects "much more breakage to follow
". Indeed, similar problems
were triggered in sympow, pari, and monitoring-plugins. Other bugs
of this nature can be found by searching
the Debian bug tracking system for a special tag (but this search also
finds other kinds of issues).
Jackson provided a good explanation of the mechanics of what has happened, quoted below.
- R's autoconfery is autodetecting the location of (say) sed at build time by searching the PATH. R then bakes the discovered path into the built binaries.
- With usrmerge, /bin is a symlink to /usr/bin [...].
- Consequently the R autoconfery always detects (say) sed in the first place out of /usr/bin and /bin it looks, and bakes /usr/bin/sed into its binaries.
- On a system without usrmerge, /usr/bin/sed does not exist, because sed is in /bin.
Matthias Klumpp suggested that a sensible build system would automatically detect the correct paths even on a /usr-merged system; he also wondered why the issue with incorrect paths has not been encountered by other distributions. The crucial difference was found to be that all other distributions had a flag day, before which merged /usr was unsupported, and after which it was mandatory. That is, on other distributions there was no requirement that a package built on a /usr-merged system should work correctly on a system without a merged /usr.
D'Itri said that there is only a handful of packages affected and provided an example of what a fix would look like. Namely, the proper locations of programs should be provided to the configure script or its equivalent, or the software has to be modified so that it always looks for programs using $PATH at runtime. In fact, Debian maintainer scripts (that is, scripts written by Debian maintainers that run before or after the package installation, upgrade, or removal) are already required, by Debian policy, to use $PATH instead of hard-coding the paths to the tools they use. There is, however, no such requirement in the policy for programs other than maintainer scripts.
The argument that there is only a handful of affected packages was not compelling enough, because the number of packages affected is, in fact, unknown without someone trying to rebuild them all. So Simon McVittie added a new check to the reproducible builds infrastructure, specifically for differences caused by /usr merge. This check caught problems in perl, quilt, and systemd.
On November 20, the change was reverted — build daemon chroots were recreated without a merged /usr.
Plan B?
Due to the perceived damage to the distribution caused by /usr
merge, Adam Borowski started a new
discussion thread
on the debian-devel list, with the subject "usrmerge — plan B?". He
proposed to scrap the usrmerge package, and, instead, move binaries
and libraries to /usr one by one. "If it takes 10 years, so
what?
" McVittie
pointed out
that moving binaries on a package-by-package basis would not solve anything.
Indeed, any move of a binary would break all packages that hard-coded its
expected location, which is exactly the same problem that was triggered by
/usr merge on the build daemons.
The arguments usually presented by /usr-merge proponents have also been questioned, for example on the basis that atomic upgrade is worthless if it leaves the system with mismatching /etc or /var. McVittie wrote a long email in response to the complaint about a lack of detailed justification of the change. The criticism regarding /var and /etc mismatch was answered by showing how existing systems with immutable /usr (like OSTree and systemd-based stateless systems) are dealing with it.
McVittie explained the /usr merge failure mode again and mentioned the one-way compatibility property: packages built on a system without a merged /usr would work fine on a merged-/usr system, but not the other way around. Therefore, he concluded that the transition simply happened, by accident, in the wrong order: the build daemons should have transitioned last.
The rest of the email presents the benefits of /usr merge. In
particular, "it's a significant simplification for reliable
special-purpose systems
" such as consumer appliances. It brings benefits
for end-user systems, too, because of increased portability of scripts from
other systems.
This is important because some upstream authors work on merged-/usr
systems and don't know (and don't need to know) the "proper" location of
binaries. In addition, it was noted that the change would simplify the
security-relevant code of sandboxing systems like bubblewrap.
But Borowski's objection was not to /usr merge itself, but of the need to support systems both with and without a merged /usr. In other words, according to him, in order to be supportable, /usr merge must either be mandatory or not happen at all.
The discussion also touched upon the topic of the transition freeze that
will happen in January 2019 and limits the time available for planning
and testing any changes.
So far, there is no consensus, but the most popular opinion
seems to be that a mandatory /usr merge has to be postponed until
after the
release. As Russ Allbery
put it:
"That's going to be a disappointing delay for a lot of
people, I'm sure, but it's still better for them than never doing this at
all.
"
Ubuntu also plans to go through /usr merge. Dimitri John Ledkov described the situation this way: in the upcoming Ubuntu 19.04 release, new installations will use merged /usr, but existing ones will not have their /usr merged on upgrade. Of course, the official Ubuntu build daemons will not have merged /usr.
The status quo in Debian is that debootstrap (and, thus, the Debian installer) installs a system with /usr merged, and the build daemons explicitly pass a command-line switch to avoid this. This is still dangerous, because initial uploads of new packages in Debian (but not in Ubuntu) require a binary package built by the developer on his system. As long as a single uploader has a system with merged /usr, there is a potential for the bug to resurface in his Debian package. On the other hand, as d'Itri noted, there are already too many possibilities for a Debian maintainer to misbuild a package by not using a chroot, so there is nothing new in this danger. Anyway, support for marking packages as "tainted" by the build environment has been implemented in dpkg in response to this concern, and dpkg taints the package if it detects the symbolic links related to /usr merge.
The discussion is still in progress, though; no consensus has been reached. A bug was filed against debootstrap by Jackson to revert the change to merge by default for the next release of Debian. Due to the disagreement of the debootstrap maintainer to the proposed change, Jackson reassigned the bug to the Debian Technical Committee, which is the ultimate authority for resolving otherwise unresolvable technical disputes within Debian. There is also a request from the Debian backports FTP master that the default should be the same in Debian stable backports and in Debian testing. Emilio Pozuelo Monfort, a member of the release team, also spoke in favor of reverting to non-merged /usr in new installations.
It is impossible to predict now how the Technical Committee will rule. In the worst case for /usr-merge proponents, proper introduction of a merged /usr into Debian may be delayed by a few more years. But, if it votes for keeping the status quo, new end-user systems in the next stable release of Debian will have merged /usr, old but upgraded ones won't, and the build daemons will reliably build packages suitable for both cases, just like what's planned for Ubuntu 19.04. No flag day is needed in this scenario, so it would follow the best Debian traditions of not forcing transitions onto users.
| Index entries for this article | |
|---|---|
| GuestArticles | Patrakov, Alexander E. |
Posted Dec 4, 2018 19:14 UTC (Tue)
by rweikusat2 (subscriber, #117920)
[Link] (6 responses)
Posted Dec 4, 2018 20:53 UTC (Tue)
by quotemstr (subscriber, #45331)
[Link] (5 responses)
Posted Dec 4, 2018 21:05 UTC (Tue)
by rweikusat2 (subscriber, #117920)
[Link] (4 responses)
Considering that you've already managed to come up with a boilerplate (and uncalled for) conjecture about "my hidden personality defects", there's obviously no point in discussing this.
Posted Dec 4, 2018 21:13 UTC (Tue)
by quotemstr (subscriber, #45331)
[Link] (3 responses)
Posted Dec 6, 2018 6:02 UTC (Thu)
by k8to (guest, #15413)
[Link] (2 responses)
The generic you is often part of a conjecture.
"Say you want ...."
Posted Dec 6, 2018 15:59 UTC (Thu)
by quotemstr (subscriber, #45331)
[Link] (1 responses)
I maintain that the explosive reaction to my original post was uncharitable and hostile, and I have zero desire to interact with someone who's demonstrated that he'll adopt a creative interpretation of an innocious post and treat it, in public, as some kind of ridiculous offense.
Posted Dec 7, 2018 2:17 UTC (Fri)
by k8to (guest, #15413)
[Link]
Posted Dec 4, 2018 19:21 UTC (Tue)
by mbiebl (subscriber, #41876)
[Link] (33 responses)
Posted Dec 4, 2018 20:40 UTC (Tue)
by clugstj (subscriber, #4020)
[Link] (28 responses)
Posted Dec 4, 2018 22:48 UTC (Tue)
by nix (subscriber, #2304)
[Link] (27 responses)
(Right?)
Posted Dec 4, 2018 23:12 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (21 responses)
Then it evolved again, initializer systems were kinda made obsolete by initrd/initramfs and Busybox took over the task of the recovery system.
Posted Dec 4, 2018 23:57 UTC (Tue)
by nix (subscriber, #2304)
[Link] (20 responses)
Posted Dec 5, 2018 0:02 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Posted Dec 11, 2018 15:56 UTC (Tue)
by nix (subscriber, #2304)
[Link]
(Immutable executable code, really? I know people with /usr/backup, /usr/finance, /usr/god-only-knows-what-stuff. Sure, it's not in the FHS, but /usr/src *is*, and is neither immutable nor executable.)
Posted Dec 5, 2018 8:31 UTC (Wed)
by epa (subscriber, #39769)
[Link] (16 responses)
Posted Dec 5, 2018 9:52 UTC (Wed)
by mgedmin (subscriber, #34497)
[Link] (1 responses)
(There's still a 'bin' user in my /etc/passwd on my Ubuntu 18.10, but its home directory is just /bin now, and it doesn't own any files.)
Posted Dec 6, 2018 6:20 UTC (Thu)
by draco (subscriber, #1792)
[Link]
Posted Dec 5, 2018 17:53 UTC (Wed)
by jezuch (subscriber, #52988)
[Link] (12 responses)
Posted Dec 5, 2018 19:20 UTC (Wed)
by mpr22 (subscriber, #60784)
[Link] (2 responses)
Posted Dec 5, 2018 19:52 UTC (Wed)
by karkhaz (subscriber, #99844)
[Link]
https://en.wikipedia.org/wiki/Ken_Thompson#/media/File:Ve...
It's Version 6 Unix running on an emulator; it shows a directory called /usr/ken.
Posted Dec 6, 2018 9:35 UTC (Thu)
by epa (subscriber, #39769)
[Link] (7 responses)
Posted Dec 6, 2018 10:12 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (5 responses)
Posted Dec 6, 2018 13:29 UTC (Thu)
by pbonzini (subscriber, #60935)
[Link]
Posted Dec 7, 2018 10:17 UTC (Fri)
by k8to (guest, #15413)
[Link] (3 responses)
I suppose directories. Though that isn't in the name. So I guess it's just wrong.
Posted Dec 7, 2018 10:43 UTC (Fri)
by excors (subscriber, #95769)
[Link] (2 responses)
> Program Files are not the same as Programs. Programs are things like Calc, Notepad, Excel, Photoshop. They are things you run.
Posted Dec 7, 2018 21:34 UTC (Fri)
by epa (subscriber, #39769)
[Link] (1 responses)
Posted Dec 10, 2018 14:53 UTC (Mon)
by cortana (subscriber, #24596)
[Link]
Posted Dec 11, 2018 16:01 UTC (Tue)
by nix (subscriber, #2304)
[Link]
As for 'resources', Unix never uses that term at all. *Windows* calls things 'resources', but Unix? The most you get is the relatively recent /usr/share, which emphasises the shareability of data, not whether or not it is a 'resource', whatever sort of thing that's supposed to be.
If /usr really did mean 'Unix system resources', it would contain the shell, the kernel source, libc, and probably not a lot else. :)
Posted Dec 6, 2018 1:47 UTC (Thu)
by djbon2112 (guest, #101854)
[Link]
Posted Dec 5, 2018 12:03 UTC (Wed)
by smcv (subscriber, #53363)
[Link]
I think this was briefly considered for Fedora, but it's actually less good than the /usr merge. In the layout the Hurd tried, the static-data directories (/bin, /sbin, /lib* and /share) are siblings of /etc (which must be on the root filesystem because it's where /etc/fstab is located). However, the static-data directories are more similar than they are different (for instance they can be mounted read-only during normal operation, if you have a strategy for making them briefly read/write during upgrades), so it makes sense to group them together.
The name /usr for that group is historical accident, sure, but the fact that they're a group does seem good.
Posted Dec 5, 2018 14:05 UTC (Wed)
by nivedita76 (subscriber, #121790)
[Link]
Posted Dec 5, 2018 22:41 UTC (Wed)
by wahern (subscriber, #37304)
[Link] (1 responses)
I'm not taking sides, either wrt /bin -> /usr/bin or even /var/tmp -> /tmp, but there are good reasons for opposing those changes. The original motivation or design of something is not the sole determiner of its utility. It doesn't matter if a benefit is by accident or path dependency.
Posted Dec 11, 2018 16:16 UTC (Tue)
by nix (subscriber, #2304)
[Link]
In my most recent setups I've just been using one huge XFS filesystem where possible, perhaps with others hived off where encryption with different keys requires it, but otherwise just One Big FS. XFS slices the fs into ags anyway, so we don't see a performance loss from the fs putting things hugely far apart from each other: the only real risk is fs corruption causing the fs to not mount, and the fix there is a proper initramfs or a rescue disk, and good backups (of course I have *both*, a rescue partition rsynced at intervals from selected parts of the main system *and* an initramfs that recognizes boot failure and boots off it if need be).
You do need protection against disk corruption, but with modern titanic disk sizes the right approach is probably to store the core system more than once, so you can use *it* as a recovery tool, rather than slicing everything off the core system into smaller pieces so you only lose one bit to corruption at once. (And often corruption comes in the form of the whole disk dying or a bug somewhere roasting *all* your writeable filesystems, not just one. So it doesn't win you as much as it might, and its costs are substantial.)
As with all engineering, it's a tradeoff, and often the right answer is the simplest ("a few big filesystems and forget all about it, dammit").
Posted Dec 6, 2018 6:04 UTC (Thu)
by k8to (guest, #15413)
[Link] (1 responses)
Posted Dec 11, 2018 16:10 UTC (Tue)
by nix (subscriber, #2304)
[Link]
The *article* is excellent. :)
Posted Dec 5, 2018 8:11 UTC (Wed)
by edeloget (subscriber, #88392)
[Link]
I read it twice (once before I read your comment, and another time after) and I can't see how this article is opinionated. Care to explain?
Posted Dec 6, 2018 15:59 UTC (Thu)
by jschrod (subscriber, #1646)
[Link] (2 responses)
2) Would you please disclose if you are involved in that discussion, and, if yes, what your viewpoint is? Maybe we can then better determine the background of your argument.
FTR: I'm not involved in that discussion. I have lived with separate /bin and /usr/bin for several decades. (I still remember the time when my home directory was *directly in* /usr/ and not in some new-fangled-thing named /home or /usr/home. Oh yes, and I have a grey beard.) I also use some systems with merged /usr since a few years, doesn't make a difference.
Posted Dec 8, 2018 5:13 UTC (Sat)
by dirtyepic (guest, #30178)
[Link] (1 responses)
Posted Dec 14, 2018 0:39 UTC (Fri)
by jschrod (subscriber, #1646)
[Link]
/usr/lib came later, btw -- we didn't have shared libraries yet. I don't remember exactly, but maybe that's why we don't have a pointless "lib" account on our Linux systems, while "bin" still is there, without any rationale.
When I read nowadays that /bin vs. /usr/bin was a great architectural decision that should not be removed because it ain't the Unix Way(tm) -- well, I scratch my grey beard and try to avoid the worshipping discussion of the "Unix fathers". I had occasion to have several dinners with Dennis (Ritchie) and Ken (Thompson), and from that personal experience I'm sure that they would laugh at the thought that this might have been a technical decision and not some technical coincidence.
Posted Dec 4, 2018 19:50 UTC (Tue)
by ballombe (subscriber, #9523)
[Link] (4 responses)
There is one common use case where you have to use the full path:
Of course you can use env (even though it is discouraged by Debian policy)
> The Debian /usr merge history started in 2016
What is missing from the article is what problem usrmerge is trying to solve.
Debian never followed the lib32/lib64 trend and it was for the best.
Posted Dec 4, 2018 20:41 UTC (Tue)
by wahern (subscriber, #37304)
[Link]
The other problem you mentioned, shebang word splitting semantics, remains.
Posted Dec 5, 2018 7:13 UTC (Wed)
by eru (subscriber, #2753)
[Link] (1 responses)
I have wondered about this. With scripting languages, passing more options on a #! line would sometimes be useful. Is there a technical reason, or just tradition? If an unlimited argument count is not feasible (and is probably not needed), even raising the limit to 10 arguments or so would help.
Posted Dec 5, 2018 16:50 UTC (Wed)
by Liskni_si (subscriber, #91943)
[Link]
Posted Dec 5, 2018 13:55 UTC (Wed)
by drag (guest, #31333)
[Link]
In a ideal world env should NOT be used for any OS-level software. So OS package managers, system configuration utilities, desktop management and display software, etc. However for applications and services that run on top of the OS then env should be used.
The idea being that a user can set up their own environments how they like with their own custom versions of python, ruby, etc and libraries and custom software for development purposes (or whatever) and have the software they use be able to take advantage of that. But still have a functional OS and desktop that doesn't break.
Unfortunately in Linux-land there is often very little distinction made between what is 'OS' and what is 'Applications'. They tend to get released all together as one big 'mush' of interconnected dependencies. So unfortunately there is no right way to do anything without resorting to sandboxing.
Posted Dec 4, 2018 20:51 UTC (Tue)
by juliank (guest, #45896)
[Link] (3 responses)
Posted Dec 5, 2018 10:30 UTC (Wed)
by smurf (subscriber, #17840)
[Link] (1 responses)
Posted Dec 7, 2018 2:28 UTC (Fri)
by spwhitton (subscriber, #71678)
[Link]
Posted Dec 7, 2018 2:29 UTC (Fri)
by spwhitton (subscriber, #71678)
[Link]
Posted Dec 4, 2018 21:13 UTC (Tue)
by epa (subscriber, #39769)
[Link] (8 responses)
Posted Dec 4, 2018 22:43 UTC (Tue)
by patrakov (subscriber, #97174)
[Link] (7 responses)
Anyway, as demonstrated by the team behind reproducible builds, there are only 80 or so packages to fix.
Posted Dec 4, 2018 22:50 UTC (Tue)
by patrakov (subscriber, #97174)
[Link]
Posted Dec 5, 2018 9:14 UTC (Wed)
by epa (subscriber, #39769)
[Link] (5 responses)
I do agree with the general goal of fixing these packages that have their own strange ideas about PATH, but it needn't stand in the way of getting /usr merge done, because the packages can be made to work with symlinks both before and after the transition. The important part is to make sure the symlinks from /usr/bin/x to /bin/x exist on deployed systems before any rebuilt packages are pushed out. But while running the usrmerge script is quite a big change, creating a few symlinks is not.
Posted Dec 5, 2018 12:09 UTC (Wed)
by smcv (subscriber, #53363)
[Link] (1 responses)
The problem case that you're missing here is partial upgrades. If (for example) ping versions < X have only /bin/ping, ping version X adds a /usr/bin/ping symlink, and gnome-nettool version Y was built with a ping version >= X installed and hard-codes /usr/bin/ping into its binaries, there is no mechanism for gnome-nettool version Y to get an automatic dependency on ping (>= X). A user could install gnome-nettool version Y with ping version < X, and it wouldn't find /usr/bin/ping.
Posted Dec 6, 2018 7:59 UTC (Thu)
by epa (subscriber, #39769)
[Link]
Posted Dec 5, 2018 14:14 UTC (Wed)
by nivedita76 (subscriber, #121790)
[Link] (2 responses)
Posted Dec 6, 2018 6:28 UTC (Thu)
by lathiat (subscriber, #18567)
[Link]
Posted Dec 6, 2018 7:57 UTC (Thu)
by epa (subscriber, #39769)
[Link]
“At which point you might as well just merge them.”
Well, probably. But as something to do to all installed systems, creating symlinks seems like a much safer option as a first step, not requiring any reboot, and easy to undo if necessary. Then the much more elaborate operation of doing the merge could happen at a slower pace, with no flag day.
Posted Dec 4, 2018 21:21 UTC (Tue)
by jezuch (subscriber, #52988)
[Link] (3 responses)
Well, without a forced upgrade this will never happen. I installed my Debian in 2003 and had to reinstall just once: to switch from x86 to amd64 (this was before multiarch; with multiarch this could probably be done without a reinstall). (I may also have had one situation when I broke my system beyond recovery, but I don't even remember this so it may have not happened at all ;) ). My point is that there are plenty of users like me who regularly upgrade and have the new and shiny Debian without ever reinstalling. If there is no forced switch to merged /usr, these systems will remain forever with unmerged /usr.
Posted Dec 4, 2018 21:36 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Dec 5, 2018 17:57 UTC (Wed)
by jezuch (subscriber, #52988)
[Link] (1 responses)
Posted Dec 5, 2018 22:17 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Dec 4, 2018 23:56 UTC (Tue)
by dkg (subscriber, #55359)
[Link] (9 responses)
(fwiw, i'm a proponent of the usrmerge, and would be happy if it happened sooner rather than later, but i still dislike having unrelated paths baked into the binaries)
Posted Dec 5, 2018 0:47 UTC (Wed)
by wahern (subscriber, #37304)
[Link] (4 responses)
Posted Dec 5, 2018 4:54 UTC (Wed)
by dkg (subscriber, #55359)
[Link] (3 responses)
If a package needs a specific version of a utility, it needs to do one of three things:
Posted Dec 5, 2018 22:23 UTC (Wed)
by wahern (subscriber, #37304)
[Link]
Asking a C program (or any program) to implement autoconf-style utility detection is like begging for CVEs. Expecting people to use a non-existent libsed instead sed is... unreasonable. Shipping a wrapper of the utility is better, but then /usr/bin or [preferably] /usr/libexec would be littered with redundant wrappers.
The cost+benefit just isn't there for a "better" solution. The current solutions were manifestly good enough. It asks too much to expect packages to foresee *and* accommodate something like a wholesale shift of utilities from /bin to /usr/bin in addition to the myriad other issues they must contend with, and especially in light of the historic stability of core utility paths. For non-shell programs, the decision to rely on shell utility at runtime is already predicated on expediency.
Regarding the issue with relying on $PATH--it's far more reasonable to rely on a sane $PATH at compile-time than runtime. The reasons are self-evident.
FWIW, Ubuntu's r-base package's configure.ac uses `AC_PATH_PROGS(SED, sed, /bin/sed, [/usr/xpg4/bin:$PATH])`. We could nit-pick that endlessly, but why bother.
Posted Dec 7, 2018 5:45 UTC (Fri)
by zblaxell (subscriber, #26385)
[Link] (1 responses)
autoconf (or its immediate predecessors) were designed for in-place self-hosted builds, where the runtime environment is fairly similar (if not entirely identical) to the build-time environment, and software is delivered to the end user as source code.
configure scripts can not only figure out where sed is, they can also test all the seds they can find, pick the one that has the right mix of extensions and bugs, and burn its complete path into the binary so the Wrong Sed will never be accidentally run.
On the other hand, that assumption is ~25 years old, give or take a decade...which is probably why no modern build systems look anything like autoconf.
Posted Dec 7, 2018 8:59 UTC (Fri)
by wahern (subscriber, #37304)
[Link]
Rather than hardcoding paths to shared resources all the modern build systems (Docker, Go, pipenv, FlatPak, etc) literally duplicate multi-megabyte environments. They're increasingly hardcoding snapshots of the entire OS.
Posted Dec 5, 2018 1:25 UTC (Wed)
by KaiRo (subscriber, #1987)
[Link] (3 responses)
Posted Dec 5, 2018 2:05 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Dec 5, 2018 14:21 UTC (Wed)
by nivedita76 (subscriber, #121790)
[Link] (1 responses)
Posted Dec 7, 2018 2:37 UTC (Fri)
by k8to (guest, #15413)
[Link]
Of course compiling in paths and using fixed strings on exec as opposed to execvp etc aren't the same, but "just use the PATH' is often not the right approach from a defense in depth perspective.
Posted Dec 5, 2018 3:08 UTC (Wed)
by flussence (guest, #85566)
[Link] (4 responses)
Wouldn't it be nice if there'd been separate /usr/*-pc-linux-gnu-c++0x and -c++1x directories instead of a flag day where everything had to be rebuilt/reinstalled in-place when GCC 5 came out? If CPU-dependent code was placed in separate directories, /usr/lib and /usr/bin could be kept arch-neutral.
Posted Dec 5, 2018 4:16 UTC (Wed)
by sbaugh (guest, #103291)
[Link]
The problem that Debian has run into (a resolved symlink being baked into the built binaries, resulting in the binaries not being portable to systems where that symlink points to something else or indeed doesn't exist at all) is a well known and long-solved problem from the perspective of Nix/Guix. Nix even refuses to work and requires overriding if the system is set up with the top-level package directory as a symlink - a bind mount is recommended instead. (Perhaps Debian could use the same solution?)
Posted Dec 5, 2018 9:39 UTC (Wed)
by matthias (subscriber, #94967)
[Link] (2 responses)
Is there a chance that this would have worked without pain? How is it with programs pulling libs from both directories? These libs could then pull in the same lib from both directories and probably even try to pass around pointers across the barrier. This might be bad design, but if we would eliminate all software that uses some bad practices, there would not be a need for these directories anyway, as I would assume they would be empty ;)
Maybe you could build every library in two versions before starting to migrate applications. But this would also be a pain, as the build processes are not designed to build every library twice.
Posted Dec 8, 2018 3:36 UTC (Sat)
by wahern (subscriber, #37304)
[Link] (1 responses)
As for mixing ABIs, the obvious solution is to mimic the existing multiarch support and build everything twice. Projects and packages can't normally build for different arches simultaneously, but packaging systems don't expect them to. It's why RPM and Debian packages do in-tree builds by default instead of out-of-tree builds: even most automake projects are subtly broken in that regard. It's a lost cause and not worthy of consideration.
[1] I mean, you can obviously do it by explicitly setting rpaths during compilation. But to do it in a manner consistent with default search paths rules and the application of global policy (i.e. ldconfg), I'd be surprised if there weren't more footwork required.
Posted Dec 8, 2018 6:47 UTC (Sat)
by matthias (subscriber, #94967)
[Link]
OK. Then this would not require a flag day for switching from gcc4 to gcc5, but instead a flag day from when on we expect every library to exist twice. Or we should convert libraries for double build one by one according to the dependency graph (a tree would be to simple), which would probably take ages.
On Gentoo, I really liked the flag day approach. And it was more or less a day to recompile everything. But on Gentoo, every user could decide on his own when this flag day should happen. And after this day, everything was done.
Posted Dec 5, 2018 6:18 UTC (Wed)
by abo (subscriber, #77288)
[Link] (2 responses)
This seems dangerous to me. Why even allow uploads of binaries?
Posted Dec 5, 2018 8:42 UTC (Wed)
by laarmen (subscriber, #63948)
[Link]
Of course, this reasoning is somewhat flawed since the maintainer's build environment might not be clean, and I don't think there's any proof that the .deb uploaded has been built with the accompanying source package. However, one other factor to consider is that when the policy was implemented, I'm guessing the Debian project had a lot less infrastructure available to build a package, thus this might also have been a sort of optimization.
Posted Dec 5, 2018 9:05 UTC (Wed)
by plugwash (subscriber, #29694)
[Link]
In the early days of Debian all binaries were built and uploaded manually alongside the source. When m68k support was added porters would build m68k binaries from the existing source and upload them. Gradually as Debian got bigger this process of building binaries for additional ports became automated but still the archive Demanded at least one binary with each source package and there was no mechanism for autobuilding architecture dependent packages.
Another quirk of Debian is that uploads of new packages (including adding of extra binary packages to existing source packages) go through extra checks by the ftpmasters, some of this checking is manual some is automated and some of it involves the binary packages.
The result if this is that while in 2014 the ftpmasters allowed source-only regular uploads and in 2015 the autobuilder system added support for building architecture independent packages**, binaries are still required to be part of the upload for "new" packages.
One option would be to make the archive discard the binaries from source+binary uploads unless a special flag was set (or discard them unconditionally and require two seperate uploads) but there are complications with doing that. For example the archive currently only allows each changes file name to be used once, but the normal changes files names for a source+binary upload and a binary only upload are the same, so if the archive discards the maintainer binaries for an architecture and then the autobuilder builds them the upload for the autobuilder is likely to be rejected.
None of this stuff is unfixable, but it takes time and effort to firstly persuade people that a solution is the correct one and then actually implement it.
* In ubuntu binary uploads that do not come from the autobuilders are considered highly exceptional and can only be performed by a handful of people, but they are occasionally necessary. Consider for example compilers that require themselves to build.
** architecture independent packages are always autobuilt on amd64, there are a handful of architecture independent packages that can only be built on a subset of architectures that does not include amd64 and so cannot be autobuilt.
Posted Dec 5, 2018 9:09 UTC (Wed)
by plugwash (subscriber, #29694)
[Link]
And it would still be dangerous even if Debian did not allow binary uploads because not all packages used on Debian systems come from Debian. Is it really acceptable for a binary package built on a freshly installed regular user system to fail on a regular user system of the same release that resulted from an upgrade?
Posted Dec 5, 2018 17:23 UTC (Wed)
by dbnichol (subscriber, #39622)
[Link]
1. Some packages install to /bin and then add a symlink in /usr/bin (or vice versa) to support both paths. That would break in weird ways when installing on merged /usr systems. I just hacked all these packages (there were maybe 3 of the subset of Debian we use) to not create the symlink.
2. dpkg-shlibdeps would think a library wasn't in the expected location when /lib was a symlink. My approach (since we had no intention of supporting old lib64 style multiarch) was to just remove the relevant code - https://github.com/endlessm/dpkg/pull/2/commits/e08c3c1c0....
3. The package builder always uses non-merged /usr so that installs work correctly in either case. I didn't want to run into the case where package files suddenly started moving around on upgrades based only on the builder. The downside is that if you ask dpkg about a file it may not be the canonical path and dpkg will says it's unhandled. E.g., "dpkg -S /bin/sed" works, but "dpkg -S /usr/bin/sed" does not even though the latter is the real file on the system.
Posted Dec 5, 2018 20:52 UTC (Wed)
by jccleaver (guest, #127418)
[Link]
The solution? Hard-code the historical locations (see below):
https://fedoraproject.org/wiki/Packaging:Guidelines#Effect_of_the_UsrMove_Fedora_Feature
So even though it fixes the problem for all the atomic folks, paths never really went away and probably never really will. Surprised Debian hadn't prepared for this if they had decided they did eventually want to proceed after all this time.
Posted Dec 13, 2018 15:23 UTC (Thu)
by mightypenguin (guest, #129211)
[Link] (4 responses)
Posted Dec 14, 2018 10:41 UTC (Fri)
by mattheww (guest, #108205)
[Link] (3 responses)
Posted Dec 15, 2018 6:28 UTC (Sat)
by fest3er (guest, #60379)
[Link] (2 responses)
IMO, I would've thought it better to merge /usr/bin and /usr/lib into /bin and /lib, respectively. That is, return to the original UNIX tree and discard the hack that was required when hard drives were ' too small'.
Posted Dec 15, 2018 6:39 UTC (Sat)
by mjg59 (subscriber, #23239)
[Link]
Posted Dec 15, 2018 16:54 UTC (Sat)
by smurf (subscriber, #17840)
[Link]
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
No. /usr serves a clear purpose - it indicates that this part of the filesystem contains immutable executable code and its supporting data. It makes it easy to reason about it and do atomic operations on it.
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
>
> Program Files are things like ACRORD32.DLL and TWCUTCHR.DLL. They are files that make programs run.
>
> If the directory were named Programs, then people who wanted to run a program would start digging into that directory and seeing a page full of weird DLL names and wonder "What the heck kind of programs are these?" And eventually they might figure out that if they want to run PowerPoint, they need to double-click on the icon named POWERPNT. "Computers are so hard to use."
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
#! path.
It is not uncommon to do
#! @PATH@
and have configure put the right absolute path.
but that does not help because env itself is affected.
Also usin genv prevnt you to pass an option since the kernel allows a single option.
Actually it started in 2002 in Debian GNU/Hurd (see debian-devel) with a /usr ->/ symlink
Unexpected fallout from /usr merge in Debian
Also usin genv prevnt you to pass an option since the kernel allows a single option.
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Add symlinks in /usr
Add symlinks in /usr
Add symlinks in /usr
Add symlinks in /usr
Add symlinks in /usr
Add symlinks in /usr
Add symlinks in /usr
Add symlinks in /usr
Add symlinks in /usr
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
The problem here is baked-in paths in binaries that should know better. those binaries should generally rely on $PATH at runtime anyway, so i'm glad we're discovering this and getting it fixed, rather than punting it to a flag day. I ran into this with GnuPG recently, and ended up patching out this misbehavior in debian once it became clear that upstream wasn't going to fix it, due to abandoning the tool that was doing it.
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Sure, but relying on $PATH at build time to learn the location of a tool (e.g. the use of AC_PATH_PROG during ./configure), and then expecting it to match at runtime also isn't always likely to yield the utility you expect, which is exactly the class of bugs we're uncovering here.
Unexpected fallout from /usr merge in Debian
Debian is showing that, given the possibility (but not hard requirement) of a /usrmerge, AC_PATH_PROG doesn't meet any of those standards.
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Unexpected fallout from /usr merge in Debian
Although not identical, there's a similar problem on the RedHat side with RPM dependencies hitting symlinks (or not) where the binary was usually expected to be in a different place.
Not even Fedora has done this completely
[RPM] file dependencies don't work according to what's on the filesystem, they work according to the path specified in the rpm %files section. So an rpm which specified:
%files
/bin/sh
would be able to satisfy a file dependency for /bin/sh but not for /usr/bin/sh. As a packager you may need to pay attention to where other packages expect to find your files. Things that history has placed into /bin, /sbin, /lib, or /lib64 should be listed in the %files section as being in those directories. Things that history placed in /usr/bin, /usr/sbin, etc, should be listed in the %files section as being in %{_bindir}, %{_sbindir}, etc. If you feel that there is some historical confusion as to which directory a program is placed in, you can use a Virtual Provides to list the alternate path.Simple fix?
Simple fix?
Simple fix?
Simple fix?
Simple fix?
