LWN: Comments on "Unexpected fallout from /usr merge in Debian"

Simple fix?

smurf — Sat, 15 Dec 2018 16:54:54 +0000

Nobody prevents you from marking the /bin symlink as immutable, assuming file system support. tar, rsync & co. will then no longer be able to annoy you.

Simple fix?

mjg59 — Sat, 15 Dec 2018 06:39:52 +0000

If you want an immutable environment with binaries then making /bin the canonical path means /etc and /var (and, if appropriate, /srv and friends) need to be separate mount points and that's more awkward than just making /usr immutable and having / as the thing that holds state.

Simple fix?

fest3er — Sat, 15 Dec 2018 06:28:09 +0000

Never mind that some versions of tar have a tendency to remove the /usr/bin symlink and replace it with a real dir. But it is kind of hard for tar to know what the sysadm wants it to do....

IMO, I would've thought it better to merge /usr/bin and /usr/lib into /bin and /lib, respectively. That is, return to the original UNIX tree and discard the hack that was required when hard drives were ' too small'.

Simple fix?

mattheww — Fri, 14 Dec 2018 10:41:20 +0000

On an unconverted system, both /bin and /usr/bin are real directories containing many binaries (that's what being unconverted means). So /usr/bin can't be a symlink.

Unexpected fallout from /usr merge in Debian

jschrod — Fri, 14 Dec 2018 00:39:05 +0000

For sure, but: I experienced personally the introduction of the account "bin" -- introduced to hold binaries that could not be stored in /bin anymore because root disk space ran out. As standard at this time, the home directory of bin was /usr/bin.

/usr/lib came later, btw -- we didn't have shared libraries yet. I don't remember exactly, but maybe that's why we don't have a pointless "lib" account on our Linux systems, while "bin" still is there, without any rationale.

When I read nowadays that /bin vs. /usr/bin was a great architectural decision that should not be removed because it ain't the Unix Way(tm) -- well, I scratch my grey beard and try to avoid the worshipping discussion of the "Unix fathers". I had occasion to have several dinners with Dennis (Ritchie) and Ken (Thompson), and from that personal experience I'm sure that they would laugh at the thought that this might have been a technical decision and not some technical coincidence.

Simple fix?

mightypenguin — Thu, 13 Dec 2018 15:23:22 +0000

Would it work to make /usr/bin a symlink to /bin on unconverted systems?

Unexpected fallout from /usr merge in Debian

nix — Tue, 11 Dec 2018 16:16:50 +0000

I too had a very split system once, /var, /usr, /usr/src, /var/spool/news, /home, /, all different filesystems. But while this did save my bacon in a certain massive filesystem corruption incident (that made its way to LWN :) ). I argued in favour of a mega-split layout here more than once. The eventual problem there is that the tools to manipulate filesystems are... cruder than the ones to manipulate the fs hierarchy, and in the end half these filesystems end up mis-sized and fixing it is a right drag and they're often in half a dozen noncontiguous fragments on rotating storage and LVM is not terribly good at defragmenting them (you have to do it by hand via repeated manual pvmoves).

In my most recent setups I've just been using one huge XFS filesystem where possible, perhaps with others hived off where encryption with different keys requires it, but otherwise just One Big FS. XFS slices the fs into ags anyway, so we don't see a performance loss from the fs putting things hugely far apart from each other: the only real risk is fs corruption causing the fs to not mount, and the fix there is a proper initramfs or a rescue disk, and good backups (of course I have *both*, a rescue partition rsynced at intervals from selected parts of the main system *and* an initramfs that recognizes boot failure and boots off it if need be).

You do need protection against disk corruption, but with modern titanic disk sizes the right approach is probably to store the core system more than once, so you can use *it* as a recovery tool, rather than slicing everything off the core system into smaller pieces so you only lose one bit to corruption at once. (And often corruption comes in the form of the whole disk dying or a bug somewhere roasting *all* your writeable filesystems, not just one. So it doesn't win you as much as it might, and its costs are substantial.)

As with all engineering, it's a tradeoff, and often the right answer is the simplest ("a few big filesystems and forget all about it, dammit").

Unexpected fallout from /usr merge in Debian

nix — Tue, 11 Dec 2018 16:10:46 +0000

It's not in the article -- it's implied by the grandparent (well, great-great-grandparent now) comment's suggestion that this article is 'opinionated', thus that there is something sensible and natural and right about this split between /usr and / and that attempting to clean the mess up, or even *writing an article about it*, is somehow wrong.

The *article* is excellent. :)

Unexpected fallout from /usr merge in Debian

nix — Tue, 11 Dec 2018 16:01:50 +0000

Quite. It's not like Unix has the term 'system resources' anyway. Heck it doesn't even use the term 'system' to mean anything like that, not if the system() libc function is any indication. (It's a rather strange choice of name that shows the historically close connections between the shell and system calls. They really were meant to be the same sort of thing at different layers.)

As for 'resources', Unix never uses that term at all. *Windows* calls things 'resources', but Unix? The most you get is the relatively recent /usr/share, which emphasises the shareability of data, not whether or not it is a 'resource', whatever sort of thing that's supposed to be.

If /usr really did mean 'Unix system resources', it would contain the shell, the kernel source, libc, and probably not a lot else. :)

Unexpected fallout from /usr merge in Debian

nix — Tue, 11 Dec 2018 15:56:21 +0000

Disproof: /usr/src. :)

(Immutable executable code, really? I know people with /usr/backup, /usr/finance, /usr/god-only-knows-what-stuff. Sure, it's not in the FHS, but /usr/src *is*, and is neither immutable nor executable.)

Unexpected fallout from /usr merge in Debian

cortana — Mon, 10 Dec 2018 14:53:29 +0000

I don't think anything in Windows 95 prevented XCOPY deployment. It's just that by then the Windows world was used to the insanity of running an installer that could do _anything_ to your system...

Unexpected fallout from /usr merge in Debian

Thomas — Mon, 10 Dec 2018 14:52:56 +0000

And unmount as umount.

Unexpected fallout from /usr merge in Debian

matthias — Sat, 08 Dec 2018 06:47:43 +0000

Yes, of course. I thought of the problem of pulling in different so-names of the same library. But here, this should already fail at compilation time in any sane build environment.

OK. Then this would not require a flag day for switching from gcc4 to gcc5, but instead a flag day from when on we expect every library to exist twice. Or we should convert libraries for double build one by one according to the dependency graph (a tree would be to simple), which would probably take ages.

On Gentoo, I really liked the flag day approach. And it was more or less a day to recompile everything. But on Gentoo, every user could decide on his own when this flag day should happen. And after this day, everything was done.

Unexpected fallout from /usr merge in Debian

dirtyepic — Sat, 08 Dec 2018 05:13:19 +0000

What you are witnessing is what happens when someone who can't possibly be mistaken encounters something that doesn't support their personal world view. Facts become opinions, objective truth become subjective interpretation, and neutral retelling of events becomes biased slander intended to further some agenda. It's not unusual. We all do it, in some way or another. Some people just do it on the internet.

Unexpected fallout from /usr merge in Debian

wahern — Sat, 08 Dec 2018 03:36:41 +0000

I don't think that's possible without support from both the compilers and the system's run-time linker (e.g. ld-linux.so). And I don't think that support currently exists, or at least isn't sufficient to get the job done.[1] Probably not terribly difficult. Compilers already emit all manner of annotations, and both compilers and linkers (at least glibc's implementation) already have multiarch support. But that sort of toolchain work is the kind of thing that elicits very strong opinions. It might be why it doesn't already exist---nobody wanted to deal with the final gauntlet of coordinating and resolving the policy details.

As for mixing ABIs, the obvious solution is to mimic the existing multiarch support and build everything twice. Projects and packages can't normally build for different arches simultaneously, but packaging systems don't expect them to. It's why RPM and Debian packages do in-tree builds by default instead of out-of-tree builds: even most automake projects are subtly broken in that regard. It's a lost cause and not worthy of consideration.

[1] I mean, you can obviously do it by explicitly setting rpaths during compilation. But to do it in a manner consistent with default search paths rules and the application of global policy (i.e. ldconfg), I'd be surprised if there weren't more footwork required.

Unexpected fallout from /usr merge in Debian

epa — Fri, 07 Dec 2018 21:34:08 +0000

Which is why application directories as used by NextStep (later Mac OS) and RISC OS (inspiring Rox Desktop) are such a good idea. It is strange that Microsoft did not adopt them for Windows 95. Too late now.

Unexpected fallout from /usr merge in Debian

excors — Fri, 07 Dec 2018 10:43:39 +0000

See https://blogs.msdn.microsoft.com/oldnewthing/20131119-00/...

> Program Files are not the same as Programs. Programs are things like Calc, Notepad, Excel, Photoshop. They are things you run.
>
> Program Files are things like ACRORD32.DLL and TWCUTCHR.DLL. They are files that make programs run.
>
> If the directory were named Programs, then people who wanted to run a program would start digging into that directory and seeing a page full of weird DLL names and wonder "What the heck kind of programs are these?" And eventually they might figure out that if they want to run PowerPoint, they need to double-click on the icon named POWERPNT. "Computers are so hard to use."

Unexpected fallout from /usr merge in Debian

k8to — Fri, 07 Dec 2018 10:17:21 +0000

Sad that adding the word "files" offers no value to the user. What else would be in there?

I suppose directories. Though that isn't in the name. So I guess it's just wrong.

Unexpected fallout from /usr merge in Debian

wahern — Fri, 07 Dec 2018 08:59:25 +0000

> which is probably why no modern build systems look anything like autoconf.

Rather than hardcoding paths to shared resources all the modern build systems (Docker, Go, pipenv, FlatPak, etc) literally duplicate multi-megabyte environments. They're increasingly hardcoding snapshots of the entire OS.

Unexpected fallout from /usr merge in Debian

zblaxell — Fri, 07 Dec 2018 05:45:33 +0000

> Sure, but relying on $PATH at build time to learn the location of a tool (e.g. the use of AC_PATH_PROG during ./configure), and then expecting it to match at runtime also isn't always likely to yield the utility you expect

autoconf (or its immediate predecessors) were designed for in-place self-hosted builds, where the runtime environment is fairly similar (if not entirely identical) to the build-time environment, and software is delivered to the end user as source code.

configure scripts can not only figure out where sed is, they can also test all the seds they can find, pick the one that has the right mix of extensions and bugs, and burn its complete path into the binary so the Wrong Sed will never be accidentally run.

On the other hand, that assumption is ~25 years old, give or take a decade...which is probably why no modern build systems look anything like autoconf.

Unexpected fallout from /usr merge in Debian

k8to — Fri, 07 Dec 2018 02:37:05 +0000

Agreed, though the class of concern isn't entirely limited to suid. Any priveledged process or even a process running as another users which runs things via PATH could be potentialy subverted.

Of course compiling in paths and using fixed strings on exec as opposed to execvp etc aren't the same, but "just use the PATH' is often not the right approach from a defense in depth perspective.

Unexpected fallout from /usr merge in Debian

spwhitton — Fri, 07 Dec 2018 02:29:16 +0000

AIUI this is in the works.

Unexpected fallout from /usr merge in Debian

spwhitton — Fri, 07 Dec 2018 02:28:47 +0000

Also to enable their inspection by those reviewing the NEW queue, since the source couldn't be sent out to the buildd network until it's been reviewed.

Unexpected fallout from /usr merge in Debian

k8to — Fri, 07 Dec 2018 02:17:35 +0000

Not engaging on that axis.

Unexpected fallout from /usr merge in Debian

quotemstr — Thu, 06 Dec 2018 15:59:55 +0000

Even if one does read the "you" as specific and not general, there's nothing wrong with saying "X seems okay because it's familiar but if you take a step back, one can see that it's actually pretty bad". This kind of argument says nothing about one's "personality", much less "defects".

I maintain that the explosive reaction to my original post was uncharitable and hostile, and I have zero desire to interact with someone who's demonstrated that he'll adopt a creative interpretation of an innocious post and treat it, in public, as some kind of ridiculous offense.

Unexpected fallout from /usr merge in Debian

jschrod — Thu, 06 Dec 2018 15:59:00 +0000

1) Would you please explain why you think that the article is "very opinionated"?

2) Would you please disclose if you are involved in that discussion, and, if yes, what your viewpoint is? Maybe we can then better determine the background of your argument.

FTR: I'm not involved in that discussion. I have lived with separate /bin and /usr/bin for several decades. (I still remember the time when my home directory was *directly in* /usr/ and not in some new-fangled-thing named /home or /usr/home. Oh yes, and I have a grey beard.) I also use some systems with merged /usr since a few years, doesn't make a difference.

Unexpected fallout from /usr merge in Debian

andrewsh — Thu, 06 Dec 2018 14:39:57 +0000

And create as creat.

Unexpected fallout from /usr merge in Debian

pbonzini — Thu, 06 Dec 2018 13:29:29 +0000

Interesting. Some localized versions not have the space though.

Unexpected fallout from /usr merge in Debian

Cyberax — Thu, 06 Dec 2018 10:12:51 +0000

“Program Files” name was intentional, to force every program to care about spaces in file names.

Unexpected fallout from /usr merge in Debian

epa — Thu, 06 Dec 2018 09:35:15 +0000

An unwieldy acronym like "Unix System Resources" doesn't ring true to me. Not from the people who came up with the name 'Unix' as a pun on 'Multics', and gave short, pithy names to everything else. (Apart from the kernel, does any file on a Unix system have the word 'unix' in its name?) Only large organizations can come up with cumbersome names like "Program Files".

Add symlinks in /usr

epa — Thu, 06 Dec 2018 07:59:22 +0000

I didn’t mean that the symlink would be owned by the ping package itself, but that it and the others would be created by a script run as a one-off. Like the usermerge script which all systems are going to run at some point in the future, but making the symlinks is a much simpler first step.

Add symlinks in /usr

epa — Thu, 06 Dec 2018 07:57:19 +0000

Yes, I understand that. Which is why I suggested adding the symlinks on deployed systems as a first step — then packages like this will just work, whatever path they have baked in.

“At which point you might as well just merge them.”

Well, probably. But as something to do to all installed systems, creating symlinks seems like a much safer option as a first step, not requiring any reboot, and easy to undo if necessary. Then the much more elaborate operation of doing the merge could happen at a slower pace, with no flag day.

Add symlinks in /usr

lathiat — Thu, 06 Dec 2018 06:28:08 +0000

I had to re-read the article twice to get this point (that unmerged systems only have 1 working path), it would be a good clarification to add to the article.

Unexpected fallout from /usr merge in Debian

draco — Thu, 06 Dec 2018 06:20:46 +0000

Thank you. I've long wondered what the bin user was for...

Unexpected fallout from /usr merge in Debian

k8to — Thu, 06 Dec 2018 06:04:53 +0000

Confused. That doesn't seem to be in the article at all (either direction).

Unexpected fallout from /usr merge in Debian

k8to — Thu, 06 Dec 2018 06:02:03 +0000

This escalated quickly, but your comment doesn't read as referencing the general you at all. It can be read that way for sure, but I don't think most people would. Something to read back next time, perhaps.

The generic you is often part of a conjecture.

"Say you want ...."

Unexpected fallout from /usr merge in Debian

djbon2112 — Thu, 06 Dec 2018 01:47:18 +0000

According to a common history, yes. /usr originally had homedirs on disk #2, then applications that wouldn't fit in /bin got put in /usr/bin, then eventually the homedirs themselves moved to /home leaving the "user" programs in /usr. It makes sense in an abstract way - /bin is for OS-critical binaries, while /usr/bin are "things the user installs", with 45+ years of growth on top of it. At least that's how I've always thought of it.

Unexpected fallout from /usr merge in Debian

wahern — Wed, 05 Dec 2018 22:41:11 +0000

A few releases ago OpenBSD made /var/tmp a symlink to /tmp. Like many people (and per OpenBSD's recommendations) I put /var and /tmp on different partitions or devices. Among other things, /tmp can be cleaned on startup by simply initializing the superblock or encrypting with an ephemeral key. But I also generally make /tmp rather small. /var/tmp was useful for the rare times where I needed alot of space for temporary files, such as when making backups. Making /var/tmp a symlink to /tmp has consistently been annoying.

I'm not taking sides, either wrt /bin -> /usr/bin or even /var/tmp -> /tmp, but there are good reasons for opposing those changes. The original motivation or design of something is not the sole determiner of its utility. It doesn't matter if a benefit is by accident or path dependency.

Unexpected fallout from /usr merge in Debian

wahern — Wed, 05 Dec 2018 22:23:01 +0000

It comes down to not letting perfect be the enemy of good. On the surface hardcoding the path seems like a bad idea, but when you're actually in a situation where you need to solve this issue and you consider the cost of those other options then it quickly becomes the least bad option.

Asking a C program (or any program) to implement autoconf-style utility detection is like begging for CVEs. Expecting people to use a non-existent libsed instead sed is... unreasonable. Shipping a wrapper of the utility is better, but then /usr/bin or [preferably] /usr/libexec would be littered with redundant wrappers.

The cost+benefit just isn't there for a "better" solution. The current solutions were manifestly good enough. It asks too much to expect packages to foresee *and* accommodate something like a wholesale shift of utilities from /bin to /usr/bin in addition to the myriad other issues they must contend with, and especially in light of the historic stability of core utility paths. For non-shell programs, the decision to rely on shell utility at runtime is already predicated on expediency.

Regarding the issue with relying on $PATH--it's far more reasonable to rely on a sane $PATH at compile-time than runtime. The reasons are self-evident.

FWIW, Ubuntu's r-base package's configure.ac uses `AC_PATH_PROGS(SED, sed, /bin/sed, [/usr/xpg4/bin:$PATH])`. We could nit-pick that endlessly, but why bother.

Unexpected fallout from /usr merge in Debian

Cyberax — Wed, 05 Dec 2018 22:17:38 +0000

No and no. But there's already a path for migration. Probably not for this release though.