Various notes on /usr unification

By Jonathan Corbet
February 28, 2012

"/usr unification" (or simply "usrmove") is the idea of moving the contents of /bin, /lib, and related root-level directories into their equivalents under /usr. That this scheme is controversial can be seen from that fact that, when LWN pointed to a document describing the motivations behind the move, a thread of well over 250 comments resulted. One would think, from the volume of comments on LWN and elsewhere, that this change is a threat to the very nature of Linux. Either that, or it has inspired one of the biggest bikeshedding exercises in some time.

One of the "advantages" of running a Rawhide system is that one gets to try out the new toys ahead of the crowd. Said toys also have a certain tendency to arrive at random and inopportune times. In the case of the /usr move, most Rawhide users got little or no warning before updates simply failed to work. The Rawhide repository, it seems, had suddenly been populated with packages requiring a system where the /usr move had already been done. Actually causing this move to happen was left as an exercise for the system administrator.

As it happens, this is one of the more controversial aspects of the /usr move in the Fedora community. Fedora policy has always said that using the yum tool to update an installation to a newer release of the distribution is not supported; one is supposed to use a CD or the preupgrade tool for that purpose. But, in reality, just using yum tends to work; the prospect of it not working to get to Fedora 17 has caused some grumpiness in the Fedora user community. If this limitation remains in the Fedora 17 release, expect to hear some complaints; users don't like to have their ponies taken away, even if nobody had ever said they could have a pony in the first place.

For Rawhide users, it is possible to move past the usrmove flag day by following this set of instructions. The process involves editing the grub configuration files, installing a special initramfs image, then rebooting with a special option that causes the actual thrashing of the system to be done from the initramfs before the real root is mounted. Your editor, setting out along this path with a well backed-up system, was reminded of doing the manual transition from the a.out to the ELF executable format on a Slackware system many years ago. In both cases, one had the sense of toying with fundamental forces with no guarantee of a non-catastrophic outcome.

In both cases, as it happens, things worked out just fine. Directories like /bin, /lib, /lib64, and /sbin are now symbolic links into /usr, and the system works just like it always did. No programs or scripts needed to be changed, custom kernels work as they always did, and so on. That will almost certainly be most users' experience of this transition - they will not even notice that it has happened. This is not a particularly surprising result; the practice of moving files and leaving a symbolic link in their place has a long history. Of course, somebody has probably patented it anyway.

Of course, an equally successful result for all systems depends on the Fedora developers creating a bombproof automatic mechanism for carrying out this transition prior to the Fedora 17 release. Hopefully something will have been learned from the GRUB 2 mess, where the Fedora 16 upgrade left a lot of bricked systems in its wake. If Fedora 17 creates similar messes on systems that, perhaps, are not set up quite as the Fedora developers expect, a lot of unhappiness will ensue.

As an aside, it is interesting to compare how Fedora is managing this transition with Debian's multiarch transition. Fedora has forced a flag day requiring the entire thing to happen at once; Debian, instead, has carefully avoided flag days in favor of a carefully-planned step-by-step change. One could argue that the Debian approach is better: it lets the transition "just happen" through normal package updates without the need for any special actions on the part of administrators or users. On the other hand, the multiarch transition has been a multi-year process and is still not complete; Fedora, instead, has pushed this change through in a small number of months.

Now that Fedora has made this jump and seems to be past the turning-back point, will other distributions follow suit? Some investigation suggests that Fedora will not be alone in this change:

There have been no official statements from Red Hat, naturally, but it seems likely that, given how much effort has been put into this change by Red Hat employees, RHEL7 will feature the unified directory organization. The RHEL clones, including Oracle's distribution and CentOS, will have little choice but to incorporate this change.
OpenSUSE has been discussing the idea for a while and seems to be receptive to it. There does not appear to be an official statement that openSUSE will make the /usr move, but there is a web page set up to track progress in that direction.
Debian has had a number of long discussion threads about usrmove; it will not surprise longtime Debian watchers that there is a variety of opinions on the issue. There is some concern about possibly breaking systems with strange setups, though the most often cited case - systems with /usr on a separate partition - has been thought about and is relatively easy to support. In the end, it may come down to making the change to keep life easier; as Marco d'Itri put it: "I am not really looking forward to keep reverting these changes in my package, and since Red Hat controls most Linux infrastructure now other packages will face the same problem."
Of course, Debian has some time to think about the problem. Nobody has suggested that usrmove could be added to the list of goals for the upcoming "wheezy" release, so any transition, even in the unstable tree, is likely to be at least a year away.
Ubuntu has been relatively quiet about the idea; what has been posted by Ubuntu developers suggests a lack of enthusiasm for the idea. But if Debian makes the change, it could be hard for Ubuntu to retain the current filesystem layout.

In short, it looks like, over time, the move toward keeping all binaries and libraries in /usr will spread through most Linux distributions.

Given that most users will likely not even notice the change, it is natural to wonder why the change is so controversial. Undoubtedly, some people dislike significant changes to how traditional Linux systems have been laid out. Possibly, some creatively-organized systems will not handle the transition well, leading to problems that must be fixed. Conceivably, the change strikes some people as useless churn with no real benefits - at least for them. And, almost certainly, it is a bikeshed-type issue that is easy to have an opinion on and complain about.

The truth of the matter is that it appears to be a reasonably well-thought-out change driven by some plausible rationales. Filesystem hierarchies are not set in stone; there are times when it makes sense to clean things up. The usrmove operation yields a filesystem that is easier to understand, easier to manage, and more consistent in its organization. Having been through the transition, your editor thinks it is probably worth the trouble.

Various notes on /usr unification

Posted Feb 28, 2012 0:51 UTC (Tue) by tetromino (guest, #33846) [Link] (14 responses)

For anyone interested in Gentoo's plans, here is the epic gentoo-dev thread on the topic: http://thread.gmane.org/gmane.linux.gentoo.devel/74464.

The overall conclusion as I understood it is that there will be no flag day, individual packages will be migrated one by one from /bin to /usr/bin if and when forced to do so by upstream changes, and some sort of an initrd solution will be developed so that the people with a separate /usr partition would still be able to boot.

Various notes on /usr unification

Posted Feb 28, 2012 7:59 UTC (Tue) by jackb (guest, #41909) [Link] (13 responses)

I think a lot of the resistance comes from the fact that this change will make initramfs needed for more setups and there's a lot of hate for initramfs in the Gentoo community. I was in the same camp for a long time too before I started using whole disk encryption and thus needed to have one.

A lot of this resistance could be overcome tools like Dracut had better documentation, not just command line options but a step-by-step explanation of how it constructs the initramfs and what happens between when the kernel starts executing it and it hands over control to /sbin/init. Right now it's just a black box that works most of the time but when it fails those failures tend to be inexplicable.

I'm guessing the documentation that does exist was written based on a binary distribution paradigm where most users boot a pre-compiled kernel so the users for the most part don't need to know how dracut works internally to make sure their system will boot.

Various notes on /usr unification

Posted Feb 28, 2012 11:04 UTC (Tue) by smurf (subscriber, #17840) [Link] (11 responses)

The problem with an existing seperate (possibly readonly) /usr partition can be mitigated by swapping things around so that you have a separate (read-write, obviously) /var partition instead.

The only remaining directory that's unlikely to be a mount point is /etc; symlinking the two or three files that still need to be read-write off into /var/lib land is left as an exercise to the reader.

Various notes on /usr unification

Posted Feb 28, 2012 13:27 UTC (Tue) by jwakely (subscriber, #60262) [Link]

But having a separate /var partition broke upgrading to F16, because apparently noone does that so it didn't get tested (oops, turns out lots of people do that.)

Various notes on /usr unification

Posted Feb 28, 2012 20:17 UTC (Tue) by smcv (subscriber, #53363) [Link] (9 responses)

> The only remaining directory that's unlikely to be a mount point is /etc

/etc has to be on the root filesystem (or possibly be mounted by an initramfs if you use one): it contains /etc/fstab, which is required to mount the rest.

Various notes on /usr unification

Posted Feb 29, 2012 0:09 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (8 responses)

Unless you move /root (how could we forget about that?) to /var/root first. At that point, /etc would be the only non-mount point under /, so you could convert / into a virtual filesystem that just contained a bunch of mount points, with /etc being one of those mounts.

Various notes on /usr unification

Posted Feb 29, 2012 8:14 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

Just move /root to /home where it belongs and be done with it.

Root will still be able to login even without their home directory in emergencies.

Various notes on /usr unification

Posted Feb 29, 2012 15:56 UTC (Wed) by jengelh (guest, #33263) [Link]

>Just move /root to /home where it belongs and be done with it.

The point of /root not being in /home is even more volatile (as in: potentially not available due to whatever outage) than /usr.

Various notes on /usr unification

Posted Mar 1, 2012 12:38 UTC (Thu) by smurf (subscriber, #17840) [Link] (5 responses)

Not always, no.
Some login methods require files in $HOME. sshd with authorized_keys is only one of them. Not every system has a console, and even if it has one I don't want to force somebody to go to the data center and plug something magic into some blade system just because a reboot has managed not to mount $HOME.
(Which might as well be on NFS. To require a working NFS for root login is stupid.)

Various notes on /usr unification

Posted Mar 1, 2012 12:56 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

Well, in this case /root might be necessary. But it still strikes me as an extra top-level directory which is hardly ever needed now.

Maybe it could be a job for union mounts (once Valerie Aurora gets them mainlined).

Various notes on /usr unification

Posted Mar 3, 2012 17:00 UTC (Sat) by dpquigl (guest, #52852) [Link] (3 responses)

Has there been any progress on union mounts in the last year? Last I saw the conversation ended with the proposal dead in the water. I don't believe they ever determined who's responsibility it was to do culling of duplicate entries (the kernel, or glibc). Also the FUSE guy chimed in and claimed all the work was unnecessary and he could just do it in fuse.

Various notes on /usr unification

Posted Mar 3, 2012 22:07 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

FUSE union mounts actually work, but way too slow for real use.

And no, I'm not aware of any progress in that area.

Various notes on /usr unification

Posted Mar 4, 2012 18:02 UTC (Sun) by dpquigl (guest, #52852) [Link] (1 responses)

If you want to go with file system approaches unionfs and aufs have been around long before the FUSE version of unionfs. How full featured is the FUSE one? From what I understand everyone has been focusing on the most common case which is one RW branch and one RO branch. They don't want to do an arbitrary number of branches.

Various notes on /usr unification

Posted Mar 4, 2012 22:36 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

As I understand, unionfs in FUSE is pretty complete.

However, FUSE itself is not ready at all for high-performance filesystems. It's fine for things like sshfs over WAN but totally sucks for local filesystems.

Various notes on /usr unification

Posted Mar 8, 2012 9:48 UTC (Thu) by slashdot (guest, #22014) [Link]

Just use a custom init that mounts /usr and then execs the real init.

Various notes on /usr unification

Posted Feb 28, 2012 2:00 UTC (Tue) by russell (guest, #10458) [Link] (12 responses)

There is bigger picture with this kind of churn. Obsolete knowledge, processes, scripts,... This eventually leads to these things having to updated and re-learnt. Raising the costs (for little benefit) of supporting a linux environment.

Various notes on /usr unification

Posted Feb 28, 2012 8:59 UTC (Tue) by niner (subscriber, #26151) [Link]

On the other hand it will lower future costs by having less to learn. Two instead of four directories with binaries for example is definitely a step in the right direction.

Various notes on /usr unification

Posted Feb 28, 2012 14:32 UTC (Tue) by SEJeff (guest, #51588) [Link] (9 responses)

Am I the only one who uses PATH here? Sure, hardcoded stuff will change, but thats exactly the beauty of PATH, not worrying about that.

Various notes on /usr unification

Posted Feb 28, 2012 14:38 UTC (Tue) by mpr22 (subscriber, #60784) [Link] (8 responses)

ISTR there are definitely contexts where you either shouldn't be trusting PATH at all, or should be forcing it to a known value yourself. (#! lines are one of them, IIRC.)

Various notes on /usr unification

Posted Feb 28, 2012 14:41 UTC (Tue) by SEJeff (guest, #51588) [Link] (4 responses)

Very true, but that also depends on the context. For virtualenv + python as a general rule, it is best to do:
#!/usr/bin/env python

Something that isn't pointed out is that people who aren't ok with this move don't have to use Fedora / a systemd distro moving to this model. They could fork fedora or build their own distro. It isn't as though distributors are forcing a free distro down your throat :)

Various notes on /usr unification

Posted Feb 28, 2012 17:43 UTC (Tue) by drag (guest, #31333) [Link] (2 responses)

anything other then "#!/usr/bin/env python" is essentially broken, unless your specifying a specific python version, in which case it should be "#!/usr/bin/env python2.8" or whatever.

Various notes on /usr unification

Posted Feb 28, 2012 20:39 UTC (Tue) by lindi (subscriber, #53135) [Link] (1 responses)

http://www.debian.org/doc/packaging-manuals/python-policy...

seems to give quite opposite advice at least for official debian packages:

"Maintainers should not override the Debian Python interpreter using /usr/bin/env python or /usr/bin/env pythonX.Y. This is not advisable as it bypasses Debian's dependency checking and makes the package vulnerable to incomplete local installations of python."

Various notes on /usr unification

Posted Feb 29, 2012 16:10 UTC (Wed) by cortana (subscriber, #24596) [Link]

That document is aimed at Debian package maintainers. Stuff shipped by Debian is supposed to to use Debian's own Python interpreter, hence hacks like running python via env are not permitted.

Various notes on /usr unification

Posted Feb 28, 2012 20:43 UTC (Tue) by samroberts (subscriber, #46749) [Link]

I think env itself is /bin/env instead of /usr/bin/env on some unixen, though it might be in /usr/bin for all linux distros, which makes it the kind of thing the unification might help.

Various notes on /usr unification

Posted Mar 5, 2012 9:53 UTC (Mon) by jschrod (subscriber, #1646) [Link] (2 responses)

But with /bin being a symlink to /usr/bin this would actually work, for once.

Not like now, where you can't write a bash script for both Linux (/bin/bash) and FreeBSD (/usr/bin/bash) without using #!/usr/bin/env. Then it could just be /usr/bin/bash and we would be done. Just as in Solaris, since many years.

Various notes on /usr unification

Posted Mar 5, 2012 10:55 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

I never understood why can't we just write:
>#!bash
or
>#!env bash

instead of using absolute paths.

Various notes on /usr unification

Posted Mar 5, 2012 12:12 UTC (Mon) by jwakely (subscriber, #60262) [Link]

You can, but it means the same as #!./bash so probably doesn't do what you wanted.

Various notes on /usr unification

Posted Mar 1, 2012 8:32 UTC (Thu) by AndreE (guest, #60148) [Link]

Yes, specific knowledge becomes obsolete because the entire system is more predictable. That is a good thing.

I don't think you are going to convince anyone that having to learn the arbitrary determinations of binary and library location (that differ from distro to distro) ,is a better state of affairs than knowing everything will be found under /usr

Various notes on /usr unification

Posted Feb 28, 2012 3:21 UTC (Tue) by pj (subscriber, #4506) [Link] (26 responses)

A bit offtopic, but:
> Fedora policy has always said that using the yum tool to update an installation to a newer release of the distribution is not supported; one is supposed to use a CD or the preupgrade tool for that purpose.

...is why I've run debian continuously since 1995 or thereabouts. And when I say continuously, I mean that one system image has survived that long (module restores from backup after failed hardware). And been upgraded (remotely, via ssh or (back in the day) rsh, in most cases!) for that long.

I can't believe the haberdashy distros don't consider this a feature.

Various notes on /usr unification

Posted Feb 28, 2012 8:32 UTC (Tue) by michich (guest, #17902) [Link] (8 responses)

In practice yum upgrades work in Fedora. There are users who upgraded all the way through from Fedora Core 1. One just has to read http://fedoraproject.org/wiki/YumUpgradeFaq to learn about the few version-specific gotchas that may have to be handled manually. In Debian you should read upgrade instructions as well (http://www.debian.org/releases/squeeze/amd64/release-note...).

Various notes on /usr unification

Posted Feb 28, 2012 17:53 UTC (Tue) by jspaleta (subscriber, #50639) [Link] (6 responses)

Indeed,
With enough care and attentiveness most if not all gotchas for any distribution can be worked around. Based on the information in Chapter 4 and Chapter 5 in that Debian document you reference I wouldn't really call the live Debian upgrade a fire and forget automated process either. You really do need to be aware of that document before you attempt it.

-jef

Various notes on /usr unification

Posted Feb 28, 2012 23:36 UTC (Tue) by juliank (guest, #45896) [Link] (5 responses)

That's just official release notes. In practice you change your sources.list to point to the new distro, run apt-get update, apt-get upgrade, apt-get dist-upgrade, and reboot. Or at least my upgrade from lenny to squeeze worked like this IIRC.

And if you're running unstable, things are much more easier. You just upgrade once in a while and be happy.

Various notes on /usr unification

Posted Feb 28, 2012 23:57 UTC (Tue) by jspaleta (subscriber, #50639) [Link] (3 responses)

Wow.

Uhm okay. I guess reports like this are just fantasy right?
http://bugs.debian.org/cgi-bin/pkgreport.cgi?package=upgr...

The discussion for this bug:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=614159
relies heavily on pointing to the release notes on how to avoid the problems encountered.

And I particularly like this upgrade bug
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=598068
"Unfortunately there's no safe solution for it"

I will standby my previous statements as to anyone who thinks that live upgrading is a fire and forget process on any linux distribution. It is not..especially as the use case in question looks more and more like a median end-user usage case(with median end-users as admins) and less like a production server (with knowledgable admins at the helm making sure they do the necessary pre-upgrade prep)

-jef

Various notes on /usr unification

Posted Feb 29, 2012 3:28 UTC (Wed) by foom (subscriber, #14868) [Link]

Shrug, sure, there are edge cases and bugs you could run into. But everyone I know who runs debian does do live upgrades, and it works. The parent comment's "in practice" is, in practice, what you actually do. (And speaking for myself, I don't bother to read the release notes before doing so.)

Now, I don't actually know how things work out in practice for Fedora, but the upgrade page says "Version updates without using anaconda - such as the yum method described here - is unsupported and not recommended!" and "Although upgrades with yum do work, they are not explicitly tested as part of the release process by the Fedora QA and are not documented in the Fedora installation guide."

Maybe that warning is wrong, but it really makes it *sound* like running yum upgrade is something that is not recommended and that is untested. While, in contrast, apt-get on a running system *is* the tested, documented, and supported upgrade mechanism for Debian. They really try to make sure it works properly for most users.

Various notes on /usr unification

Posted Mar 5, 2012 22:41 UTC (Mon) by man_ls (guest, #15091) [Link]

Sounds like a smokescreen to me. All we are saying is that live upgrading a Debian system is supported, while on Fedora it is not. Not that it is flawless.

Various notes on /usr unification

Posted Mar 8, 2012 9:28 UTC (Thu) by kragil (guest, #34373) [Link]

Stupid Debian-bashing (from a person who probably does not use it)
In place upgrades generally work just great and are supported on Debian. Sure they are not 100% perfect for everyone, but they have been for me.

Your Canonical-bashing is a lot better than this.

Various notes on /usr unification

Posted Feb 29, 2012 11:57 UTC (Wed) by etiennez (guest, #53056) [Link]

Seriously, read the release notes before attempting upgrades of your Debian system. I never broke an upgrade because I always do, and a lot of my friends broke their system (not behind repair but still) because they didn't and it seems everybody on the Internet repeat that edit sources.list + apt-get update && apt-get dist-upgrade just works (except when it doesn't).

Important informations I can remember reading in release notes for example: apt-get/aptitude incompatibility that could cause half your system to be uninstalled, recommended upgrade tool (aptitude vs apt-get) for best result, renaming of hard drive device (hd* => sd*), + lots of useful advices to avoid breaking things (packages you should probably upgrade first, etc).

The squeeze upgrade was quite trouble free IIRC, but it is not always the case.

Various notes on /usr unification

Posted Feb 29, 2012 4:47 UTC (Wed) by jmalcolm (subscriber, #8876) [Link]

I have a system that has been upgraded from Red Hat Linux 9 to Scientific Linux 6.2.

Red Hat Linux 9 -> Fedora Core 1 -> FC2 -> FC3 -> FC4 -> FC5 -> Centos 5 -> Scientific Linux 6.2

I must confess though that there are some old 'mailman' mailing lists on that thing that I am afraid to mess with. The way 'mailman' is configured changed somewhere along the way.

Various notes on /usr unification

Posted Feb 28, 2012 8:52 UTC (Tue) by mjthayer (guest, #39183) [Link] (3 responses)

pj wrote:
> A bit offtopic, but:
>> Fedora policy has always said that using the yum tool to update an installation to a newer release of the distribution is not supported; one is supposed to use a CD or the preupgrade tool for that purpose.

> ...is why I've run debian continuously since 1995 or thereabouts.
[...]
As a long-ish term Ubuntu user I am in the same situation (and since I don't tend to fiddle more than necessary it actually works for me). I'm interested to hear from someone in the other camp how much work it ends up being reinstalling your system on every release. I can't help thinking (warning, mild troll coming up) that it should be easy but is probably made difficult by the way that Linux systems and the software running on top of them aren't more cleanly separated in popular distributions.

Upgrading

Posted Feb 28, 2012 14:20 UTC (Tue) by corbet (editor, #1) [Link] (1 responses)

Note that you don't have to reinstall to get to a new release. Well, I guess RHEL doesn't support major release upgrades, but it's the exception. Fedora does upgrades just fine (modulo the occasional difficulty), but you're supposed to running anaconda from a separate root to do it in the supported way.

Upgrading

Posted Feb 28, 2012 21:55 UTC (Tue) by rvfh (guest, #31018) [Link]

And it's not like upgrading [K]Ubuntu was always a breeze... Almost every time I have to fix something - though I am probably no typical [Ubuntu] user either.

Various notes on /usr unification

Posted Feb 29, 2012 5:01 UTC (Wed) by jmalcolm (subscriber, #8876) [Link]

I have run RPM systems including Fedora for many years and have never used the install disk to upgrade. I have always used RPM (via Yum, up2date, URPMI, Red Carpet, SMART, or others). It has always gone well.

Well, I have run into package incompatibilities here and there. I would say that knowing how to 'temporarily' break the RPM database has probably been required a couple times along the way. For example, you might have to force the installation or removal of a package that is blocking other change. Or, a newer package might want to overwrite a file that still falls under the dominion of an older package.

My memory of when these have been required is a bit cloudy as I have also done dumb stuff like go from Fedora to RHEL or vice versa. I even went from Mandrake to Fedora once.

Other than a few Scientific Linux and CentOS machines I am mostly off RPM based systems these days though. Having to source from multiple (inevitably incompatible) RPM repositories has always caused more trouble for me than moving from version to version of the OS. I have found it necessary far less often to source foreign packages (or install unmanaged applications) when using Ubuntu or Debian.

I have run into similar issues converting an Ubuntu machine to Debian or back as well. The machine I am typing on was originally an Ubuntu machine, was migrated to Linux Mint, and is now Debian Unstable. The whole migration was nothing more than overwriting sources.list and using 'apt-get' each time.

Various notes on /usr unification

Posted Feb 28, 2012 9:07 UTC (Tue) by niner (subscriber, #26151) [Link]

Please note that using a CD or preupgrade tool does not mean a complete reinstallation. It just means that the upgrade should not be done from a running system but by booting a known stable and non-changing system to do the upgrade. It's not as convenient, but still not that bad a situation. At least it allows major changes to the system without endangering the update system itself. For example it's kinda bad if the update removes an obsolete compression library before installing its replacement leaving the system unable to read the package containing said replacement.

It's certainly not impossible to avoid such problems in an on the fly upgrade, but it takes quite some effort. Effort which may be seen as better invested in other features like a solid unattended installation or upgrade process.

Same was true for openSUSE btw. On the fly upgrades tended to work but have only been recommended for the last couple of versions.

Various notes on /usr unification

Posted Feb 28, 2012 12:20 UTC (Tue) by Sho (subscriber, #8956) [Link] (11 responses)

Actually I consider preupgrade a much better way to go. To recap, preupgrade works by downloading a bootable installer image, performing a dry-run and then adding the installer as the default option to the bootloader configuration. Then it reboots into the installer, performs the upgrade and reboots again.

The reason this is better is that the system that is being upgraded is offline during the upgrade. A lot of software doesn't support being upgraded while it's running Things can go subtly wrong when files get swapped out underneath running instances, or config files get rewritten by upgrade processes to fit the new version and then the old version overwrites the changes when it's flushing state to disk while quitting. "Being replaced while running" is a scenario very few projects test for.

Various notes on /usr unification

Posted Feb 28, 2012 13:06 UTC (Tue) by khim (subscriber, #9252) [Link] (2 responses)

"Being replaced while running" is a scenario very few projects test for.

Every rule has an exception. Google Chrome (and Chromium) are quite explicitly developed for such usage. Of course "silent upgrades" are their core design decision thus they have to support them. Other projects... are less forgiving. Our Linux support team likes to push upgrades every time you connect [loaned] laptop to a corp network. Firefox often becomes unusable as a result :-( Usually restart fixes it, but once or twice (in last five years) it corrupted profile beyond repair.

Various notes on /usr unification

Posted Feb 28, 2012 13:25 UTC (Tue) by Sho (subscriber, #8956) [Link] (1 responses)

One large contingent of software that potentially has problems with in-place upgrades is any KDE application. This is only very, very rarely a real problem in practice, though, so there's no reason to panic. Still, it's good to be aware of it:

Basically, when code evolves, the format of its configuration file will sometimes change. One way to handle existing user config files using the old format is to retain support for reading the old format, and then write out the new format later, and that's indeed what apps often do. However, the KDE platform also contains an alternate mechanism called "kconf_update". With kconf_update, the application developer supplies scripts that will be run to transform a config file from one format into another (either in a simple DSL or something free-form specifying the interpreter), and a section in the config file is used to keep track of what scripts have already been run.

The problem lies in the "will be run" part. KDE sessions run a process called kded, the KDE daemon, which plays host to lightweight plugins that perform a variety of tasks, from holding binary caches of things like the app db to disk space warnings, etc. One of those plugins also watches directories for newly-installed kconf_update scripts and runs them immediately as it discovers them.

However, KDE apps also tend to flush their config out to disk while quitting.

You can see where this goes:
1. The package manager installs a new version of $app that ships a kconf_update script while the user is logged into KDE and has $app running.
2. The kconf_update plugin in kded notices the new script and runs it, reformatting $app's config file.
3. The user quits the currently-running old version, which flushes its config out to disk as it shuts down, potentially overwriting changes made by the script.
4. The user now runs the new version which may now run into problems reading its config file.

The reason this is very rarely a problem in practice is because config file format changes are rare to begin with, and the changes that do happen often include changing of the config key names, so the number of actual conflicts that occur is very low. But it's a possibility.

And yes, this is definitely a case of bad design.

Various notes on /usr unification

Posted Feb 29, 2012 11:53 UTC (Wed) by Lennie (subscriber, #49641) [Link]

I would change the path of the config file and only look at the new path first in the newer version of the program.

Because otherwise things will just go wrong.

Especially if you have several machines with /home on NFS or whatever which don't get upgraded at the exact same time.

Various notes on /usr unification

Posted Feb 28, 2012 13:08 UTC (Tue) by drago01 (subscriber, #50715) [Link] (6 responses)

Yeah I don't see preupgrade as worse but actually as a better solution.

Updating the whole OS while it is running sounds to error prone to me (it can work but it is kinda fragile).

Various notes on /usr unification

Posted Feb 28, 2012 15:17 UTC (Tue) by sorpigal (guest, #36106) [Link] (5 responses)

It may sound error prone but in practice I've had incredibly few issues, not with servers or workstations. I do make sure I shut down outside access for servers to prevent user access during an upgrade. Sometimes you get warnings like "You should shut down X first," which I ignore unless a video driver is involved, and even doing this have had no more than a small handful of issues in the last decade.

It seems like we accidentally managed to make this work nearly without flaw.

Various notes on /usr unification

Posted Feb 28, 2012 15:21 UTC (Tue) by Sho (subscriber, #8956) [Link] (4 responses)

Thing is, though, you're power user who is equipped to deal with the breakage in the event that it does arise, so it's a calculated risk for you. A distro may legitimately decide to care about serving users who aren't equipped for it. And in the face of the possibility of breakage, an upgrade process that tries to avoid it sounds like a good idea.

Various notes on /usr unification

Posted Feb 28, 2012 23:37 UTC (Tue) by cmccabe (guest, #60281) [Link] (2 responses)

If things get broken by the upgrade, users would simply reboot. And if the issue was the version skew you're worried about, that would fix it.

Does knowing how to press the power button make you a "power user"? If so, I weep for the future of computing.

Various notes on /usr unification

Posted Feb 29, 2012 0:57 UTC (Wed) by mgedmin (subscriber, #34497) [Link] (1 responses)

If the upgrade crashes halfway through, there is a chance the machine won't boot.

A power user might fix it (dpkg --configure -a; apt-get install -f; apt-get dist-upgrade).

Various notes on /usr unification

Posted Mar 4, 2012 6:25 UTC (Sun) by cmccabe (guest, #60281) [Link]

And if a Boeing 747 crash-lands on top of the computer, that would probably be bad too. But how is it related to what we were talking about?

Just to be clear, the issue we were talking about was version skew in libraries causing applications to misbehave during the upgrade.

Various notes on /usr unification

Posted Feb 29, 2012 13:21 UTC (Wed) by sorpigal (guest, #36106) [Link]

You're not wrong, but I find myself wondering whether, if we've come this far without really trying, it might not be worthwhile to test for this usage and make the "unequipped" users able to do it in relative safety.

Various notes on /usr unification

Posted Mar 4, 2012 17:45 UTC (Sun) by dirtyepic (guest, #30178) [Link]

> A lot of software doesn't support being upgraded while it's running

I'm going to have to call bullshit on that one. See every rolling distro ever.

> or config files get rewritten by upgrade processes to fit the new version
> and then the old version overwrites the changes when it's flushing state
> to disk while quitting

Well, first of all, don't overwrite config files. Store the new config somewhere else and give the user an interactive diff tool. Second, any service that writes state to a file that would get overwritten on a reinstall is broken. Any service that can't recognize state information from its previous version is broken.

> "Being replaced while running" is a scenario very few projects test for.

Because it rarely causes problems. The majority of Gentoo systems are upgraded continually while in use. The biggest problem I had after upgrading and recompiling every package installed on a system that hadn't been updated in six months, which included major gcc and glibc updates, was Firefox wouldn't open links sent to it from external programs until I restarted it.

Various notes on /usr unification

Posted Feb 28, 2012 3:30 UTC (Tue) by mrons (subscriber, #1751) [Link]

Rusty's blog "Why Everyone Must Oppose The Merging of /usr and /" pretty much gets to the main objections of this move.

http://rusty.ozlabs.org/?p=236

Various notes on /usr unification

Posted Feb 28, 2012 3:51 UTC (Tue) by bats999 (guest, #70285) [Link] (18 responses)

Actually, a large portion of that 250+-comment thread had little to do with /usrmove itself.

On a somewhat related note, I'd like to make a motion to retire the term "bikeshedding".

Various notes on /usr unification

Posted Feb 28, 2012 6:01 UTC (Tue) by krakensden (subscriber, #72039) [Link] (2 responses)

The danger of retiring the term "bikeshedding" is the inevitable bikeshedding about what to replace it with.

Various notes on /usr unification

Posted Feb 28, 2012 6:07 UTC (Tue) by felixfix (subscriber, #242) [Link]

How about "rearranging the deck chairs on the Titanic"?

Various notes on /usr unification

Posted Feb 28, 2012 16:25 UTC (Tue) by JEFFREY (guest, #79095) [Link]

<--- I think this thread has jumpped the shark

Various notes on /usr unification

Posted Feb 28, 2012 7:10 UTC (Tue) by peter-b (guest, #66996) [Link] (12 responses)

> On a somewhat related note, I'd like to make a motion to retire the term "bikeshedding".

Why?

Various notes on /usr unification

Posted Feb 28, 2012 17:37 UTC (Tue) by drag (guest, #31333) [Link]

Because it's the wrong color.

Various notes on /usr unification

Posted Feb 28, 2012 18:21 UTC (Tue) by pboddie (guest, #50784) [Link] (10 responses)

Because it is tiresome, appearing in just about every discussion on the Internet, often accompanied by the equally tiresome accusations of "ad hominem" and "bad journalism" that are also due for retirement.

Various notes on /usr unification

Posted Feb 28, 2012 19:32 UTC (Tue) by josh (subscriber, #17465) [Link] (5 responses)

The terms won't go away as long as the issues they describe continue to exist.

Various notes on /usr unification

Posted Feb 29, 2012 12:11 UTC (Wed) by pboddie (guest, #50784) [Link] (4 responses)

Wolves may still roam the forest, but that doesn't mean that the shepherd boy needs to cry their name every time he sees the sheepdog.

Various notes on /usr unification

Posted Feb 29, 2012 12:21 UTC (Wed) by mpr22 (subscriber, #60784) [Link] (2 responses)

From where I'm sitting, moving to "retire the term 'bikeshedding'" sounds like a request to remove the word from use. All that could be achieved thereby is to cause people to concoct a new word to express the same sentiment (see also the history of words like 'moron', 'idiot', etc). That process would, in turn, probably result in a great deal of bikeshedding and miscommunication until a new status quo was achieved... whereupon someone who has then the same motives that you do now would doubtless demand that we stop using the new word for bikeshedding.

Various notes on /usr unification

Posted Feb 29, 2012 19:45 UTC (Wed) by jospoortvliet (guest, #33164) [Link] (1 responses)

... which is exactly what has happened a few times already with words referring to immigrants in NL. A word is used, it becomes 'a dirty word', the political correct start using another word, that word becomes similarly emotionally laden, new word gets picked, etcetera. Doesn't change a thing...

Various notes on /usr unification

Posted Mar 1, 2012 14:34 UTC (Thu) by nix (subscriber, #2304) [Link]

This is called the euphemism treadmill.

Various notes on /usr unification

Posted Mar 1, 2012 8:45 UTC (Thu) by AndreE (guest, #60148) [Link]

Should we get rid of the word "wolves" though?

Various notes on /usr unification

Posted Feb 29, 2012 9:02 UTC (Wed) by marcH (subscriber, #57642) [Link] (3 responses)

> Because it is tiresome, appearing in just about every discussion on the Internet, often accompanied by the equally tiresome accusations of "ad hominem" and "bad journalism" that are also due for retirement.

Wow! So, bikeshedding, ad hominems and bad journalism have all vanished from the Internet? This should make the headlines!

Various notes on /usr unification

Posted Feb 29, 2012 18:43 UTC (Wed) by jond (subscriber, #37669) [Link] (2 responses)

That's a straw man argument.

(sorry.)

Various notes on /usr unification

Posted Feb 29, 2012 22:56 UTC (Wed) by marcH (subscriber, #57642) [Link] (1 responses)

Please explain the meaning of "motion to retire" and "due for retirement".

One of the easiest ways to troll: fight exaggeration with exaggeration. Sorry I fed it.

Various notes on /usr unification

Posted Mar 1, 2012 18:12 UTC (Thu) by pboddie (guest, #50784) [Link]

I actually forgot to suggest accusations of trolling, along with "correlation does not imply causation" and "the plural of anecdote is not data". Of course such terms and phrases describe actual phenomena, but they also seem to take the role of the panic button in any discussion.

Various notes on /usr unification

Posted Feb 29, 2012 23:01 UTC (Wed) by marcH (subscriber, #57642) [Link]

> On a somewhat related note, I'd like to make a motion to retire the term "bikeshedding".

The Internet IS bikeshedding (once you remove porn).

Various notes on /usr unification

Posted Mar 1, 2012 16:27 UTC (Thu) by drago01 (subscriber, #50715) [Link]

> On a somewhat related note, I'd like to make a motion to retire the term "bikeshedding".

The term at least has a meaning. I'd rather retire the term "power user" ;)

Various notes on /usr unification

Posted Feb 28, 2012 5:19 UTC (Tue) by pr1268 (guest, #24648) [Link] (1 responses)

As a Slackware (13.37) user, I'm curious how much impact usrmove would have on my system. Refer to:

me@mybox:~$ which ls
/usr/bin/ls
me@mybox:~$ file $(which ls)
/usr/bin/ls: symbolic link to `../../bin/ls'
me@mybox:~$ file /bin/ls
/bin/ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),  dynamically linked (uses shared libs), stripped

Apparently, Slackware places a lot of the "mission-critical" utilities in /bin and symlinks to them from /usr/bin¹. Now, It'd be easy for me to move everything in /bin to /usr/bin (removing the symlinks first, of course), but then I'm wondering how much other stuff would break...

FWIW, my /usr is not on a separate partition. Accordingly, I'm guessing not much would break, and if it did, I could "restore" /bin via a symlink (as the article said other distros are doing). Might warrant some hacking time (after I back everything up first)...

> the practice of moving files and leaving a symbolic link in their place has a long history. Of course, somebody has probably patented it anyway.

On the one hand, our editor is known for his wry sense of humor. On the other hand, I get a little nervous sometimes when I think he might be closer to the truth with that statement than we'd like. :-\

¹ I'm assuming this makes it easier for the shell to find executables as Slackware's profile scripts put /usr/bin in the $PATH earlier than /bin.

Various notes on /usr unification

Posted Feb 28, 2012 6:23 UTC (Tue) by josh (subscriber, #17465) [Link]

No, symlinks like that (from /usr/bin/$foo to /bin/$foo in Slackware's case, or from /bin to /usr/bin for the /usr merge) have nothing to do with $PATH search; they exist to avoid breaking hard-coded paths to programs. For instance, a program might expect to find "/bin/bash".

Various notes on /usr unification

Posted Feb 28, 2012 8:51 UTC (Tue) by pflykt (subscriber, #2757) [Link] (2 responses)

When we focus on /usr, we tend to keep forgetting the trend of moving the old role of / into initramfs. The role of / has been a minimal system that contains enough tools to initialize a "multi-user" system, including a /usr partition that was not readable by the boot loader. With more complex setups reaching beyond RAID 1 on the / partition, initramfs has come to our rescue.

I don't see any signs of getting rid of / anytime soon, it just moves over to initramfs instead. And when all necessary repair tools are usable from an initramfs shell, it will all look just too familiar like the way it was in 1992.

Various notes on /usr unification

Posted Feb 28, 2012 22:03 UTC (Tue) by rvfh (guest, #31018) [Link] (1 responses)

On Xstations, / would be the strict necessary software to boot, and /usr would be mounted from THE server.

That also lead some embedded systems to making / the core boot stuff that remains read-only (with proper links in /etc such as /etc/mtab) and /usr where the real stuff takes place and upgrades can occur.

Various notes on /usr unification

Posted Aug 19, 2021 9:17 UTC (Thu) by immibis (subscriber, #105511) [Link]

And now initramfs would be the strict necessary software to boot, and / would be mounted from THE server.

I actually have one system which has a real partition instead of an initramfs. It works fine. Apparently the only problem is that the tools to maintain it don't exist.

initramfs typically terminates by mounting something else over / using the pivot_root syscall. But there's no reason only an initramfs can do that! You are free to install a minimal system locally and then have its init set up the network and NFS and then pivot_root to NFS.

Various notes on /usr unification

Posted Feb 28, 2012 9:20 UTC (Tue) by cjwatson (subscriber, #7322) [Link] (7 responses)

Another example of flag day vs. gradual transition was the move to UTF-8 manual pages. Fedora did that by changing their man configuration and all packages shipping non-ASCII pages in a flag day; in Debian, I made man detect page encodings on the fly and then changed the packaging tools to produce UTF-8 pages by default, at which point there was no need to upload all packages shipping non-ASCII pages in one go.

Various notes on /usr unification

Posted Feb 28, 2012 11:21 UTC (Tue) by cesarb (subscriber, #6266) [Link] (5 responses)

Fedora also loves doing mass rebuilds, where *almost* every packages gets rebuilt. Fedora 15 had one of these (https://fedoraproject.org/wiki/Fedora_15_Mass_Rebuild); on Fedora 16 desktop, for instance, there are only 371 packages last built for other than Fedora 15 or 16, from a total of 25995 packages (these numbers are from a simple grep of a repoquery -a output). Fedora 17 apparently had yet another mass rebuild (https://fedoraproject.org/wiki/Fedora_17_Mass_Rebuild).

Various notes on /usr unification

Posted Feb 28, 2012 14:30 UTC (Tue) by pbonzini (subscriber, #60935) [Link] (1 responses)

Fedora mass rebuilds are done when the compiler is updated, mostly.

Mass rebuilds

Posted Mar 1, 2012 12:59 UTC (Thu) by smurf (subscriber, #17840) [Link]

Same for Debian, except that it's usually a single developer with too much free time and/or raw CPU power on their hands who decides to do their own personal mass rebuild – and file bugs about everything that fails to build.

Various notes on /usr unification

Posted Feb 28, 2012 14:44 UTC (Tue) by wookey (guest, #5501) [Link]

And the Fedora ARM dept has realised that they are going to need a lot of ARM build machinery if they wish to continue doing that (because it still takes a whole heap longer - although of course this improves over time).

Mass rebuilds are good for checking everything is still buildable (Debian regularly finds old stuff that doesn't actually rebuild anymore due to changes around it, because we do in fact do test mass rebuilds on x86 from trime to time), and it propogates toolchain improvements. But it does need much more build resource than incremental building. The more arches you support the harder it is to do.

Various notes on /usr unification

Posted Feb 28, 2012 14:48 UTC (Tue) by sorpigal (guest, #36106) [Link]

This is one of those key-but-unimportant distinctions between Debian users and Fedora users. Debian users like the way that Debian does things and Fedora users like the way that Fedora does things, that's why they use those systems, and tend to look askance at each other when they hear about the differences in the way the systems work. This is one of the reasons that it's often hard to explain to new users what exactly is so different between distributions when the packaged software is nearly identical.

Various notes on /usr unification

Posted Feb 28, 2012 23:43 UTC (Tue) by juliank (guest, #45896) [Link]

Debian does the same, but throws the binaries away and only keeps the logs of failed builds (in bug reports).

Various notes on /usr unification

Posted Feb 28, 2012 17:39 UTC (Tue) by jspaleta (subscriber, #50639) [Link]

Do you have an estimate on how long the gradual process took to complete for the case you bring up?

I think there is a truism that just can't be gotten around. The different approaches to distribution release timescale being used by Debian versus Fedora greatly influence decision making.

I think there is a lot of truth to the idea that Debian's "release when its ready" approach with much longer timescales between official releases of a stable offering is more aligned with gradual technology transitions.

I also think that the Fedora twice yearly release cycle timescale imposes constraints which weigh against gradual processes in favor of distinct coordinated transitions.

And I further believe that this is an entirely okay situation as Debian and Fedora as projects put the emphasis on very different priorities in their project focus. Hopefully the good new ideas will cross-pollinate. And the bad ideas will be allowed to quietly be allowed to taken outback and shot.

-jef

Various notes on /usr unification

Posted Feb 28, 2012 11:42 UTC (Tue) by dps (guest, #5725) [Link] (2 responses)

I personally prefer not to use an initial ram disc, so moving all binaries into /usr is a plain *bad* idea. I do keep most binaries my separate, read only, /usr partition but I need the tools to mount it on the root filesystem.

Keeping at least some tools allows me to upgrade those tools without re-installing grub and running of out of space in /boot, which is also a separate partition, which I usually mount read only.

I would like to be able to mount / read only too but that is more difficult.

Various notes on /usr unification

Posted Feb 28, 2012 13:01 UTC (Tue) by niner (subscriber, #26151) [Link]

Since we're talking about Linux here, this will always be possible. There's always LFS giving you extreme control over every minute detail.

Just don't expect every general purpose distribution to conform to your very special requirements.

Various notes on /usr unification

Posted Feb 28, 2012 17:57 UTC (Tue) by hmh (subscriber, #3838) [Link]

You're not looking at it with the proper mindset.

An initramfs enforces that the worst of the bling is kept away from your early boot.

Various notes on /usr unification

Posted Feb 28, 2012 14:11 UTC (Tue) by jzbiciak (guest, #5246) [Link] (3 responses)

I have to say, two things in this article caused me two completely different flashbacks.

[U]sers don't like to have their ponies taken away, even if nobody had ever said they could have a pony in the first place.

...flashed me back to this day on a different website. And then this:

Your editor . . . was reminded of doing the manual transition from the a.out to the ELF executable format on a Slackware system many years ago.

I still remember that. No more QMAGIC (or my favorite for very, very small programs, NMAGIC).

Ok... enough reminiscing. There's one question I do have, related to this:

Fedora has forced a flag day requiring the entire thing to happen at once; Debian, instead, has carefully avoided flag days in favor of a carefully-planned step-by-step change. One could argue that the Debian approach is better: it lets the transition "just happen" through normal package updates without the need for any special actions on the part of administrators or users.

The question that came to mind for me immediately was why package updating broke at all. Why couldn't all of these unable-to-update packages do their own piecemeal move? If a few related packages have to move as a group, give them a common dummy package as a dependency that does the group move first. Why break package updates and require this flag day?

Without the package-breaking flag day, I don't think it follows that this will become a lingering transition. You have a flag day for package standards, and the packages will all end up moving themselves.

I guess the downside is that now you're left with a /bin and /lib full of symlinks rather than a single symlink each for /bin and /lib. I'm no packaging whiz (far, far from it), but it seems like that could be cleaned up with another package that would depend on all the piecemeal moves. But, all the intermediate states do look messier. And, it doesn't account for any locally-installed software that probably should be in /usr/local/*. (*tsk* *tsk*!)

Ugh. Yeah, this does smell a bit of the ELF/a.out transition.

Various notes on /usr unification

Posted Feb 28, 2012 14:59 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

The largest reason for the flag day, AFAIK, is that RPM doesn't support turning a directory into a symlink or vice versa. Instead, the dracut module was used to do this safely before the system booted which allowed RPM to continue blissfully unaware of the change.

Various notes on /usr unification

Posted Feb 28, 2012 19:11 UTC (Tue) by hpa (guest, #48575) [Link] (1 responses)

Of course, those of us who remember the a.out -> ELF transition also remember that it was WAY smoother than the libc5 -> libc6 (ELF to ELF) transition...

Mercy

Posted Feb 28, 2012 19:18 UTC (Tue) by corbet (editor, #1) [Link]

Ouch, man, some of us have brutally suppressed those memories and don't welcome the reminder...

Various notes on /usr unification

Posted Feb 28, 2012 16:55 UTC (Tue) by dskoll (subscriber, #1630) [Link] (22 responses)

This is all wrong!!!! We should get rid of /usr and have top-level directories only, like /bin, /share, /lib and /man.

Shorter paths are better!! (No seriously, what's the point of even having /usr?)

(/me ducks...)

Various notes on /usr unification

Posted Feb 28, 2012 17:42 UTC (Tue) by drag (guest, #31333) [Link] (1 responses)

So I can go:

$ find /usr | grep whatamilookingforexactlynow

$ tar cz /usr > /tmp/lunix.tar.gz

$ cat /etc/fstab|grep /usr
nfs4.example.com/lunix/usr/ /usr nfs4 defaults 0 0

etc. etc. :)

Wondering why we keep /usr/bin instead of /bin too

Posted Feb 28, 2012 22:17 UTC (Tue) by rvfh (guest, #31018) [Link]

Do you mean that it makes it easier to have local, specific /etc /tmp /var and a global, generic, maybe NFS mounted /usr?

Various notes on /usr unification

Posted Feb 28, 2012 23:51 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link] (19 responses)

No seriously, what's the point of even having /usr?

The idea is that you should be able to have a separate partition for each different kind of data. It should be possible to keep read-only data (or data that is only supposed to be written by a sysadmin) on a separate partition from data that's frequently written, data that's specific to a particular machine separate from data that can be shared across multiple machines, and data that is volatile across a reboot separate from data that needs to be preserved across reboots. So the idea is that standard partitions are supposed to be:

/ Machine specific, read-only
/var Machine specific, read-write, stable across reboots
/tmp Machine specific, read-write, volatile across reboots
/usr Shared, read-only
/home Shared, read-write

Note that not all combinations make sense. It only makes sense to have a volatile partition that's read-write (how would a volatile read-only partition accumulate any files after being wiped?) and machine specific (whose reboot would a shared volatile partition be volatile across?). Other than that, though, you have all the possible combinations.

I think that part of the problem right now is that there's a legitimate disagreement about the function of /. Some people are big believers in the "separate partition for separate kinds of data" idea, and they want to move all binaries and libraries into /usr. Other people see the main function of / as a minimal system partition that can be used to bring the rest of the system up, and they want to keep essential binaries and libraries in /.

Various notes on /usr unification

Posted Feb 29, 2012 0:02 UTC (Wed) by dlang (guest, #313) [Link]

Also, while disks are a lot bigger today than in the past, there are still cases where drives are not large enough to store everything.

you may want your main system on a small SSD for fast reading, while data that is being archived (and seldom read like log files) are on a slower, but large SATA drive, and other data may need to be on a raid array for redundancy.

Various notes on /usr unification

Posted Feb 29, 2012 13:42 UTC (Wed) by sorpigal (guest, #36106) [Link] (2 responses)

I think this whole / -> /usr or /usr -> / discussion highlights the need to re-think the FHS more drastically. Clearly not everyone (like me, for example) is happy with moving / into /usr and I keep seeing complaints about use cases that break when you do this. I also see despair from some people who say that we're fooling ourselves if we pretend that there aren't programs that presume /usr is always there when they shouldn't.

Maybe it's time to lay out a new FHS, beginning by describing all of the uses we need to get out of it, and make one big painful move instead of this piecemeal nobody-is-happy stuff.

Q: If initramfs is the new "minimal root" and we have to maintain it anyway just to boot, can we maintain it in its own partition mounted on /init/ and re-do the boot process to use that path at all times whether it's a real disk or a ramfs?

Q: If /tmp and and /etc are local-only, but /usr is not, is there a way we can signal that in the FHS so that people can't be easily confused in the future?

I don't know, but it seems like fighting about what should go in /usr and whether anybody really needs to share it over NFS is idiotic and tangential to the real issue, which is how to solve problems people actually have. Nobody seriously cares where files are located as long as everyone can still do what they need to do.

It's a bit like Debian's multi-arch. There are a lot of ways to solve a lot of little problems but only a few that solve them in a big way for a long time.

Various notes on /usr unification

Posted Feb 29, 2012 20:22 UTC (Wed) by dlang (guest, #313) [Link] (1 responses)

while I do not dispute that there are valid use cases for initramfs, why should it be a requirement on all systems?

Various notes on /usr unification

Posted Mar 1, 2012 11:27 UTC (Thu) by sorpigal (guest, #36106) [Link]

I don't insist on one. If the issue is packaging effort it seems logical to take what is built for initramfs purposes and make that the "minimal root" stuff for people without an initramfs, just place it on disk as usual.

iff you accept /* -> /usr/, then why not instead /usr/ -> /*

Posted Mar 1, 2012 7:29 UTC (Thu) by filteredperception (guest, #5692) [Link] (5 responses)

/me ducks ahead of time, but...

While your answer to the question I was trudging through dozens of comments to get to (not worth a sentence in the article?) was quite long, I don't think it was sufficient. I think your answer was still within the debate of 'should the move happen'. But what I think the original ducking questioner was asking was-

If you are one of the people who believe in moving /* stuff to /usr/, then please explain why you specifically think that is better than the opposite of moving /usr/* stuff to /, as that would achieve shorter paths, and since, at least from a limited perspective, _once you are in the usrmove camp_, it seems hard (unless us duckers are missing something obvious) to explain what the point of /usr is at all.

iff you accept /* -> /usr/, then why not instead /usr/ -> /*

Posted Mar 1, 2012 8:00 UTC (Thu) by filteredperception (guest, #5692) [Link] (1 responses)

replying to self after reading parent explanation more closely. I guess I understand better the shared vs machine-specific aspect as you enumerated/described. Though given the linking of /bin(actual) to /usr/bin (as opposed to symlinks for each content file/binary), would mean that if I understand correctly, that now there _is no non-shared/machine-specific_ place to put a 'binary' (i.e. under /bin in the old-way/your-explanation). In which case our question remains valid. (I think, still ducking). Or rather, you could say- under /usr/local, in which case I would go back to, why not /local.

/local vs /usr/local

Posted Mar 1, 2012 18:25 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

I think something like /local for code that is specific to an individual machine makes sense. Of course, software that's specific to an individual machine is probably being handled outside the distribution, or it would go wherever the distro decides it belongs. That makes dealing with it a local policy issue, so you're free to name and deal with it however you see fit.

iff you accept /* -> /usr/, then why not instead /usr/ -> /*

Posted Mar 1, 2012 21:11 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (2 responses)

You know....
bringing the /me directory into the conversation is just unnecessary complexity.

-jef

iff you accept /* -> /usr/, then why not instead /usr/ -> /*

Posted Mar 1, 2012 23:48 UTC (Thu) by filteredperception (guest, #5692) [Link] (1 responses)

+1 for the joke jef, but cmon, I know _you_ could have given a legitimate answer to my question... Seriously, what is the short answer for 'why have /usr at all in the post merge world?' (as anything but the compatability symlink, versus putting the opposite compatability symplinks under /)

iff you accept /* -> /usr/, then why not instead /usr/ -> /*

Posted Mar 1, 2012 23:52 UTC (Thu) by filteredperception (guest, #5692) [Link]

actually maybe I'm just being stupid. network mounted /usr. Probably I should follow one of these links and either read for the first time, or more likely discover something I never bothered to remember, as to what 'usr' means. (?SharedResource).

Various notes on /usr unification

Posted Mar 1, 2012 20:42 UTC (Thu) by dskoll (subscriber, #1630) [Link] (1 responses)

I'm sold. That's a beautiful, clear and logical explanation. Thanks.

I guess /var/tmp either has to go or we have to separate the "volatile" filesystems into "volatile-and-likely-to-fit-in-RAM" vs. "volatile-and-possibly-enormous".

Various notes on /usr unification

Posted Mar 1, 2012 21:28 UTC (Thu) by nybble41 (subscriber, #55106) [Link]

> I guess /var/tmp either has to go or we have to separate the "volatile" filesystems into "volatile-and-likely-to-fit-in-RAM" vs. "volatile-and-possibly-enormous".

I believe /var/tmp is generally considered to be "stable across reboots", so at least that much would remain consistent. *Old* files may be deleted from /var/tmp, of course, but while files in /tmp are only valid for the life of a single process (and thus can always be deleted after a reboot), files in /var/tmp represent transient data which may nonetheless remain useful well after the process which created it has exited.

Since /var/tmp is defined as stable it's unlikely to be located on a RAM disk, but applications using large temporary files must still be prepared for the possibility of running out of free space.

Various notes on /usr unification

Posted Mar 2, 2012 5:37 UTC (Fri) by david.a.wheeler (subscriber, #72896) [Link] (6 responses)

Parent said, "The idea is that you should be able to have a separate partition for each different kind of data...." and listed:
/ Machine specific, read-only
/var Machine specific, read-write, stable across reboots
/tmp Machine specific, read-write, volatile across reboots
/usr Shared, read-only
/home Shared, read-write

Also add:
/etc Machine specific, read-only

(Where "read-only" means "written only by special sys admin actions")

BTW, I am *glad* that there was a usrmove argument. When someone proposes a big change to some convention that is widely used, people should examine it for problems BEFORE it's changed, and any change has some pain. I think usrmove is overall a good idea, though.

Various notes on /usr unification

Posted Mar 2, 2012 15:54 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link] (5 responses)

But there's no particular reason to split /etc out as a separate directory. It's already a subdirectory of /, which has the same general requirements of being machine specific and only alterable by root. And you probably don't want it as a separate partition because it contains fstab. which is important to have immediately on mounting /.

Various notes on /usr unification

Posted Mar 2, 2012 19:38 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

fstab is becoming more and more useless by the day. If initramfs is going to be mounting the real / and /usr then maybe we could move mounting of other filesystems there?

After all, the only major remaining filesystem is /home and maybe a swap partition.

Various notes on /usr unification

Posted Mar 6, 2012 1:36 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link] (3 responses)

I don't see fstab as losing its importance. You still need some place to record the identifiers and mount points for the key partitions, which may include things like /var, /tmp, /opt, and /usr/local in addition to the ones you mentioned. That information has to remain available after initramfs has done its job, and root still needs to be able to edit it in case the hardware configuration changes. It seems to me that it's simpler to have one canonical copy of fstab that's located on the one disk that initramfs knows about by itself. The mount process is a bit more complicated than it would be if initramfs had its own copy of fstab- you have to mount that disk, read fstab, and then mount any other disks mentioned there- but that's still simpler than trying to keep two copies of fstab in sync by rebuilding initramfs any time the canonical copy is edited.

Various notes on /usr unification

Posted Mar 6, 2012 11:37 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

I wouldn't mind something fstab-y but a little bit more modern. Fstab format is way too obsolete, there are no real notions of dependencies and ordering.

For example, Amazon EC2 nodes have specialized disks for scratch space. I want to assemble a RAID-0 of them, format them as XFS and then mount them on /tmp.

In my ideal world the fstab-ng would have entry like:
>none /tmp xfs defaults,noatime depends-on=assemble-scratch-raid

Where assemble-scratch-raid is, perhaps, a systemd task.

Various notes on /usr unification

Posted Mar 6, 2012 21:48 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

I could definitely see a reworked fstab format. Even if you don't want to work in dependencies- there's some concept of ordering, at least- the format is also terse and cryptic. I wouldn't mind something a bit more verbose and readable. Just don't let Lennart Poettering get involved, or it'll generate another tempest in a teapot no matter how good an idea it is.

Various notes on /usr unification

Posted Mar 6, 2012 22:03 UTC (Tue) by dlang (guest, #313) [Link]

why would there be enough benefit in a more human readable fstab (which is mostly not manipulated by humans anyway) that would make it worth breaking all existing tools that understand the current format?

while I would agree that there are nicer ways to configure what's managed by fstab, there is a large pool of tools and training that know the existing format, what may have been a great change to make 30 years ago needs much more justification now.

Various notes on /usr unification

Posted Feb 28, 2012 19:47 UTC (Tue) by miguelzinho (guest, #40535) [Link] (4 responses)

After reading this I was completely amazed to know how the creation of /bin, /sbin and /lib were actually an workaround to hardware limitation. The article's title must change to "Various notes on /usr RE-unification, 30 years late".

Various notes on /usr unification

Posted Feb 28, 2012 21:35 UTC (Tue) by khim (subscriber, #9252) [Link] (1 responses)

Often it does not really matter just why this or that decision was made. Other, different things start depending on it. And often even if the initial decision was done on a whim it can not be changed later because other, different things start to depend on it.

One example from my own practice: when I've developed compiler for x86-64 NaCl I've noticed that our security model makes it actually impossible to address more then 4GiB of address space and this makes 8bytes pointers (normal for x86-64) kinda pointless. We've talked a bit about it, I've tried to switch GCC to model with 4 byte pointers in memory (which was easy enough to do) and we used the result (it was faster and used less memory: what's not to like?). For about month or two our compiler had insane model where pointers were 4 bytes long and "longs" were 8 bytes long, but we've found out quite soon that such combination just triggers too many bugs in programs. Eventually we've standardized on ILP32 model. Later, when PNaCl effort was started they found out that all NaCl compilers (X86-32, X86-64 and ARM) are using ILP32 model - and this was embedded in the bitcode format!

Now, if we'll try to change x86-64 NaCl memory model we'll find out that it'll not be easy to do. Even if the initial motivation was just to save some memory by removing useless zero bytes (and to get slightly better results on benchmarks as a bonus) today there are another justification.

And this happened just in a few last years! As you can guess Linux has somewhat longer history and / vs /usr split was used for different things along these years. It looks like today they all are not really valid (Linux does not include one set of programs in /bin and another, totally different, in /usr, for example) but it's not enough to say "hey, Ken and Dennis leaked their OS into the equivalent of home because an RK05 disk pack on the PDP-11 was too small thus today it's all is not needed because we don't use RK05 disk packs". No, we should evaluate situation today and look on the use of said split today.

Why it was done in first place is not important question (unless you like UNIX folklore). Why it's used today is an important question. That's why 40 years old fairy tales are not even mentioned in the appropriate place.

Various notes on /usr unification

Posted Feb 29, 2012 9:16 UTC (Wed) by jezuch (subscriber, #52988) [Link]

> Often it does not really matter just why this or that decision was made.

It's basically how culture "happens". "We do this because, uh, well, waidaminute... We do this because our parents and out grandparents did this, go away and don't ask stupid questions!"

;)

Various notes on /usr unification

Posted Feb 28, 2012 21:41 UTC (Tue) by flewellyn (subscriber, #5047) [Link] (1 responses)

Very interesting link, thank you! It's funny seeing how a temporary hack got turned into a rule with lots of post-hoc justification.

I do note this amusing bit from the post:

>I'm still waiting for /opt/local to show up...

I have seen this in the wild! If you use Macports on OS X, /opt/local is where the ports tree and all packages installed through it live.

Various notes on /usr unification

Posted Feb 29, 2012 13:53 UTC (Wed) by sorpigal (guest, #36106) [Link]

I think the point is that it's not just some justifications after the fact, it's new requirements that happen to rely on old behaviors. Change the old behaviors, break the new requirements. Maybe the original behavior was silly but the new requirement still relies on it and if you change the original behavior users of the new thing will not be satisified with "the original reason for that behavior no longer exists."

The most important question not asked

Posted Feb 28, 2012 23:43 UTC (Tue) by bojan (subscriber, #14302) [Link] (8 responses)

Which is: is /user going to be a symlink to /usr or the other way around?

Oh, and when is umount finally going to get an 'n'?

;-)

The most important question not asked

Posted Feb 29, 2012 5:32 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

Or creat that final 'e'.

[1]http://en.wikiquote.org/wiki/Kenneth_Thompson

The most important question not asked

Posted Feb 29, 2012 13:47 UTC (Wed) by james (subscriber, #1325) [Link]

AIX has had an unmount command (hardlinked to umount and doing the same thing) for over twenty years, judging by this AIX 1.3 manpage.

When is Linux going to get this sort of enterprise-level functionality out-of-the-box?

(Three buzz-phrases in one sentence -- my fingers feel dirty.)

The most important question not asked

Posted Mar 2, 2012 16:08 UTC (Fri) by dmm (guest, #9815) [Link] (5 responses)

While we're at it, fstab is so 70s, how about /etc/filesystems.

The most important question not asked

Posted Mar 2, 2012 22:52 UTC (Fri) by anselm (subscriber, #2796) [Link]

That's already spoken for.

The most important question not asked

Posted Mar 8, 2012 14:05 UTC (Thu) by nye (subscriber, #51576) [Link] (3 responses)

>While we're at it, fstab is so 70s, how about /etc/filesystems.

ITYM '/Et Cetera/My Filesystem Table'

(So apparently one of the reasons for 'c:\Program Files', when other options like 'c:\Programs' seem more sensible is that it was a way to flush out programs which didn't handle spaces in file paths.)

The most important question not asked

Posted Mar 8, 2012 14:46 UTC (Thu) by khim (subscriber, #9252) [Link] (2 responses)

And, of course “c:\Windows” is not “c:\Windows System” because Windows itself does not like spaces (or long file names). Typical for Microsoft: everyone else should fix their programs when they port to new version of Windows (old programs saw “C:\PROGRA~1” and were happy), but Microsoft don't.

P.S. This is not a speculation: Windows9x gave you ability to select name of directory where you can install it. I've installed it in “c:\Program Files\Windows” (to keep it confined to a single subdirectory) and then various Microsoft packages started complaining that they should be moved from “c:\Program Files\Windows” to “c:\PROGRA~1\WINDOWS”… When I've reported it as a bug in DirectX the bug was closed as INVALID with comment that you should not use non-MSDOS names from the main Windows directory.

The most important question not asked

Posted Mar 8, 2012 18:45 UTC (Thu) by farnz (subscriber, #17727) [Link] (1 responses)

Windows 95 is not a fair comparison here, though. It, Windows 98 and Windows ME were all stop-gaps until Windows NT was able to replace them (which it got close to with Windows 2000, and managed with Windows XP), and they did a variety of interesting things related to their ability to use MS-DOS device drivers for low level filesystem access. It's much the same situation as criticising Linux, because when you run certain apps on a FreeBSD system with their Linux compatibility layer they don't work well.

Of course, this shows one side of the "gradual change" coin - Windows XP was released in 2001, so it took Microsoft 6 years to shift people in their preferred direction.

If you could repeat the experiment with NT-based Windows, that would be interesting; I don't know if any NT Windows could install to a directory other than C:\Windows or C:\WINNT, however.

The most important question not asked

Posted Dec 13, 2012 15:18 UTC (Thu) by mirabilos (subscriber, #84359) [Link]

I’ve done C:\WIN for 3.x, 9x and 2000 (NT-based) successfully.