Preparing for a merged /usr in Debian
Debian is the latest Linux distribution to consider moving away from the use of separate /bin, /sbin, and /lib directories for certain binaries. The original impetus for requiring these directories was due to space limitations in the first Unix implementations, developers favoring the change point out. But today, many of the services on a modern Linux system impose requirements of their own on the partition scheme—requirements that make life far simpler if /bin, /sbin, and /lib can be symbolic links to subdirectories within a unified /usr directory. Although some resistance was raised to the change, the project now seems to be on track to make "merged /usr" installations a supported option. And perhaps more importantly, the arguments favoring the merge suggest that many Debian developers would like to see that configuration eventually become the default.
The current discussion began when Marco d'Itri raised the issue of committing to a /usr
merge plan on the debian-devel list
on December 31. D'Itri, who maintains several Debian packages
critical to the boot process, is also the developer behind the usrmerge utility,
which is designed to convert an existing Debian system. As an optional
tool for use by individual system administrators, usrmerge does not
impact systems where the user opts not to convert their machine to a
merged-/usr configuration. Usrmerge has
seen sufficient testing, he said, so "there is no need for a flag
day, archive rebuilds or similar complexity
". Once a few
outstanding bugs in other packages have been fixed, a change
to official Debian policy would be next. If that policy change
passes, support for
installing Debian on new systems with the new merged /usr would be the
next step. D'Itri noted, though, that it may be possible to continue
supporting unmerged systems as well.
In response to d'Itri's email, some list subscribers voiced objections to the /usr-merge plan. Unfortunately, considerable heat was generated in the resulting thread, much of it stemming from the assumption that adding support for a merged-/usr installation option meant that support would be dropped for non-merged installations. That was not in d'Itri's proposal, although there were clearly those in the thread who favor such an eventual change.
Consequently, it is worth examining the objections raised, since they are likely to surface again should a merged /usr be proposed as the default (or only) option in the future. Generally speaking, the concerns fell into three categories. First, some users were concerned about the technical impact of the merge, such as how it would affect the ability to run Debian on small embedded systems.
Second, some users were unhappy with the change merely on the grounds that it breaks with longstanding traditions and (at least in the eyes of some) the Filesystem Hierarchy Standard (FHS). They were not persuaded they had heard a sufficient justification for the change, so they felt it was an unwise move. Some thought implementing it, even it it had some benefits, would result in considerable additional work for the Debian project.
Finally, some objected to the change on the grounds that it appeared to be a change imposed by outside projects—thus making it pressure that Debian should resist as a matter of independence. Some of these users believed the /usr-merge change originated with the systemd developers (a proposition that, predictably enough, drags extraneous feelings and baggage into the debate), while others believed the change was being pushed by developers from the Fedora, CentOS, and Red Hat communities (Fedora having made the change quite some time ago).
Technical considerations
Early Unix-like operating systems placed binaries essential to the system start-up process in /bin, /sbin, and /lib in order to place /usr on a separate disk partition from /. This helped ensure that the / partition would be as small as possible, allowing the system to boot from a single drive (given the disk sizes available at the time). But such disk-size limitations are not encountered on modern systems—except for the smallest of embedded systems. Simon Richter pointed that out on the list, also noting that the additional time required to mount a temporary root filesystem (e.g., one in an initial ramdisk or initrd) was not an appealing alternative either. Ian Jackson pointed out several other scenarios that could be similarly complicated by the merge, such as systems that prefer to mount /usr as a read-only filesystem, perhaps even mounted over NFS.
A slightly different concern was raised by Marc Haber, who contended that upgrades would be complicated on systems where /usr and / were currently on separate partitions, such as the case where the larger, merged /usr no longer fits in its partition.
Using an initrd (or initramfs) was frequently cited as a path
forward, both for those working on resource-constrained embedded
systems and for Jackson's NFS-mounted /usr case. Understandably, the
concern was that workaround means imposing an undue amount of work on
the user. To disprove that thesis, Christian Seiler argued that it is not really that
complicated to implement one's own initramfs-like workaround, and posted
a 300-line
example that he wrote "just for the heck of it
".
Seiler's code tries to mount /usr before starting the init system, and
does so by examining /etc/fstab to determine which partition holds the
/usr directory.
The code garnered some accolades from those involved, although it does not seem like Seiler intends to develop it further any time soon. Still, it does illustrate that the issue of partitioning-scheme workarounds may not be as daunting as some have feared. The discussion thus eventually turned toward whether or not Debian would risk losing a significant number of embedded users by not addressing their specific concerns. That question remains open to debate, given its speculative nature.
But the discussion thread also revealed that there are still misunderstandings about how the merge would impact different system configurations. In all of the serious technical objections raised, the core issue is that the /usr directory must be accessible during system start up, because /bin/foo would be a symbolic link to /usr/bin/foo. As Ansgar Burchardt reminded everyone, it does not matter whether /usr and / are on the same partition.
Furthermore, as Russ Allbery pointed out,
Debian has depended on /usr being mounted "early" for quite some
time—he speculated that the project may not have shipped a
release where the default configuration
did not depend on /usr for boot in four or five years. Up until now,
he said, there have been "not particularly clean
"
workarounds required to ensure that /usr is available early:
As to the upgrade issue, Seiler noted that a need to re-partition has always been a risk during upgrades, such as to increase the space allocated for a /boot partition. As for those wanting to mount /usr in read-only mode, several people pointed out that this, too, should already be doable. D'Itri, for example, noted that it should already be possible to run with / (including /usr and /etc) mounted read-only; not being able to do so with a default Debian install is considered a bug—one with known workarounds.
FHS, init, and more
Those users objecting to the /usr merge on the grounds that it breaks with tradition were, for the most part, appeased by the technical debate. Seiler pointed out that the FHS does not rule out using symbolic links in /bin, /sbin, and /lib; it only mandates that / must have the tools necessary to mount other filesystems. He also noted that Debian already breaks with the FHS in several places, in each case where it has a good technical reason:
Similarly, he added, the FHS was created to offer compatibility between distributions. So, as other distributions implement /usr merges of their own, as seems to be the trend, it is in Debian's interest to follow suit—lest some future FHS revision, ironically, dictate a merged /usr.
Those participants in the thread whose objections stemmed from the perception that systemd was forcing the /usr merge onto Debian were, perhaps predictably, the most difficult to win over. But as Seiler (in the message cited earlier) and others explained, systemd is not the issue. The issue is the various services started by systemd, such as any service involving a udev rule that uses a binary in /usr to set up a piece of hardware.
Since udev and cheap
hot-pluggable hardware did not exist in the earliest days of Unix, the
traditional init approach has had to adapt in order to handle them.
Ben Hutchings pointed out that the udev hooks of
concern to systemd also
cause problems for sysvinit. Seiler added that "the _only_ reason you
associate [the issue] with systemd is
that systemd developers chose to add a warning to systemd 5 years ago
in order to put people on the right track
".
The effort and the positives
The last remaining objection—and, perhaps, the one that is most nebulous—is the notion that implementing the change will be a difficult or time-consuming process for Debian project members.
In reply to Hutchings, Daniel Reurich asked if the proper solution would not be to fix all of the packages that broke if /usr was not available early in the boot process. To that, Vincent Bernat replied that the labor (much of it subtle) required to fix all of the affected packages exceeds the volunteer-power available and, conversely, implementing the /usr merge will reduce the amount of code Debian needs to maintain going forward because it simplifies the early-boot stages. In addition, as other distributions merge their /usr directories, it will be more common for upstream packages to hardcode references (such as those in scripts) to the resulting paths. Altering all of those packages would create substantially more work for the Debian project.
Allbery also contended that a non-merged /usr imposes considerable overhead that Debian would be better off without:
He also noted that, as far as he is concerned, the merge solves a
"coordination issue
" rather than a technical one, since "it affects
thousands of packages maintained by numerous different
maintainers
". Sane prioritization, he said, dictates making
the merge. Furthermore, Debian cannot maintain support for every possible variant of
merged and unmerged /usr directories, and attempting to do so
would mean "tons of painful work
" that is useless to most users.
Seiler also provided a positive spin on the change. Once one agrees that having / mounted without /usr no longer works in practice, he said, the /usr merge proposal becomes more interesting. Duplicate binaries are no longer needed, package maintenance is simplified, consistency across packages is improved, and consistency with other distributions is increased. Moreover, there are interesting new avenues to explore, because OS snapshotting becomes simpler, sharing binaries between machines is easier, and truly stateless systems become a possibility.
In a discussion where much of the talk started from the notion that
Debian was being forced to implement the /usr merge, Seiler's points
are a particularly welcome addition. Although there is no time frame
established for implementing the change, it seems likely that a merged
/usr directory will become an installation option at some point in the
near future. Further out, there appears to be significant support for
making a merged /usr the default—but that change is likely to
spawn another round of debate.
Posted Jan 7, 2016 10:37 UTC (Thu)
by Seegras (guest, #20463)
[Link] (53 responses)
Posted Jan 7, 2016 18:32 UTC (Thu)
by mattdm (subscriber, #18)
[Link]
Posted Jan 8, 2016 0:18 UTC (Fri)
by luto (guest, #39314)
[Link] (51 responses)
We're not there yet for general use, but things like OSTree and a bunch of container solutions are moving strongly in that direction.
Posted Jan 8, 2016 15:05 UTC (Fri)
by roblucid (guest, #48964)
[Link] (3 responses)
Merging /bin, /sbin & /lib into /usr, means all the files are in 1 folder, simplifying $PATH and reducing compatability links. I guess it makes it easier for developers to avoid command name clashes to.
Unfortunately the result of symlinking things, causes developers who hard code paths to have portability problems, going to more traditionally laid out systems.
Posted Jan 8, 2016 16:15 UTC (Fri)
by kokada (guest, #92849)
[Link]
Nowadays this is the work of initramfs, so there isn't much sense to run separate /bin, /sbin and /lib anymore (except for embedded systems, however I think a traditional distro like Debian wouldn't matter for them).
Posted Jan 12, 2016 12:48 UTC (Tue)
by smcv (subscriber, #53363)
[Link]
The "/usr read-only, root read/write" setup is considerably more effective if /bin, /sbin and /lib* are merged into /usr, because now libc.so.6 and other important components would naturally be read-only too.
/etc has to be part of the root filesystem because it contains the system-specific configuration that tells early boot what else to mount, namely /etc/fstab.
Posted Nov 23, 2016 7:34 UTC (Wed)
by raliscenda (guest, #100421)
[Link]
Your assumptions are that GNU/Linux is only one favour of UNIX like system developers have to support. For sure all BSD based systems does not have plan to merge /usr in near future, so portability issue stay with us for a while.
Posted Jan 8, 2016 22:48 UTC (Fri)
by walex (guest, #69836)
[Link] (46 responses)
You know that "simple" in this discussion to me sounds really funny. Because the Very Important People who have decided that merging is Just Like Microsoft Does It And Therefore Is Good have forgotten two far from trivial sources of huge complexity: The original UNIX system had a / of 5MiB, and a /usr of 20MiB, and as the name says it was originally for user home directories, and later it got the traditional /usr/bin and /usr/lib for user-installed software, which then became sort of standard. Today there is simply no point in having something of the extraordinary complexity and fragility of something like GRUB or the "initramfs": all we need is a simple / partition with a simple default filesystem, a kernel with that filesystem type built-in, and a bootloader that just loads that kernel. Ah we would also need simple, static binaries and a simple init that just spawns a service supervisor (or a shell or whatever) to make the boot process indeed simple and robust. But that aspect of the story is long lost. Because the clever people (many of them working hard inside Debian and Fedora) who love complexity and fragility want to have base tools and init systems dependent on DBUS or a dozen shared object libraries, and are worried about the complications of keeping / and /usr, but obviously are perfectly comfortable with GRUB and "initramfs". But shouldn't then /etc/ itself be under /var? And what about a different classification for top-level hierarchies, based on whether they are host-specific or site-specific? For example having a host-specific / hierarchy and a site-wide /usr one? :-)
Posted Jan 8, 2016 22:55 UTC (Fri)
by luto (guest, #39314)
[Link] (44 responses)
You're assuming that your "simple / partition" can be loaded by just passing the root device path to a kernel. If it's a plain partition on a plain ol' disk, that's probably the case. If it's on LVM or RAID or uses dm-verity or sits on an SPI flash chip because you're net-booting and don't want a disk at all, then this isn't going to work.
Initramfs is, in fact, a simple filesystem containing /. (It's not the *final* /, but that's more or less irrelevant, except insofar as this tends to cause dbus-daemon to explode, but that's purely dbus' fault.)
Given that you need the bootloader to find the kernel image no matter what you do, it's not much of a stretch to ask the bootloader to use the same facility to find initramfs, and everything else is fine after that.
> But shouldn't then /etc/ itself be under /var?
Probably not. In the ostree model, /etc merges in changes from updated packages, whereas /var does not.
Posted Jan 10, 2016 3:36 UTC (Sun)
by rqosa (subscriber, #24136)
[Link]
> If it's a plain partition on a plain ol' disk, that's probably the case. Even then, it might not be possible to mount the / partition without an initramfs, if parts of the kernel needed to access the disk in question were compiled as modules instead of built-in. (This used to be a common problem with booting from SCSI drives, and SATA drives before AHCI was commonplace — different host-controller chips required different drivers, and distributions don't want to have to build all of these drivers into the kernel statically, so they instead build the various drivers as kernel modules and provide a program to generate an initramfs image at kernel-install time with whichever modules are needed for the specific hardware in question.)
Posted Jan 14, 2016 10:11 UTC (Thu)
by hitmark (guest, #34609)
[Link] (42 responses)
The people calling the shots are so far up in devops lala land (architecture astronauts do not even begin to cover it) that they should really be forked off and left to make their own distros rather than foist their world onto those that has been around for a decade or longer and are quite happily doing it their way.
Posted Jan 14, 2016 13:13 UTC (Thu)
by pizza (subscriber, #46)
[Link] (41 responses)
I'm sorry, I don't follow. The OP is saying that real-world Linux systems are already quite complex (due to the sheer variety of real-world hardware and deployment options), and you turn around and claim that architectural work to accommodate how folks in the real world deploy Linux on real hardware is a sign that the folks doing the design and implementations to make it all possible, supportable, and generally JustWork(tm) are divorced from reality?
*shakes head* If you have an actual point, I'd love to hear it, because at this point you're basically complaining that water is wet.
Posted Jan 14, 2016 18:39 UTC (Thu)
by hitmark (guest, #34609)
[Link] (17 responses)
Never mind if X is an option that has been with the very concept of *nix since day one. But is now being taken away because it may get in the way of someone doing Y. Even though if you do Y you have had the option of foregoing X for ages.
Or maybe we are far beyond the point where trying to build one generic distro image is possible, and continuing to strive for it is a fools errand.
How about this, provide a minimal, streamlined base, and then let people build from that. Rather than try to placate every last "potential" out of the gate.
Posted Jan 14, 2016 18:55 UTC (Thu)
by pizza (subscriber, #46)
[Link] (16 responses)
The hilarious bit is providing the former is exactly what they're trying to do. It turns out that different people disagree on what constitutes a "minimal, streamlined" base.
Posted Jan 14, 2016 23:11 UTC (Thu)
by raven667 (subscriber, #5198)
[Link]
Posted Jan 16, 2016 14:32 UTC (Sat)
by walex (guest, #69836)
[Link] (14 responses)
Indeeed and I think that we can all agree that having a 4 stage boot process with 4 different "root" filesystems (/boot, "initramfs" snapshot, /, /usr) is too complex and fragile.
Given that the end goal is having /usr mounted, that cannot be removed, so the alternatives are: To me dropping / and keeping the far more complex and fragile /boot and "initramfs" stages seems solving the wrong problem at best.
Posted Jan 16, 2016 19:49 UTC (Sat)
by mjg59 (subscriber, #23239)
[Link] (12 responses)
How does the kernel obtain the passphrase to decrypt an encrypted /? The network configuration to mount a / over iscsi? Interact with the TPM in order to attest system state to the user?
Posted Jan 18, 2016 23:14 UTC (Mon)
by walex (guest, #69836)
[Link] (11 responses)
« In exactly the same way GRUB currently prompts for the passphrase to mount an encrypted /boot, the same way GRUB currently gets the network configuration to mount /boot over iSCSI, and using the same way GRUB uses to interact with the TPM to attest system state to the user. Equivalently all the same ways all those things are currently done by the Linux kernel booted by GRUB to mount an encrypted "initramfs", to get the network configuration to mount an "initramfs" over iSCSI, and for the "init" processor in the "initramfs" to attest the system state to the user. :-). Isn't that totally obvious? Didn't I write very clearly that «
If you dispute that, this discussion is pointless like most discussions with the members of the poetteringista tendency, because evidently the understanding of how booting works and the role of the / filesystem in UNIX/Linux systems has been completely lost.
Posted Jan 18, 2016 23:31 UTC (Mon)
by mjg59 (subscriber, #23239)
[Link] (10 responses)
Things that grub doesn't do. And nor does the kernel. But an initramfs does. So we use the solution that exists, and we'll continue doing so until somebody dislikes it enough to come up with an alternative that works better.
Posted Jan 19, 2016 8:53 UTC (Tue)
by walex (guest, #69836)
[Link] (9 responses)
The solution that exists and has existed for decades and has worked well for decades is to mount am ordinary '/' filesystem at first-stage boot and use that to mount '/usr' (encrypted over iSCSI using TPM if necessary) for second stage boot. If you think that is more complicated and fragile than to mount first '/boot', then "initramfs", then '/', then '/usr', and that in such a sequence the clever move is to drop '/', good luck.
Posted Jan 19, 2016 9:02 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link] (3 responses)
That's certainly a thing you could do, but now you have two choices:
1) Have /var, /home and /etc be part of /, and so be unencrypted
Both of these choices are bad, and so your solution is a bad solution and has not worked well for decades.
Posted Jan 19, 2016 19:25 UTC (Tue)
by walex (guest, #69836)
[Link] (2 responses)
It seems that I need to explain again the basics of how the boot process works and how UNIX/Linux sytems wor k in repect of the mount and pivot_root system calls, I'll try to keep it short: The outline boot sequence can therefore well be: That the above, which seems to me basic system knowledge, needs explaining again and again in detail is quite frustrating; as I wrote a few times already, the first-stage boot / (or kernel) does not need to be the very same as the the second-stage / (or kernel), and indeed in the current arrangement there are another two. Plus there is mount --bind for an alternative option. Anyhow in the current practice GRUB's root (first-stage root) /boot is inside an unencrypted partition distinct from that containing / (second/third stage root) and someone could replace anyhow its contents with suitably "improved" kernel images or "initramfs" snapshot images. I know people who understand boot processes and who keep /boot (only) on a USB storage device they carry with them at all times because they reckon that an encrypted / with an unencrypted /boot is not giving them peace of mind.
Posted Jan 19, 2016 19:38 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
Posted Jan 20, 2016 9:00 UTC (Wed)
by anselm (subscriber, #2796)
[Link]
This is essentially what happens with initramfs, except that the initramfs is loaded using the boot loader/system firmware, and can in principle come from anywhere (including a remote server). Your approach forces you to have the first-stage “/” available locally as an actual file system on a storage medium, which sucks if you want to have a diskless system.
You still need to ensure that whatever is on the first-stage “/” matches the rest of the system, just like with an initramfs. The difference is that the process of making initramfses is fairly well-understood and there are good tools for it already.
Two points:
Posted Jan 19, 2016 11:56 UTC (Tue)
by jond (subscriber, #37669)
[Link] (4 responses)
Posted Jan 19, 2016 16:54 UTC (Tue)
by paulj (subscriber, #341)
[Link] (3 responses)
Not all the world is anglo-phile.
Posted Jan 19, 2016 16:57 UTC (Tue)
by paulj (subscriber, #341)
[Link]
«This is common in France apparently»
Posted Jan 20, 2016 14:22 UTC (Wed)
by jond (subscriber, #37669)
[Link] (1 responses)
Posted Jan 20, 2016 17:27 UTC (Wed)
by apoelstra (subscriber, #75205)
[Link]
Posted Jan 19, 2016 12:39 UTC (Tue)
by nix (subscriber, #2304)
[Link]
Now linked-in initramfses are, admittedly, not what distros use, but they still work (and if they stop working people like me will yell bloody murder so they should keep working!)
Other benefits of using an initramfs include that the terrible hack whereby fsck has to work on mounted filesystems and then you sometimes have to crash-reboot is simply gone: fsck of filesystems needed by early boot, be that /, / and /usr, or whatever, is just like fsck of everything else. Plus, of course, it means that those of us using setups you consider intolerably unusual such as RAIDed / can use them without effort. Heck, I've suddenly switched an existing system from RAID to RAID-partially-over-NBD in the past when a sudden disk failure left me bereft on one machine but with a superfluity of storage on another: without an early userspace, this would have been an entirely hopeless quest without support in the *kernel* for such a baroque configuration, which is unlikely in the extreme.
This was not something I did recently, either -- it was way back in 2005. This configuration has been useful for over a decade now. Recent architecture astronautics it is not.
Posted Jan 15, 2016 21:29 UTC (Fri)
by walex (guest, #69836)
[Link] (22 responses)
The OP is saying that real-world Linux systems are already quite complex (due to the sheer variety of real-world hardware and deployment options), and you turn around and claim that architectural work to accommodate how folks in the real world deploy Linux on real hardware is a sign that the folks doing the design and implementations to make it all possible, supportable, and generally JustWork(tm) are divorced from reality? What seems divorced to reality to me seems worrying a lot about the simplest and most robust aspect of booting, the mounting of /usr having mounted a full /, while the previous stages are both more complex/fragile and pointless. There are some distinct arguments here as to that: Because of what seems to me stupidity, currently most GNU/Linux distributions have a four stage boot process where the first three stages are quite more complex/fragile than just booting into a full Linux kernel that mounts a full / as per UNIX tradition: Now what's the point of eliminating the last step, which is the simplest and most robust, because if it fails one has available a full (if small) GNU/Linux system in the final / to recover while leaving the first three steps unchanged? Very little, it is just stupid. Consider the alternative of LILO loading a "first" stage Linux kernel mounting a / on the local disk, which then has a full environment to do the job currently done by GRUB with /boot and by the Linux kernel with the "initramfs" before pivoting. That's how UNIX worked originally, and is a sensible two-stage arrangement. Currently thanks to pivot/chroot and kexec the initial Linux on the initial / can replace itself or the initial / very easily. An "initramfs" is only needed for diskless workstations, and in that case a virtual disk image can be simply downloaded and treated as the /.
Posted Jan 15, 2016 22:24 UTC (Fri)
by pizza (subscriber, #46)
[Link]
[ citation needed ] (especially as my experience says that things are vastly more reliable and robust now then they used to be..)
> The first argument is whether very complex boot situations should be supported.
Yes, they absolutely must be.
> The second argument is: assuming that very complex boot situations should be supported, whether they should be supported with a simple robust first stage boot process into a full UNIX-like environment with a full / and then continue into a much more flexible second stage boot process that among other things may "pivot" to a new / or even "kexec"s a new kernel before mounting /usr and everything else.
Um, so how is that second-stage created? Unique for every system, or is it something more generic? If it's unique, who/what creates it, and how? If it's generic, how does it differ meaningfully from what we have today with the initramfs model?
> Now what's the point of eliminating the last step, which is the simplest and most robust, because if it fails one has available a full (if small) GNU/Linux system in the final / to recover while leaving the first three steps unchanged? Very little, it is just stupid.
You're correct, it is stupid -- Because it's highly unlikely that stage 3 will succeed but stage 4 will fail, and enforcing such a separation adds complexity for very little actual gain. Furthermore, the distinction between '/' and '/usr' has blurred to the point of being pretty arbitrary.
Posted Jan 15, 2016 22:53 UTC (Fri)
by anselm (subscriber, #2796)
[Link] (20 responses)
The problem with that is that your “first-stage” Linux kernel must know how to get at “/”. This means it must have all sorts of disk controller drivers and file system implementations statically linked (not to mention the modules taking care of disk encryption, RAID, and LVM), because it can't load any modules from disk before “/” is mounted.
This is the problem that initramfs (in its various evolutionary stages) was designed to fix. It basically lets Linux distributions provide bare-bones kernels that can find arbitrary modules at runtime without actually having access to “/lib/modules” on the root partition. The alternatives are (1) putting up with a large kernel that has all sorts of stuff statically built into it that you don't need, or (2) building your own kernel that is statically targeted to your specific hardware. That's how things were in the late 20th century, and the Linux community has largely moved past this because it is wildly impractical and the modern methods, though somewhat more complicated, are also much more versatile and, in the end, easier to maintain.
Posted Jan 15, 2016 23:19 UTC (Fri)
by raven667 (subscriber, #5198)
[Link] (19 responses)
Posted Jan 16, 2016 0:29 UTC (Sat)
by anselm (subscriber, #2796)
[Link]
If you have UEFI you can in principle do away with GRUB. GRUB is more flexible and can do a few things the usual UEFI implementations don't support, but reasonably recent Linux kernels can be booted from UEFI without GRUB.
Posted Jan 16, 2016 8:37 UTC (Sat)
by walex (guest, #69836)
[Link] (17 responses)
« This is exactly the same problem has GRUB has in getting to /boot for GRUB and for the kernel loaded by GRUB to get at the "initramfs" from the image loaded by GRUB for it. There is some dispute whether the full boot process should allow for arbitrary complexity, but I hope that there is no dispute that the starting point of the boot process must be a simple, limited configuration "container"/"filetree" of some sort like /boot or "initramfs". In the UNIX world and early pre-hipster :-) Linux the kernel itself and a simple / filetree performed the function of GRUB and /boot/"initramfs". With pivot/chroot and kexec a Linux kernel can perform that function again as a simple first stage of booting which can lead to an arbitrary complexity of second stage boot. « « Well it looks like both of these comments are based on a poetteringesque deep admiration for the Microsoft Highway to Complexity and Fragility or deliberate obtuseness, because the argument above is that instead of a simple full featured standalone / like UNIX had it is much simpler and more robust to have three of them: one as /boot for GRUB (or UEFI), one as "initramfs" for a Linux kernel, and finally, one merged with /usr. Again, there is hopefully no dispute that booting needs at least one simple first stage and then one can do a complicated second stage; the question is whether the first stage should be just LILO-installer + Linux-kernel + / or should be the whole list of GRUB-installer + GRUB-kernel + /boot + Linux-kernel + "initramfs", and whether all the drivers that are needed to support a complex second stage boot should be replicated in GRUB-kernel + /boot and Linux-kernel + "initramfs" and in Linux + merged-/usr.
Posted Jan 16, 2016 9:45 UTC (Sat)
by anselm (subscriber, #2796)
[Link] (15 responses)
The fallacy here is in assuming that because something worked in the 1980s it must work today and indefinitely into the future. (We see this a lot in people who question the need for something like systemd by claiming that System-V init is the solution to all problems because, you know, it runs shell scripts.)
GRUB is the way it is because it supports a wide variety of use cases apart from simply booting a Linux kernel. Distributions like it because it covers pretty much all deployment scenarios one is likely to run into in Real Life™, and it is usually easier to maintain a single solution that does everything you need than a variety of specific solutions that are all designed to optimise a step in a system's life cycle that happens fairly rarely in the first place. (Most people don't boot their system several times per hour.) As we've mentioned, it is perfectly possible to get rid of GRUB on a UEFI-based system, because the UEFI boot loader can boot a Linux kernel directly, so this makes your “first-stage boot” that much simpler already. Mainstream Linux distributions, however, can't rely on everybody having UEFI, which is why they still use GRUB because that works everywhere.
The reason why Unix systems in former times managed to get away with a much simpler boot structure is that the range of supported disk controller hardware, file systems, etc. was a lot smaller, and therefore it was reasonable for manufacturers to distribute kernels that had everything needed to get at “/” statically built into them. (As a matter of fact, traditional Unix kernels often didn't support the idea of loadable modules in the first place.) On contemporary Linux, there is no practical way of getting rid of “initramfs” that reliably supports things like encryption, RAID, or diskless systems and does not present a maintenance nightmare compared to which initramfs is really the simpler and more versatile solution. You end up trying to build distribution kernels that have everything in them statically on the off-chance (including stuff like a DHCP client that is better left to user space), and generally distributions are very happy to be past that, and to be able to use initramfs to offer modular, maintainable kernels. This means that you're unlikely to find anyone who actually deals with that sort of thing in actual practice who will agree with you that essentially going back to “pre-hipster” Linux is anything remotely like a good idea. In any case, the approach is fairly well-understood and rarely causes problems, so this is really a tempest in a teacup.
Posted Jan 16, 2016 14:09 UTC (Sat)
by walex (guest, #69836)
[Link] (14 responses)
That's a crazy starting point, because the assumed goal here is to cover a complex boot configuration, not to have a complex boot structure for its own sake. Complex structures don't bring any benefit in themselves, except the pride of a job poorly done :-).
There is no need to assume the that a complex boot configuration requires a complex, fragile, boot structure with 4 stages instead of a simple structure like "boot a stage one Linux kernel and / and then use that to do stage two of the boot process using the full power of a Linux system to boot a bigger kernel and mount a /usr filesystem".
But GRUB in order to mount /boot makes very strong and limiting assumptions about the range of possibilities it can cope with. Can you have GRUB loading its kernel, configuration file and modules from a /boot being a BTRFS filesystem on an encrypted LVM volume on top of an MD RAID1 spanning 2 DRBD volumes? If not, why are GNU/Linux users being so limited to the range of drivers statically built into the distributed GRUB kernel? :-) What is the case for the additional complexity and fragility of having GRUB + /boot replicating in complex yet limiting way a large part of the functionality of the Linux kernel and of /? What is the case for the additional complexity and fragility for having an "initramfs" that is a statically built and customized snapshot of a large subset of / and then "pivot" to a full / or in the latter-day proposal to a full /usr?
Posted Jan 16, 2016 16:34 UTC (Sat)
by MattJD (subscriber, #91390)
[Link] (4 responses)
Regarding initramfs and /, initramfs provides what / used to provide. Now / can require several steps before it mounts (for instance, on my laptop I need to type in my encryption passphrase). An initramfs can also provide a complete recovery shell, that relies on minimal other parts of the system to work. So what you propose already basically exists. The only difference is that you want a / that comes from a partition, whereas initramfs provides a / that comes from memory. Either way they need to be maintained, and neither provide a full system.
So I don't think there is a large gap between what both sides of this conversation want. It's just that we haven't done the cleanup to get closer to the desired simpler boot sequence.
Posted Jan 18, 2016 23:45 UTC (Mon)
by walex (guest, #69836)
[Link] (3 responses)
« It comes as amazing news to me that the UNIX and Linux kernels didn't have « « GRUB (the bootloader) has exactly the same fault! Please read the manual: it can only ever boot one statically configured (with grub-install) GRUB kernel. If you upgrade the GRUB kernel you have to rerun its (static) installation. It is the GRUB kernel that spawns its shell environment, loads perhaps dozens out of several hundred (most them entirely Linux specific) possible GRUB kernel modules from /boot, and run a system initialization script that loads a Linux kernel, a toy / snapshot in the "initramfs", and then passes control the Linux kernel. These are well described and documented booting steps, and I am astonished that so many people here seem to ignore them. This ridiculous and fragile complexity can be replaced by LILO replacing the GRUB bootloader, a "first stage" Linux kernel replacing the GRUB kernel, one that mounts a / directly instead of a pointless /boot or "initramfs", loads modules and scripts from it to mount /usr from anywhere, and load optionally a new kernel chosen dynamically by a user-interaction script, and kexec it. As I just wrote in another comment, it cannot be disputed that the booting sequence must involve: #1 a bootloader that can only load one statically configured kernel, #2 a simple filesystem mounted by that kernel using statically builtin drivers and configuration, and #3 a series of drivers, scripts and other executables run dynamically by the kernel loaded in #1 from the filesystem mounted in #2. There is simply no reason to have #1 as the GRUB stage1, #2, as the GRUB stage2 kernel, #3 as the /boot filesystem, then #2 again as a Linux kernel loaded from /boot with #3 again as the "initramfs".
Like in traditional UNIX and Linux systems this can all be far more easily arranged with #1 as LILO, #2 as a "first stage" Linux kernel with just (for example!) the VESA, serial, SATA, "ext4" drivers built in, mounting / from which any further driver modules, initialization scripts, etc. can be run to support arbitrarily complex boot configurations, including optionally loading a dynamically selected "second stage" Linux kernel to boot with kexec. This may be the fourth time I try to explain how boot sequences actually work and how a traditional UNIX/Linux boot sequence works and is more robust and flexible than involving needlessly both /boot and "initramfs" as replacement for /.... Note: as always, a completely storage-less system needs a RAM disk and to download a disk image into it to setup /, but that's then the job of the bootloader like PXE.
Posted Jan 19, 2016 0:27 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
grub stage 1 loads grub stage 2. Scripts in grub stage 2 allow the choice of kernel and initramfs, loaded via firmware provided drivers. initramfs contains a subset of /. initramfs mounts the real /.
What you're proposing is:
lilo loads the boot kernel. Boot kernel mounts a / (we'll call it "boot /") that contains a subset of the real /. Scripts there allow the choice of kernel and /, loaded via boot kernel provided drivers. After kexecing a new kernel, the new kernel mounts "boot /" again and executes further scripts that mount the real /.
In your proposal you end up mounting "boot /" twice, you have to keep "boot /" synchronised with the real /, your boot kernel is significantly more complicated than grub and can't make use of any firmware-provided functionality (making it less flexible), you're using lilo which doesn't work under UEFI (ie, modern systems) and you're relying on kexec which *doesn't work* in a number of real-world cases. Your solution is *more* complicated than what we have today.
Posted Jan 19, 2016 1:12 UTC (Tue)
by MattJD (subscriber, #91390)
[Link] (1 responses)
I apologize for the confusion. I was talking about IBM compatible BIOS. BIOS doesn't have, afaik, filesystem code. It can only read from hard drives (and depending upon age, limited sized drives!). Even Microsoft has a bootloader that loads the windows kernel. Before LILO, Linux must have had a different bootloader. Or it embedded the command line and fit in the first sector of the hard drive. Other Unix either had their own bootloader, or ran on other platforms which I don't know and won't speak for their system setup code.
UEFI, for instances, does contain file system code. And UEFI doesn't require GRUB to boot Linux from a filesystem. I'd bet two pennies that if we had UEFI from the start, GRUB would not be as popular and LILO would not held out so long, assuming all else being equal.
> GRUB (the bootloader) has exactly the same fault! Please read the manual: it can only ever boot one statically configured (with grub-install) GRUB kernel. If you upgrade the GRUB kernel you have to rerun its (static) installation.
A rarer event for me, and a step required as part of my system upgrade. Countless times I'd forget to rerun LILO when testing a kernel, and break the system. GRUB is far more resilient, even when I upgrade it. Anyways, no matter what you do there is always a risk upgrading bootloaders. Many people find GRUB, even with its complexity over LILO, more robust. I also find it much easier to debug failures with GRUB and recover systems.
> <snip> dynamically selected "second stage" Linux kernel to boot with kexec.
Also, some hardware is really unhappy about kexec. As a trivial example, my desktop GPU. It does not like being double initialized from the kernel, and I suspect it would take a lot of work to make it so for little gain (this is all on the official open source driver stack).
The boot process is always a pain on a system (my favourite example? The A20 line that needs to be dealt with in real mode). Everything is about trying to make it better. I won't argue the current system is the best thing possible (I know I'll be proven wrong eventually). I do think it's very good as is.
If you want to remove GRUB, UEFI allows that. I do see the value in making cut down systems that match your current system, by initramfs (and I've manually created an initramfs, including initial boot scripts. Interesting, but not unreasonable process. I used it for several years on an encrypted LVM /).
I'm willing to be shown that whole line of thought is wrong. I suggest you try creating your desired system, including all the necessary support structure. If it is better, I'd be happy to push it forward. As of right now, I just don't see how yours is better considering my experience with the Linux boot system.
Posted Jan 19, 2016 7:47 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
To an extent. UEFI only (typically) has support for FAT, so your kernel and initramfs need to live on FAT, and the compromises around using FAT as a meaningful filesystem tend to result in it being difficult to make it /boot. Grub is, perhaps depressingly, less awful than some of the compromises you have to make to launch kernels from UEFI directly.
Posted Jan 18, 2016 0:13 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (8 responses)
No, but in your proposed non-GRUB, non-initramfs setup you can't use LILO to get your initial kernel from there, either, so this is an obvious red herring. Try harder.
An initramfs is reasonably easy to construct and keep current and, if you're providing a Linux distribution, makes it unnecessary to use a kernel that has all sorts of drivers compiled statically to ensure wide hardware coverage, most of which drivers are unnecessary for any particular system (because the hardware isn't there) but can't be removed other than by compiling a new, customised kernel. (Remember that that was a situation we used to have to deal with in the early years of Linux, and now that we no longer need to, pretty much everyone is a lot happier.) This sucks especially if you're running an enterprise-class Linux distribution like RHEL or SLES where tampering with the kernel will mean your system no longer qualifies for vendor support in case of problems.
In addition, an initramfs makes it straightforward to do various things – such as network configuration in case your actual root file system is on a remote iSCSI volume, or obtaining a pass phrase for an encrypted root file system – that the kernel isn't really set up to do. How to do this in user space in an initramfs is quite well understood, and putting the required functionality into the kernel would only add unnecessary complexity and probably bugs.
Posted Jan 18, 2016 23:02 UTC (Mon)
by walex (guest, #69836)
[Link] (7 responses)
« No, but in your proposed non-GRUB, non-initramfs setup you can't use LILO to get your initial kernel from there, either, so this is an obvious red herring. Try harder. That objection is comically stupid, because I have proposed a setup where /usr gets mounted and the second stage kernel (if needed, and it will be rarely needed) gets booted by a full (if minimal) GNU/Linux system hosted inside a first stage / root filesystem, just like it has been done for decades by people who actually understand booting sequences, and that LILO in that setup is equivalent not to the GRUB kernel but to the GRUB bootloader that loads the GRUB kernel from the GRUB root filesystem which is /boot. I think that this is the third time that I have explained this is detail for those who ignore how booting sequences work.
« « People who understand how booting sequences work understand that doing all that with a stripped down UNIX/Linux kernel and a mounted / filesystem is much easier and less fragile than packing up every time a new static "initramfs", and prompting for passwords can be done by a trivial shell script in the mounted / filesystem, and does not need to be done by the UNIX/Linux kernel, and that there is no need to snapshot the / filesystem into an "initramfs" image unless there is a totally storage-less boot situation.
Posted Jan 18, 2016 23:17 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Please, desist from insulting people who actually do the work to make booting reliable and stable.
Posted Jan 18, 2016 23:37 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (3 responses)
Right, so please explain how you load your kernel from your first-stage “/” if that is encrypted. Something must find and decrypt the kernel before it can run. (GRUB can do it but you don't want to use GRUB.)
Nobody disagrees that this is possible in principle. We all did it that way on Linux 1.x, 20 years ago. The main reasons we prefer not to do it that way anymore are that (a) using an initramfs is much less hassle than compiling a new custom kernel for every single machine that we want to run Linux on, and (b) an initramfs lets us do various things in a reasonably straightforward manner that in the traditional setup are very difficult or impossible to achieve at all. In any case, it turns out that the initramfs approach, in actual practice, is a lot less brittle than you seem to assume.
OK, so “/” is encrypted (it would after all be a very bad idea to leave it unencrypted, with all the important stuff that you want to put on it). Now how do you get at that trivial shell script in the mounted “/” file system when the kernel needs the pass phrase to mount “/” in the first place?
Posted Jan 19, 2016 8:42 UTC (Tue)
by walex (guest, #69836)
[Link] (2 responses)
In exactly the same way that currently GRUB gets its modules and scripts and the new kernel in the content of an encrypted "/boot" filesystem, which also contains «important stuff that you want to put on it». :-)
As I wrote several times, disputing that there must be a statically installed bootloader and kernel mounting a simple filesystem at the start of the boot sequence from which to load another kernel, drivers, configuration files and scripts to mount anything else, is just pointless and a really stupid debating trick. Because that's how boot sequences work by necessity (the second stage is necessary because of the flexibility requirements).
The only things that can be disputed is whether mounting in sequence '/boot', "initramfs", '/', '/usr' are all necessary, and if not whether one should drop from that sequence '/boot' and "initramfs" or "/".
Posted Jan 19, 2016 8:57 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
The grub configuration and contents of /boot aren't secret and so don't need to be encrypted, so you're proposing copying much of the contents of / into /boot?
Posted Jan 19, 2016 10:05 UTC (Tue)
by anselm (subscriber, #2796)
[Link]
The issue here isn't necessity as much as convenience. As I said earlier, the Linux community used to do initramfs-free boots based on LILO for quite some time in the 1990s. We've come off that idea not because it doesn't work (it does, at least for simple scenarios), but because on the whole it is inconvenient and restrictive.
The initramfs-based approach, even though it looks more complicated at first sight, is vastly more powerful and convenient, and the added complexity is a price that most Linux distributions seem more than happy to pay. After all, Linux distribution makers are not in the business of making their life more complicated just for the sake of making it more complicated; they prefer approaches that work in a wide variety of cases and can be maintained.
If you think you can come up with a method of booting Linux systems that is better than what we have now, feel free to implement it and show it off to distribution makers. If it really is superior to the current approach, they'll be more than happy to take it on. That method worked for systemd in the basic-plumbing arena, after all …
Posted Jan 19, 2016 1:52 UTC (Tue)
by raven667 (subscriber, #5198)
[Link] (1 responses)
We can disagree without being insulting or dismissive here, some of the people here actually build distributions and know more about booting than either of us do combined.
> I think that this is the third time that I have explained this is detail for those who ignore how booting sequences work.
I think we are going to disagree about how clear your ideas have been represented by your writing, the evidence is in how few people seem to have understood your intent. Let me see if I understand what you are proposing as another valid way to boot a system.
If I understand correctly you are thinking that we should go back to a simple LILO or uEFI style boot loader that is capable of loading the kernel and passing parameters from the firmware but to disaggregate what is currently in the initramfs into /boot, presuming that the kernel will have built-in drivers equivalent to what the firmware provides to be able to access the block device that /boot lives on, or that the kernel will use the firmware APIs to read the block device. Once you can reliably access the /boot block device you can run your minimal userspace to do hardware setup and pivot into the "real" system.
> is to have a Linux kernel without needing any sorts of drivers compiled statically except those needed to mount the first-stage / filesystem from a local storage area, exactly as GRUB does it for boot,
I may be wrong I don't think GRUB has block device drivers to access /boot, it relies on the firmware built-in services to access the block device, although it does have its own minimal filesystem driver modules (BIOS or uEFI or whatever), which are loaded LILO-style using static block lists, which has the same problems as LILO but GRUB modules are updated more infrequently than the kernel so is less of a problem in practice.
> plus optional kexec
As mjg59 has said elsewhere, and he has more operational experience with this feature than I do, it is a neat feature and great when it works, it's not something that is so reliable that you'd want to depend on it for the common case of everyday booting. In any event, if you are going to boot a Linux kernel, it might as well be the "real" one.
Posted Jan 19, 2016 7:55 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
It does, but they're not typically used in BIOS or UEFI environments. I point this out merely for precise technical accuracy rather than it altering your point in any way!
Posted Jan 16, 2016 19:44 UTC (Sat)
by mjg59 (subscriber, #23239)
[Link]
Posted Jan 18, 2016 23:32 UTC (Mon)
by dirtyepic (guest, #30178)
[Link]
I always love it when the loonies preload their arguments like this and I know I can just skip the rest of the post and all the replies. Thanks big guy!
Posted Jan 7, 2016 22:51 UTC (Thu)
by jhhaller (guest, #56103)
[Link] (21 responses)
Alas, the first Unix computer I used (a PDP 11/40) had a single RK05 removable drive, so many of those strategies didn't help. Let me tell you, booting from a magnetic tape (with a formatted filesystem) was no picnic, but was the only way to recover from a crashed hard drive. Seeking between inodes and the data took forever, and the tape was obviously read-only (by taking out the write ring in the reel). We were never able to move to Unix v7, because the hard drive was too small, plus booting off the tape was even more painful on v7 than v6.
Some of the original designs percolated into the Linux design, but filesystems and drives are much more reliable today than in the original Unix era. While there are some reasons to keep separate /usr and root, the reasons are very thin now.
Posted Jan 8, 2016 14:49 UTC (Fri)
by roblucid (guest, #48964)
[Link] (20 responses)
There were also issues before the Berkely Fast File system, the free block list for allocation would soon result in files scattered all other the partition, due to the lack of cylinder groups. Therefore, you partitioned disks for performance
Anyway, when System D was introduced and required files within /usr as well as / to be mounted on init, OpenSuSE decided to mount /usr from the initrd like / if it was seperate, rather than declare seperate /usr unsupported, so an NFS or read-only /usr was made possible. I never saw any issues with that solution, the debate was short as Kay Seiver's propesed the change to initrd fairly quickly, rather than try and ram through the freedesktop.org/System D opinion of how a modern system should be set up.
Posted Jan 8, 2016 15:59 UTC (Fri)
by johannbg (guest, #65743)
[Link]
In some cases it lead to unbootable system but in most cases things just failed gracefully ( not found errors in log if you paid attention to your bootup process ) after trials and errors and fixes to achieve that and then there where those mysterious bugs that could be traced back to /usr not being mounted when needed on bootup.
All that systemd has ever done was issuing a warning but I guess 2016 will be another year to blame systemd and everyone involved with for everything that is wrong the linux ecosystem and outside it...
Posted Jan 8, 2016 21:01 UTC (Fri)
by ovitters (guest, #27950)
[Link] (3 responses)
systemd supports split /usr. IMO unacceptable that you use wording such as "ram through", while not knowing that systemd just informs that stuff might not work. The merge is encouraged, but for various good reasons. Not the suggested "opinions" and that's it.
Posted Jan 8, 2016 22:59 UTC (Fri)
by walex (guest, #69836)
[Link] (2 responses)
So one of the reasons to merge / and /usr is because the dependencies of the programs in / are in any case massive, and that's largely because of stuff like systemd and the flock of other complicated stuff around it, and this bloating of functionality and dependencies is somehow not relevant... The dependencies of systemd:
Posted Jan 8, 2016 23:32 UTC (Fri)
by pizza (subscriber, #46)
[Link]
The problem of massive boot-time dependencies was already quite pronounced before systemd was ever conceived -- because a generic, capable-of-booting-every-config-under-the-sun-and-handling-errors system is pretty complicated.
Posted Jan 19, 2016 12:09 UTC (Tue)
by jond (subscriber, #37669)
[Link]
I'm not sure if you read that before you posted it, but it basically debunks the myth that systemd has too many dependencies.
Posted Jan 10, 2016 18:10 UTC (Sun)
by giraffedata (guest, #1954)
[Link] (14 responses)
Posted Jan 10, 2016 19:16 UTC (Sun)
by raven667 (subscriber, #5198)
[Link] (3 responses)
Posted Jan 10, 2016 21:14 UTC (Sun)
by giraffedata (guest, #1954)
[Link] (2 responses)
So how do you perceive the hierarchy? How are all the directories under /usr related to each other in a way they aren't related to the other directories in the root?
I can't really tell what situation you're envisioning with system binaries separated from the root, requiring a dozen filesystems to be mounted. I'm not even sure what you mean by system binaries - but it apparently doesn't include anything that would be needed to mount filesystems.
Posted Jan 11, 2016 1:48 UTC (Mon)
by raven667 (subscriber, #5198)
[Link]
> So how do you perceive the hierarchy?
All top level directories have their own unique purpose, local config state in /etc, local persistent state in /var, both of which can be on a read/write local root volume, /dev, /proc, /sys, /tmp are all virtual, representing current system state.
> How are all the directories under /usr related to each other in a way they aren't related to the other directories in the root?
/usr is for software (binaries and all attendent read-only data) from the distribution, /usr/local if needed is for locally maintained software outside of the distribution system. /opt if needed is for 3rd party software. On most systems all executable code from the distro is in /usr and completely identical between like systems.
> I can't really tell what situation you're envisioning with system binaries separated from the root, requiring a dozen filesystems to be mounted
Old school shared filesystems like NFS are back in again for distributing software to groups of machines (or VMs/containers) with like needs, like the day of diskless workstations. There are other cases where a clear separation between standard executables, default configs, and local modified state such as making use of exec/noexec or ro/rw mount options, having separate A and B software installs for atomic updates or simple reliable factory reset. Of course you can do these things without a consolidated /usr, it just requires more work.
> I'm not even sure what you mean by system binaries - but it apparently doesn't include anything that would be needed to mount filesystems.
It is already the case that any userspace tools needed for setting up the root or /usr filesystem have to be in the initramfs, where the bootloader can get them, only in the simplest of cases can a kernel mount root with zero userspace tools. No distro can tell their users that all filesystems which require modules or userspace setup are unsupported. Since the infrastructure halready as to exist to support this, there is no additional complexity to mount /usr at the same time as root.
Posted Jan 12, 2016 13:02 UTC (Tue)
by smcv (subscriber, #53363)
[Link]
/usr/*, /bin, /sbin, /lib* are static under normal system operation: they only change when you add, remove or upgrade software (with a possible exception for /usr/local, but that can easily be a mount point or symlink if desired). That's one of the reasons for considering them to be "more similar than they are different", which makes unifying them into /usr make sense.
The rest of the directories in the root (in particular /etc, /home, /var, /srv) are normally mutable: they change under normal circumstances (either normal system operation, or sysadmin reconfiguration).
/etc needs to be on the rootfs because it contains the machine-specific configuration to know what additional filesystems need mounting (/etc/fstab, /etc/systemd/system/*.mount). It should be separated from the static part of the system (/usr) because it's inherently machine-specific, and because changes in /etc need to be preserved across upgrades.
In embedded scenarios, it's very appealing to be able to carry out upgrades by running with /usr from partition A, unpacking a new /usr into partition B, rebooting with /usr from partition B, rebooting into A to roll back the upgrade if B fails, and exchanging the roles of the two partitions for the next upgrade. OSTree is similar, although it uses a variable number of btrfs subvolumes instead of a fixed number of partitions.
Posted Jan 10, 2016 20:22 UTC (Sun)
by mjg59 (subscriber, #23239)
[Link] (9 responses)
Posted Jan 11, 2016 12:21 UTC (Mon)
by tao (subscriber, #17563)
[Link] (8 responses)
And I really don't get how you can first say that not being able to have a read-only / is a problem, then say that having everything in /usr is an advantage since you can have / read-write :P
Posted Jan 11, 2016 16:10 UTC (Mon)
by raven667 (subscriber, #5198)
[Link]
Posted Jan 11, 2016 16:13 UTC (Mon)
by nybble41 (subscriber, #55106)
[Link] (6 responses)
That's because the parts that need to be read-only have all been moved from / to /usr, leaving only /etc and /var as part of the root filesystem. The problem with making / read-only to protect /bin and /lib was that you needed a read-write filesystem for /var, and possibly for /etc as well.
Posted Jan 11, 2016 16:49 UTC (Mon)
by giraffedata (guest, #1954)
[Link] (5 responses)
And there are many storage management reasons that the parts of a system that change a lot should be in a separate filesystem from the parts that change only as part of system administration events, and /var is perfect for that. To me, that's the very definition of /var, which fuels my belief that / should be r/o, which removes one of the reasons to have /usr .
Posted Jan 11, 2016 17:01 UTC (Mon)
by mjg59 (subscriber, #23239)
[Link] (4 responses)
There's basically three regions of mutable state on a normal Linux system - /home, /etc and /var. They're all in /, so it's easier to retain / as read-write and push the non-mutable content to /usr than it is to have / as read-only and either move all the mutable content elsewhere or have some magical procedure to update it. There are certainly configurations where moving everything to / would work out fine, but moving things to /usr instead makes it possible to do some things that are otherwise difficult without interfering with anything other than certain aesthetic beliefs.
Posted Jan 12, 2016 4:30 UTC (Tue)
by giraffedata (guest, #1954)
[Link] (3 responses)
It's also my experience that /home gets split out into its own filesystem a lot more than /usr does, but there is probably some case where it makes sense to keep /home in the root filesystem even though it's not a single-filesystem system.
Posted Jan 12, 2016 4:32 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link] (2 responses)
It's how all of our users run, and that's a pretty large number.
Posted Jan 13, 2016 3:55 UTC (Wed)
by giraffedata (guest, #1954)
[Link] (1 responses)
Are you talking about a network filesystem? Because "the network" is not a storage medium, so that's confusing.
Posted Jan 13, 2016 4:00 UTC (Wed)
by mjg59 (subscriber, #23239)
[Link]
No. /usr is backed by dm-verity, so from the filesystem perspective it's a read-only block device.
Posted Jan 10, 2016 10:57 UTC (Sun)
by Yenya (subscriber, #52846)
[Link] (9 responses)
Posted Jan 10, 2016 14:08 UTC (Sun)
by raven667 (subscriber, #5198)
[Link] (5 responses)
Posted Jan 10, 2016 17:48 UTC (Sun)
by bronson (subscriber, #4806)
[Link]
Too funny. You can never go home.
Posted Jan 11, 2016 12:56 UTC (Mon)
by Yenya (subscriber, #52846)
[Link] (3 responses)
Of course. Now why this is a problem? Why it is necessary to add another level of indirection to the "dummy" root filesystem instead of having the root filesystem with the binaries, libraries, data files, etc., and have _other_ types of data as mountpoints? After all, this is what has been done to the /var/run directory as well. And we need the binaries, ld.so, etc. in the early stages of boot, so it does not make sense (to me)
Posted Jan 11, 2016 15:29 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (2 responses)
“The early stages of the boot” have by now pretty much been taken over by the initramfs, so being able to put stuff into /bin to have it available “from the start” is no longer a viable argument except as far as the most simplistic deployments are concerned.
In addition, not putting all binaries, libraries, etc. directly onto the root file system makes it possible to have a very small root file system (possibly in RAM) which should appeal, e.g., to appliance makers.
Posted Jan 11, 2016 15:47 UTC (Mon)
by Yenya (subscriber, #52846)
[Link] (1 responses)
* having a small filesystem (possibly in RAM) for / and mounting binaries from r/only media to /usr,
The only (albeit minor) difference is that the second approach has shorter paths (/lib instead of /usr/lib, etc.).
Also, my argument is not "to have stuff in /bin available from the start" (it can be surely be done for binaries in /bin as well as in /usr/bin), but to avoid having a separate root filesystem and longer paths for no good reason.
Having r/only binaries and r/w variable data on separate filesystems is a valid requirement for some systems. However, this does not imply that the / directory should belong to the former or to the later.
Posted Jan 12, 2016 12:44 UTC (Tue)
by smcv (subscriber, #53363)
[Link]
OK, so you mount your root filesystem, you look in /etc/fstab for the machine-specific list of what else to mount, oops! you don't have /etc yet.
That's why /etc is required to be on the root filesystem in Debian (and, as far as I know, in all general-purpose distributions). The first filesystem you mount (plus the initramfs, if present) must contain enough configuration to mount the rest. In Debian's support for having the initramfs mount a separate /usr, the initramfs mounts / first, and reads /etc/fstab to find out where /usr is.
It would be possible to hard-code the partition layout in the initramfs (for example by copying /etc/fstab into it), but it seems better to minimize the set of system configuration changes that require an initramfs rebuild before they will actually take effect.
Posted Jan 11, 2016 23:10 UTC (Mon)
by Karellen (subscriber, #67644)
[Link] (2 responses)
To start a new container, you can create your new "root" directory, with (nearly?) empty /var, /home and /etc, mount an appropriate /dev, /proc and /sys, then put a tmpfs on /tmp and /run, and finally bind-mount /usr read-only.... and that's it! Your chroot/jail/container-of-choice is ready to go.
OK, mounting a single /usr is only slightly easier than mounting /bin, /lib and /sbin separately; but the container angle is the one that let me see how /usr is a logical unit of the filesystem that makes sense to have its own top-level subdir.
Posted Jan 12, 2016 9:09 UTC (Tue)
by Yenya (subscriber, #52846)
[Link] (1 responses)
Posted Jan 13, 2016 0:23 UTC (Wed)
by raven667 (subscriber, #5198)
[Link]
Posted Jan 10, 2016 18:55 UTC (Sun)
by ksandstr (guest, #60862)
[Link] (1 responses)
Furthermore, the discussion appears to be moving exactly as the systemd discussion did: a change is proposed for the sake of vague[0] dogmatic benefits, described as optional, and then retention of an equal opt-out is argued coercive with regard to the innocent bystander in every package maintainer. This same pattern happened wrt init.d scripts and systemd: supposedly, maintaining a by-the-numbers start-stop-daemon script was horribly onerous times a thousand, or a requirement to the effect a violation of whichever principle was required -- and being an argument from principle, there's no need to quantify. The systemd developers may be arguably separate from "merged /usr" advocacy, but the same PR is apparent in both.
[0] in the sense that even a basic analysis of the impact of making such a change in the first place is missing, handwaved away with "sufficient testing" that's already done and gone.
Posted Jan 10, 2016 20:12 UTC (Sun)
by raven667 (subscriber, #5198)
[Link]
It should be pointed out that almost the entire purpose of distributions like Debian is to blur or remove this distinction, everything is treated as part of the system because it comes from one place. There has been no success in defining a bright line between system binaries and applications like LSB and no success in policing / and /etc to prevent tools from depending on /usr to be mounted to boot the system anyway.
> removing the option of separated root and usr filesystems is bad
That is a severe misunderstanding, any conclusions following that premise are invalid. The proposal is to mount /usr using the same mechanism as to mount / in early boot, from an initramfs for example. The initramfs already has to support all the possible userspace tools that could be required to make a block device for the root filesystem, so either having /usr on the same filesystem as / or mounting it at the same time should be no additional complexity.
Posted Jan 11, 2016 2:37 UTC (Mon)
by mcmechanjw (subscriber, #38173)
[Link] (3 responses)
Ok, so they are looking at merging the root / and /usr directories, I don't think that getting rid of a forty year old hack and replacing one set of the directories with symlinks is bad.
Personally I would merge to the left and have /usr/bin a symlink to /bin etcetera.
Lets take a look at my root directory /
-> /bin - This contains the binaries necessary to boot the system these should be (all) the commands that ordinary users can run.
-> /usr - Where the rest of the system usually lives and the issue at hand.
-> /var - This was created to store writeable state on the Linux system I seem to remember that originally it could be regenerated, currently it it has things like the log files, spool files for printers and email and other stuff not good to discard.
I have let my experience at having a separate /usr in the past form some of the opinions above.
Looking at them I would likely aim for /lib32 -> /usr/lib32 and /lib64 absorbing /usr/lib64 with a symlink back /usr/lib64-> /lib64 or if that is too hard e.g. X11 then having symlinks in /usr/lib64 for the contents of /lib64 in those cases something has already hardcoded /usr/lib64.
Those large subsystems that are not needed for boot could live in /opt and have a symlink in /bin. e.g. firefox, libreoffice, X11. I acknowledge that we may be approaching the point where X11 is considered required for system repair, and I know it would be nice to have sometimes. But is it a gain for convenience or a loss to complexity by having the failsafe X11 for example in /bin. Many systems today easily have the entire Gnome/KDE desktop environment fit into the root device.
Most systems have a relatively large amount of storage so collapsing it all to the root directory makes some sense, if the root directory is not on a easily mounted drive/partition then a initramfs is nearly required and seems to be the recommended solution. If you already have to have it for root why not for /usr also...
The many different approaches that can be taken have driven up the complexity but having the simple case still be simple is a fine goal, the question is where to draw the line on simple though.
On the other hand if /bin -> /usr/bin and /sbin -> /usr/sbin occurs instead the only directories that still have their own content on / would be /etc and /root at which point moving the last two items with content over to /usr and declaring /usr the new root mounting the other stuff under what was /usr and be done with it because there will be nothing else on / but links or mount points.
Posted Jan 11, 2016 16:08 UTC (Mon)
by raven667 (subscriber, #5198)
[Link] (2 responses)
Posted Jan 13, 2016 16:39 UTC (Wed)
by mcmechanjw (subscriber, #38173)
[Link] (1 responses)
My suggestion at the end was to basically dump /usr/lib64 into /lib64 and dump /lib32 and /usr/lib32 together in either / or /usr since I hope that they are not involved in boot, and then have all the user binaries in /bin and all the root only binaries in /sbin. All with symlinks from where old software expects them to be...
I also noted and tried to explain that if I understood correctly the / directory would be nearly empty assuming:
Directories that need to exist but can be empty
Directories that are proposed to be merged
Remaining root level directories
This basically has the entire system on /usr except for the config files /etc and writeable area for applications /var
This would of course just push the problem of deciding what is needed at boot time into the initramfs.
As for the dependencies -- everything in / hopefully does not depend on anything in /usr or there will be times when things break.
I had intended the /opt comment to be more of a suggestion, anything under /opt can do pretty much as it wishes within its own directory tree without interfering with other software and /opt is the designated location for 3rd party software, which many of the large packages act like.
I have over the years seen people reporting quite a few read-only / systems often with overlay/unionfs on top.
I can imagine systems with a R/O root where merged /usr has become / and the R/O /etc contains how to find the real /etc to mount. (Openwrt model)
For myself either initramfs or /usr seems to have become the new root, the place you end up when fixing things. To be able to fix things commands must be present which defines where root is. Root as a state of booting where your commands are?
I still don't feel this is quite as clear as I would like but maybe it is clear enough... and it is taking too long to write ;)
Enjoy.
Posted Jan 14, 2016 4:48 UTC (Thu)
by raven667 (subscriber, #5198)
[Link]
Posted Jan 12, 2016 18:58 UTC (Tue)
by flussence (guest, #85566)
[Link] (3 responses)
With those locations cleared of distro-managed files, computer owners will be free to put their actual boot dependencies directly in there instead of relying on a (usually huge) black box to set everything up. Then it's just a matter of taking the initramfs out of the bootloader command to revert to the kernel default behaviour.
Posted Jan 16, 2016 9:24 UTC (Sat)
by walex (guest, #69836)
[Link]
<p>This and other comments make me suspect that a lot of the discussion above is based on some big misunderstanding of the situation, and possible changes, and in part because the article above seems a bit confusing to me, and I'll try to clarify:</p>
<ul>
<p>Now nearly all existing distributions still support having distinct block device containers for the <tt>/</tt> and <tt>/usr</tt> (and <tt>/usr/local</tt> and <tt>/var</tt>) hierarchies, even if some in that case mount <tt>/</tt> and <tt>/usr</tt> at the same time, which requires all the GRUB and Linux drivers needed to mount both <tt>/</tt> and <tt>/usr</tt> to be present in both GRUB's root filesystem <tt>/boot</tt> and in the Linux first-stage root filesystem the "initramfs".</p>
<p>The "merged-<tt>usr</tt>" proposal can be interpreted in different ways:</p>
<ul>
<li>Keep the ability to have distinct block device containers for <tt>/</tt> and <tt>/usr</tt> but merge the <tt>/bin</tt> <tt>/lib</tt> directories into the <tt>/usr</tt> hierarchy <em>and container</em>. Leaving in the <tt>/</tt> hierarchy only <tt>/etc</tt>, <tt>/dev</tt>, <tt>/sys</tt>, ...; the main purpose of this would be to shorten the default <tt>$PATH</tt> and <tt>$LD_LIBRARY_PATH</tt>.</li>
<li>Remove the ability to have distinct block device containers for <tt>/</tt> and <tt>/usr</tt> enforcing the assumption that whichever root block device is mounted at boot time contains both the <tt>/</tt> and <tt>/usr</tt> hierarchies. This can come in two flavours:
Posted Jan 16, 2016 9:27 UTC (Sat)
by walex (guest, #69836)
[Link] (1 responses)
This and other comments make me suspect that a lot of the discussion above is based on some big misunderstanding of the situation, and possible changes, and in part because the article above seems a bit confusing to me, and I'll try to clarify: Now nearly all existing distributions still support having distinct block device containers for the / and /usr (and /usr/local and /var) hierarchies, even if some in that case mount / and /usr at the same time, which requires all the GRUB and Linux drivers needed to mount both / and /usr to be present in both GRUB's root filesystem /boot and in the Linux first-stage root filesystem the "initramfs". The "merged-usr" proposal can be interpreted in different ways:
Posted Jan 16, 2016 22:55 UTC (Sat)
by raven667 (subscriber, #5198)
[Link]
Its not just to shorten the $PATH, it's also so that _all_ of the system binaries, which are interdependent due to shared library and data file dependencies, are mounted at the same time in a self-consistent way with the same mount options.
> Leave the / and /usr hierarchies as they are with distinct /bin etc. directories because what matters is mounting a single block device.
If you require all the binaries to be on the root filesystem then when having many systems with a shared binary directory it precludes the host specific variable data from being on a single filesystem (etc, var,home) , unless you add bind mounts to the mix.
Posted Jan 18, 2016 22:35 UTC (Mon)
by mcortese (guest, #52099)
[Link] (2 responses)
After the merge, /usr will include all and only the data provided by the distribution, with one notable exception: /usr/local. Somebody will then advocate moving it to /. Just imagine the symmetry in having a /local hierarchy that mirrors /usr: for example /local/bin would be the locally-installed equivalent of distribution-installed /usr/bin, and so on...
Posted Jan 18, 2016 22:45 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Moving most of the / into /usr definitely helps to clean it up. Of course, we'll have to use compat symlinks for the next 10 years or so, but there's at least a hope of getting them removed eventually.
Posted Jan 23, 2016 19:06 UTC (Sat)
by trenton (guest, #106520)
[Link]
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
«A major benefit I like is that it's a big step toward a simple, stateless system.»
«Specifically, you'd have /usr being the system (usable read-only), /var being mutable state, and /etc being configuration»
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
different people disagree on what constitutes a "minimal, streamlined" base.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
How does the kernel obtain the passphrase to decrypt an encrypted /? The network configuration to mount a / over iscsi? Interact with the TPM in order to attest system state to the user?
»I hope that there is no dispute that the starting point of the boot process must be a simple, limited configuration "container"/"filetree" of some sort like /boot or "initramfs"
»Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
2) Have /var, /home and /etc be separate partitions, and introduce additional complexity (how do you know where /etc is if /etc isn't already there?)
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Linux kernel is booted mounting /dev/sda1 as /, which contains a fully functional but "vanilla" / filesystem.
Anyhow in the current practice GRUB's root (first-stage root) /boot is inside an unencrypted partition distinct from that containing / (second/third stage root) and someone could replace anyhow its contents with suitably "improved" kernel images or "initramfs" snapshot images. I know people who understand boot processes and who keep /boot (only) on a USB storage device they carry with them at all times because they reckon that an encrypted / with an unencrypted /boot is not giving them peace of mind.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
«
The idea that very complex boot configurations can be made to "just work" by using very complex boot processes seems very stupid to me, and easy of recovery from issues seems of paramount importance.
Because otherwise if one has a very complex and brittle (because of several complex and rigid stages) boot process if it breaks the fix is the usual "Microsoft technology": reinstall. It is amazing how often recent distributions can only be realistically fixed, because of great love of "clever" complexity, by reinstalling from scratch.
«
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Consider the alternative of LILO loading a "first" stage Linux kernel mounting a / on the local disk, which then has a full environment to do the job currently done by GRUB with /boot and by the Linux kernel with the "initramfs" before pivoting.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
your “first-stage” Linux kernel must know how to get at “/”.
»This means it must have all sorts of disk controller drivers and file system implementations statically linked (not to mention the modules taking care of disk encryption, RAID, and LVM), because it can't load any modules from disk before “/” is mounted.
»you can't get away without an initramfs, which makes the initramfs the simple to load system (since it's pulled from /boot) that has all the tools,
»
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
The reason why Unix systems in former times managed to get away with a much simpler boot structure
is that the range of supported disk controller hardware, file systems, etc. was a lot smaller, and therefore it was reasonable for manufacturers to distribute kernels that had everything needed to get at “/” statically built into them.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
When BIOS was how x86 booted, GRUB was necessary to read filesystems, as no other code in the system provided that.
»code in the system»
to «to read filesystems
». Try to imagine: before GRUB (or LILO) happened UNIX and Linux kernel could be easily booted, and mount /usr from encrypted filesystems, from iSCSI volumes, etc. How could this happen if GRUB «GRUB was necessary to read filesystems, as no other code in the system provided that
»?LILO was mentioned previously, but LILO had one large fault: if you upgraded your kernel you had to rerun LILO, or everything broke
»Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Correct me if I'm wrong, but kexec wipes out the previous system state. To build such a system would require extra work to make kexec into a proper bootloader replacement. Considering GRUB has all the drivers I need to just boot my preferred kernel and reliably load an initial /, why would I replace the simpler GRUB with a more complex system?
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Can you have GRUB loading its kernel, configuration file and modules from a /boot being a BTRFS filesystem on an encrypted LVM volume on top of an MD RAID1 spanning 2 DRBD volumes?
What is the case for the additional complexity and fragility for having an "initramfs" that is a statically built and customized snapshot of a large subset of / and then "pivot" to a full / or in the latter-day proposal to a full /usr?
Preparing for a merged /usr in Debian
Can you have GRUB loading its kernel, configuration file and modules from a /boot being a BTRFS filesystem on an encrypted LVM volume on top of an MD RAID1 spanning 2 DRBD volumes?
»An initramfs is reasonably easy to construct and keep current and, if you're providing a Linux distribution, makes it unnecessary to use a kernel that has all sorts of drivers compiled statically to ensure wide hardware coverage
»sorts of drivers compiled statically
except those needed to mount the first-stage / filesystem from a local storage area, exactly as GRUB does it for boot, but unnecessarily. Having mounted / the stripped down Linux kernel can then load whichever modules and run whichever scripts may be needed to mount the /usr filesystem, or to load a new dynamically selected kernel from any storage medium whatever, with kexec and reboot into it.an initramfs makes it straightforward to do various things – such as network configuration in case your actual root file system is on a remote iSCSI volume, or obtaining a pass phrase for an encrypted root file system – that the kernel isn't really set up to do. How to do this in user space in an initramfs is quite well understood
»Preparing for a merged /usr in Debian
I.e. not you?
Preparing for a merged /usr in Debian
That objection is comically stupid, because I have proposed a setup where /usr gets mounted and the second stage kernel (if needed, and it will be rarely needed) gets booted by a full (if minimal) GNU/Linux system hosted inside a first stage / root filesystem
The proposed setup, which is the historical UNIX setup plus optional kexec, is to have a Linux kernel without needing any sorts of drivers compiled statically except those needed to mount the first-stage / filesystem from a local storage area
and prompting for passwords can be done by a trivial shell script in the mounted / filesystem, and does not need to be done by the UNIX/Linux kernel
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
The only things that can be disputed is whether mounting in sequence '/boot', "initramfs", '/', '/usr' are all necessary, and if not whether one should drop from that sequence '/boot' and "initramfs" or "/".
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Further history of Unix of /usr
Further history of Unix of /usr
to reduce seeking. Something which still had benefits with Linux in benchmarks, even without taking fsck(8)'s memory requirements into consideration (which IIRC were proportional to the square of FS size).
Further history of Unix of /usr
Further history of Unix of /usr
Further history of Unix of /usr
ram through the freedesktop.org/System D opinion of how a modern system should be set up.
systemd supports split /usr. IMO unacceptable that you use wording such as "ram through",
Further history of Unix of /usr
Further history of Unix of /usr
What I don't get about this transition is why the unified directory is /usr instead of / . About 10 years ago, I got sick of this redundancy and removed it from systems I built, but I did it by getting rid of /usr, except for some symlinks to accommodate a few hardcoded conventional filenames. And my / is read-only, which is wonderful. Why do people want to go the other direction?
Further history of Unix of /usr
Further history of Unix of /usr
Further history of Unix of /usr
it offends a sense of hierarchy to put everything in /usr separately at the top level
Further history of Unix of /usr
Further history of Unix of /usr
Further history of Unix of /usr
Further history of Unix of /usr
due to the wonderful awesome sauce called bind-mounts.
Further history of Unix of /usr
Further history of Unix of /usr
Incidentally, I believe /etc is just as deserving of being read-only as /bin and /lib (and on my systems, it is). In particular, it's something that ought to change only via manual administrative process.
Further history of Unix of /usr
Further history of Unix of /usr
I guess there's no limit to the kind of system configurations there can be, each with its own ideal filesystem hierarchy. I think a system that has all the system executables on a read-only medium, without having the root filesystem on that same medium must be pretty rare. But I'm sure one can imagine it.
Further history of Unix of /usr
Further history of Unix of /usr
Further history of Unix of /usr
I think a system that has all the system executables on a read-only medium, without having the root filesystem on that same medium must be pretty rare.
It's how all of our users run, and that's a pretty large number.
Further history of Unix of /usr
Why not the other way round?
Why not the other way round?
Why not the other way round?
Why not the other way round?
to have it on a separate filesystem anyway.
Why not the other way round?
Why not the other way round?
and
* having these r/only media mounted as / and mounting the said small RAM-based filesystem as /var or /etc.
Why not the other way round?
Why not the other way round?
Why not the other way round?
Why not the other way round?
Points not discussed
Points not discussed
Preparing for a merged /usr in Debian
That way my root directory is the base location for stuff.
-> /boot - This is by custom a mount point for a boot partition if one is present or contains the files for the boot loader directly if not present.
-> /dev - Nowadays a tmpfs mount containing the device files. If it is anywhere else I would expect things to break
-> /etc - Directory where all sorts of configuration information is dumped by custom, lots of stuff is expected to be here and is like to break if moved. Long long ago there were some unix systems with executables in here.
-> /home - Where the users home directories currently live this has moved from place to place over the years most software could live with it elsewhere if $HOME is set correctly and ~ expansion works
-> /lib - Already a symlink to lib64 though I suppose it could be to lib32 also.
-> /lib32 - These are the 32 bit libraries on my multi-lib system, In theory only those needed for boot need live here. Though what 32 bit libraries are needed during boot is a interesting question.
-> /lib64 - These are the 64 bit libraries on my multi-lib system, In theory only those needed for boot need live here, but determining what exactly is needed is like a game of whack-a-mole.
-> /lost+found - Created by mkfs or fsck to hold any files that lose their directory entries.
-> /media - The current location to hold dynamically mounted file systems like CD/DVDs and USB sticks.
-> /mnt - An older directory under which file systems were mounted. A second hard drive for example.
-> /opt - A middle aged directory where stuff not part of the base system gets stuffed it has not been terribly popular but has things like Java or Google's chrome installed into it. It seems very free form usually with the /opt/$PROJECT or /opt/$VENDOR having its own subtree I have seen /opt/*/bin,lib,sbin and even /opt/*/etc,usr lurking about, just to confuse matters. I have never seen a system where this was needed for boot and often it is mounted from elsewhere.
-> /proc - Dynamic kernel file system with the older kernel interfaces. Lot of stuff expects it to be there, much breakage if moved.
-> /root - Home directory for root not placed under home since home may not be mounted due to breakage and you still want to login as root to try and fix it. Logins often fail if your home directory is missing. This can just be a empty directory and most login stuff will be happy.
-> /run - Directory that accumulates temporary state information pid files for example often works best if destroyed on boot this is sometimes a tmpfs and sometimes a separate partition, and often just a directory.
-> /sbin - Theoretically the system tools needed to get the rest of the system up that can only be used by root. Some commands used by regular users occasionally end up here like arp. This leads to quite a bit of whack-a-mole since people come up with very complex ways to have their system setup, encryption, lvm, dm/md, nfs, and if it involves networking the network tools are also needed before /usr.
-> /selinux - Where the user tools expect the selinux kernel system to be mounted if you have it. Common when security is a consideration.
-> /sys - The new dynamic kernel file system where new kernel interfaces are expected to be. Lots of stuff expects it to be there, and breaks when it isn't.
-> /tmp - The temporary directory this is sometimes a tmpfs and sometimes a separate partition, and often just a directory. This is where lots of stuff sticks temporary files and has over the years generated several entire classes of security problems. For higher security some systems have a private per user TMPDIR or do one of the clever tmpfs private mounts.
->> /usr/bin - Traditionally has the remaining binaries that could be used by regular users and were not needed at boot before /usr is mounted - what can go here is what is left after the whack-a-mole cycles above.
->> /usr/games - Traditional place for games to be installed in, they are seldom called by other software and so could likely move with less trouble, and I don't think any of them are needed for boot.
->> /usr/include - Traditional location for header files used when compiling. I would not expect them to be needed during boot
->> /usr/lib - another symlink in this case -> lib64 so it goes to /usr/lib64
->> /usr/lib32 - These are the 32 bit libraries on my multi-lib system, usually I would not use a 32 bit library during boot but then again see whack-a-mole.
->> /usr/lib64 - These are the 64 bit libraries on my multi-lib system, but here again what is left after whack-a-mole can be here, if any of these are needed during boot before /usr is mounted they need to be in /lib64
->> /usr/libexec - Location for executables not intended for users or scripts i.e. private to the installed program/
->> /usr/local - Often used to store software locally compiled, a tool written by the user or corporate software for example.
->> /usr/sbin - Traditionally has the remaining binaries that could only be run by root and in theory were not needed to mount /usr. Once again some commands used by users have leaked in ifconfig for example.
->> /usr/share - Shared information for example man pages, this is often not system or architure specific and so can more easily be share than other parts of the system.
->> /usr/src - By its name the location for source code often only the Linux kernel is the only source here, not needed for boot I hope.
->> /usr/tmp - On my system this is a symlink off to /var/tmp
->> /usr/x86_64-pc-linux-gnu - The tuple (despite of having 4 parts not 3) that defines the gnu tool compiler choices, there is usually a similar set under /usr/lib/x86_64-pc-linux-gnu/{gcc,binutils} allowing for cross compilers for example to live next to each other and only sometimes stomping on top of each other.
->> /var/run - A symlink off to /run this was previously where for example the pid files lived before /run
->> /var/lock - A symlink off to /run/lock where the lock files now live
->> /var/tmp - Yet another /tmp directory
Whether desired for read only mounting or over a network having /usr separate adds complexity, which has its costs.
I would also consolidate /usr/sbin -> /sbin and /usr/bin -> /bin and add the symlinks /usr/sbin -> /sbin and /usr/bin -> /bin and see about getting rid of all the /tmp, /usr/tmp and /var/tmp directories just for security purposes most likely using the private mount version.
But that has merely moved the problem over to the initramfs which now has to have all the tools needed to mount the drives and the same problems apply...
I have used tinycorelinux and it fits everything into the initramfs at about 15M including X and then loads everything else later and is a fine example of the small end of the spectrum.
On the other end we have the tightly integrated desktop environments where everything has been polished so it is all connected to the maximum extent possible, in order to give the most seamless experience.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
The wall of text after my introductory sentences was my analysis of how a standard Linux system is already configured. I checked both debian and gentoo boxes.
I had made no changes just commented in detail on the current layout.
I believe you ended up referring my commentary as "so you've created a complex, incestuous set of..."
I did not create it, and I do accept your description.
Mount points or frequently mount points
/boot
/dev
/home
/media
/mnt
/opt
/proc
/run
/selinux
/sys
/tmp
/lost+found
/root
/bin
/lib32
/lib64
/sbin
/etc
/var
to me it appears that the / directory would no longer be needed just mount /usr as root and mount /etc & /var on top with all the other directories that are already mounted on / on the new root that was formerly /usr.
The former /usr could be read only and only things mounted upon it have r/w permissions, which I regard as the strongest argument in favor of having everything under /usr or the former /usr as root /
Which will require everything needed to mount /usr to be in the initramfs -- more or less the same requirements that the root file system used to have...
Those programs on /usr can depend on anything in /usr and anything in / right?
I did not see how the proposal insures that everything under /usr only depends on other files under /usr to work, unless you mean that there is nothing left in / for it to depend on (excluding /etc & /var).
I view Openwrt as one quite popular one. This whole argument might be a naming problem if it was called the great R/O merge if might have produced less comment, though that name might still not be good enough.
Or a system that leaves the initramfs in place and mounts /usr R/O in its traditional location of /usr (like a variant on TinyCore)
The merge makes the classic case of mounting root R/W first and then /usr R/O as not nearly so useful, when things fail it would either be in the initramfs (which would feel like the old root in this case limited tools etc.) or in the root (/) where there are no commands until /usr is present, therefore it would be better to have /usr first and what was / can come later.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
<li>A "filesystem" is a tree of directories and files which is stored in its entirety in a container which is as a rule a "block device" (there are special cases but let's skip them).</li>
<li>A "hierarchy" in UNIX is a tree of directories and files that have a conventional organization, for example one where at the top there are <tt>/bin</tt>, <tt>/etc</tt>, <tt>include</tt>, <tt>lib</tt>, <tt>doc</tt>, <tt>/src</tt> etc. subtrees, because the UNIX convention is to store files by type.</li>
<li>Traditional UNIX had first one hierarchy under <tt>/</tt>, then a second one was added under <tt>/usr</tt>, and then a third one was added under <tt>/usr/local</tt>; the hierarchies were nested inside each other instead of being mounted alongside each other, as for example <tt>/core</tt>, <tt>/usr</tt>, <tt>/local</tt>.</li>
<li>Eventually differently-shaped sub-hierarchies were added under <tt>/var</tt>, and at different points in time under <tt>/usr/X11</tt>, <tt>/usr/net</tt> etc.</li>
<li>Traditionally the <tt>/</tt> and <tt>/usr<tt> hierarchy were independent filesystems, where the <tt>/</tt> hierarchy contained the core tools needed to get to the "block device" containing the <tt>/usr</tt> hierarchy and mount it.</li>
</ul>
<ul>
<li>Leave the <tt>/</tt> and <tt>/usr</tt> hierarchies as they are with distinct <tt>/bin</tt> etc. directories because what matters is mounting a single block device.</li>
<li>Also merge the <tt>/bin</tt> <tt>/lib</tt> etc. directories into the <tt>/usr</tt> hierarchy because if they enforced to be in the same container this is an opportunity to reduce the length of the default paths.</li>
</ul>
</li>
</ul>
Preparing for a merged /usr in Debian
shoving the contents of /{s,}bin and /lib into /usr will, somewhat paradoxically, *decrease* the need for an initramfs:
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian