Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Posted Jan 8, 2016 22:48 UTC (Fri) by walex (guest, #69836)In reply to: Preparing for a merged /usr in Debian by luto
Parent article: Preparing for a merged /usr in Debian
«A major benefit I like is that it's a big step toward a simple, stateless system.»
You know that "simple" in this discussion to me sounds really funny.
Because the Very Important People who have decided that merging is Just Like Microsoft Does It And Therefore Is Good have forgotten two far from trivial sources of huge complexity:
- A bootloader like GRUB has hundreds of modules and in effect is a mini-OS residing in /boot/, and it is much, much bigger and far more complicated than the original UNIX (or Linux) system was.
- The "initramfs" is also considerably far larger and more complicated than the original UNIX (or Linux) system was, and requires a lot of special case building and tools.
The original UNIX system had a / of 5MiB, and a /usr of 20MiB, and as the name says it was originally for user home directories, and later it got the traditional /usr/bin and /usr/lib for user-installed software, which then became sort of standard.
Today there is simply no point in having something of the extraordinary complexity and fragility of something like GRUB or the "initramfs": all we need is a simple / partition with a simple default filesystem, a kernel with that filesystem type built-in, and a bootloader that just loads that kernel.
Ah we would also need simple, static binaries and a simple init that just spawns a service supervisor (or a shell or whatever) to make the boot process indeed simple and robust. But that aspect of the story is long lost.
Because the clever people (many of them working hard inside Debian and Fedora) who love complexity and fragility want to have base tools and init systems dependent on DBUS or a dozen shared object libraries, and are worried about the complications of keeping / and /usr, but obviously are perfectly comfortable with GRUB and "initramfs".
«Specifically, you'd have /usr being the system (usable read-only), /var being mutable state, and /etc being configuration»
But shouldn't then /etc/ itself be under /var? And what about a different classification for top-level hierarchies, based on whether they are host-specific or site-specific? For example having a host-specific / hierarchy and a site-wide /usr one? :-)
Posted Jan 8, 2016 22:55 UTC (Fri)
by luto (guest, #39314)
[Link] (44 responses)
You're assuming that your "simple / partition" can be loaded by just passing the root device path to a kernel. If it's a plain partition on a plain ol' disk, that's probably the case. If it's on LVM or RAID or uses dm-verity or sits on an SPI flash chip because you're net-booting and don't want a disk at all, then this isn't going to work.
Initramfs is, in fact, a simple filesystem containing /. (It's not the *final* /, but that's more or less irrelevant, except insofar as this tends to cause dbus-daemon to explode, but that's purely dbus' fault.)
Given that you need the bootloader to find the kernel image no matter what you do, it's not much of a stretch to ask the bootloader to use the same facility to find initramfs, and everything else is fine after that.
> But shouldn't then /etc/ itself be under /var?
Probably not. In the ostree model, /etc merges in changes from updated packages, whereas /var does not.
Posted Jan 10, 2016 3:36 UTC (Sun)
by rqosa (subscriber, #24136)
[Link]
> If it's a plain partition on a plain ol' disk, that's probably the case. Even then, it might not be possible to mount the / partition without an initramfs, if parts of the kernel needed to access the disk in question were compiled as modules instead of built-in. (This used to be a common problem with booting from SCSI drives, and SATA drives before AHCI was commonplace — different host-controller chips required different drivers, and distributions don't want to have to build all of these drivers into the kernel statically, so they instead build the various drivers as kernel modules and provide a program to generate an initramfs image at kernel-install time with whichever modules are needed for the specific hardware in question.)
Posted Jan 14, 2016 10:11 UTC (Thu)
by hitmark (guest, #34609)
[Link] (42 responses)
The people calling the shots are so far up in devops lala land (architecture astronauts do not even begin to cover it) that they should really be forked off and left to make their own distros rather than foist their world onto those that has been around for a decade or longer and are quite happily doing it their way.
Posted Jan 14, 2016 13:13 UTC (Thu)
by pizza (subscriber, #46)
[Link] (41 responses)
I'm sorry, I don't follow. The OP is saying that real-world Linux systems are already quite complex (due to the sheer variety of real-world hardware and deployment options), and you turn around and claim that architectural work to accommodate how folks in the real world deploy Linux on real hardware is a sign that the folks doing the design and implementations to make it all possible, supportable, and generally JustWork(tm) are divorced from reality?
*shakes head* If you have an actual point, I'd love to hear it, because at this point you're basically complaining that water is wet.
Posted Jan 14, 2016 18:39 UTC (Thu)
by hitmark (guest, #34609)
[Link] (17 responses)
Never mind if X is an option that has been with the very concept of *nix since day one. But is now being taken away because it may get in the way of someone doing Y. Even though if you do Y you have had the option of foregoing X for ages.
Or maybe we are far beyond the point where trying to build one generic distro image is possible, and continuing to strive for it is a fools errand.
How about this, provide a minimal, streamlined base, and then let people build from that. Rather than try to placate every last "potential" out of the gate.
Posted Jan 14, 2016 18:55 UTC (Thu)
by pizza (subscriber, #46)
[Link] (16 responses)
The hilarious bit is providing the former is exactly what they're trying to do. It turns out that different people disagree on what constitutes a "minimal, streamlined" base.
Posted Jan 14, 2016 23:11 UTC (Thu)
by raven667 (subscriber, #5198)
[Link]
Posted Jan 16, 2016 14:32 UTC (Sat)
by walex (guest, #69836)
[Link] (14 responses)
Indeeed and I think that we can all agree that having a 4 stage boot process with 4 different "root" filesystems (/boot, "initramfs" snapshot, /, /usr) is too complex and fragile.
Given that the end goal is having /usr mounted, that cannot be removed, so the alternatives are: To me dropping / and keeping the far more complex and fragile /boot and "initramfs" stages seems solving the wrong problem at best.
Posted Jan 16, 2016 19:49 UTC (Sat)
by mjg59 (subscriber, #23239)
[Link] (12 responses)
How does the kernel obtain the passphrase to decrypt an encrypted /? The network configuration to mount a / over iscsi? Interact with the TPM in order to attest system state to the user?
Posted Jan 18, 2016 23:14 UTC (Mon)
by walex (guest, #69836)
[Link] (11 responses)
« In exactly the same way GRUB currently prompts for the passphrase to mount an encrypted /boot, the same way GRUB currently gets the network configuration to mount /boot over iSCSI, and using the same way GRUB uses to interact with the TPM to attest system state to the user. Equivalently all the same ways all those things are currently done by the Linux kernel booted by GRUB to mount an encrypted "initramfs", to get the network configuration to mount an "initramfs" over iSCSI, and for the "init" processor in the "initramfs" to attest the system state to the user. :-). Isn't that totally obvious? Didn't I write very clearly that «
If you dispute that, this discussion is pointless like most discussions with the members of the poetteringista tendency, because evidently the understanding of how booting works and the role of the / filesystem in UNIX/Linux systems has been completely lost.
Posted Jan 18, 2016 23:31 UTC (Mon)
by mjg59 (subscriber, #23239)
[Link] (10 responses)
Things that grub doesn't do. And nor does the kernel. But an initramfs does. So we use the solution that exists, and we'll continue doing so until somebody dislikes it enough to come up with an alternative that works better.
Posted Jan 19, 2016 8:53 UTC (Tue)
by walex (guest, #69836)
[Link] (9 responses)
The solution that exists and has existed for decades and has worked well for decades is to mount am ordinary '/' filesystem at first-stage boot and use that to mount '/usr' (encrypted over iSCSI using TPM if necessary) for second stage boot. If you think that is more complicated and fragile than to mount first '/boot', then "initramfs", then '/', then '/usr', and that in such a sequence the clever move is to drop '/', good luck.
Posted Jan 19, 2016 9:02 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link] (3 responses)
That's certainly a thing you could do, but now you have two choices:
1) Have /var, /home and /etc be part of /, and so be unencrypted
Both of these choices are bad, and so your solution is a bad solution and has not worked well for decades.
Posted Jan 19, 2016 19:25 UTC (Tue)
by walex (guest, #69836)
[Link] (2 responses)
It seems that I need to explain again the basics of how the boot process works and how UNIX/Linux sytems wor k in repect of the mount and pivot_root system calls, I'll try to keep it short: The outline boot sequence can therefore well be: That the above, which seems to me basic system knowledge, needs explaining again and again in detail is quite frustrating; as I wrote a few times already, the first-stage boot / (or kernel) does not need to be the very same as the the second-stage / (or kernel), and indeed in the current arrangement there are another two. Plus there is mount --bind for an alternative option. Anyhow in the current practice GRUB's root (first-stage root) /boot is inside an unencrypted partition distinct from that containing / (second/third stage root) and someone could replace anyhow its contents with suitably "improved" kernel images or "initramfs" snapshot images. I know people who understand boot processes and who keep /boot (only) on a USB storage device they carry with them at all times because they reckon that an encrypted / with an unencrypted /boot is not giving them peace of mind.
Posted Jan 19, 2016 19:38 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
Posted Jan 20, 2016 9:00 UTC (Wed)
by anselm (subscriber, #2796)
[Link]
This is essentially what happens with initramfs, except that the initramfs is loaded using the boot loader/system firmware, and can in principle come from anywhere (including a remote server). Your approach forces you to have the first-stage “/” available locally as an actual file system on a storage medium, which sucks if you want to have a diskless system.
You still need to ensure that whatever is on the first-stage “/” matches the rest of the system, just like with an initramfs. The difference is that the process of making initramfses is fairly well-understood and there are good tools for it already.
Two points:
Posted Jan 19, 2016 11:56 UTC (Tue)
by jond (subscriber, #37669)
[Link] (4 responses)
Posted Jan 19, 2016 16:54 UTC (Tue)
by paulj (subscriber, #341)
[Link] (3 responses)
Not all the world is anglo-phile.
Posted Jan 19, 2016 16:57 UTC (Tue)
by paulj (subscriber, #341)
[Link]
«This is common in France apparently»
Posted Jan 20, 2016 14:22 UTC (Wed)
by jond (subscriber, #37669)
[Link] (1 responses)
Posted Jan 20, 2016 17:27 UTC (Wed)
by apoelstra (subscriber, #75205)
[Link]
Posted Jan 19, 2016 12:39 UTC (Tue)
by nix (subscriber, #2304)
[Link]
Now linked-in initramfses are, admittedly, not what distros use, but they still work (and if they stop working people like me will yell bloody murder so they should keep working!)
Other benefits of using an initramfs include that the terrible hack whereby fsck has to work on mounted filesystems and then you sometimes have to crash-reboot is simply gone: fsck of filesystems needed by early boot, be that /, / and /usr, or whatever, is just like fsck of everything else. Plus, of course, it means that those of us using setups you consider intolerably unusual such as RAIDed / can use them without effort. Heck, I've suddenly switched an existing system from RAID to RAID-partially-over-NBD in the past when a sudden disk failure left me bereft on one machine but with a superfluity of storage on another: without an early userspace, this would have been an entirely hopeless quest without support in the *kernel* for such a baroque configuration, which is unlikely in the extreme.
This was not something I did recently, either -- it was way back in 2005. This configuration has been useful for over a decade now. Recent architecture astronautics it is not.
Posted Jan 15, 2016 21:29 UTC (Fri)
by walex (guest, #69836)
[Link] (22 responses)
The OP is saying that real-world Linux systems are already quite complex (due to the sheer variety of real-world hardware and deployment options), and you turn around and claim that architectural work to accommodate how folks in the real world deploy Linux on real hardware is a sign that the folks doing the design and implementations to make it all possible, supportable, and generally JustWork(tm) are divorced from reality? What seems divorced to reality to me seems worrying a lot about the simplest and most robust aspect of booting, the mounting of /usr having mounted a full /, while the previous stages are both more complex/fragile and pointless. There are some distinct arguments here as to that: Because of what seems to me stupidity, currently most GNU/Linux distributions have a four stage boot process where the first three stages are quite more complex/fragile than just booting into a full Linux kernel that mounts a full / as per UNIX tradition: Now what's the point of eliminating the last step, which is the simplest and most robust, because if it fails one has available a full (if small) GNU/Linux system in the final / to recover while leaving the first three steps unchanged? Very little, it is just stupid. Consider the alternative of LILO loading a "first" stage Linux kernel mounting a / on the local disk, which then has a full environment to do the job currently done by GRUB with /boot and by the Linux kernel with the "initramfs" before pivoting. That's how UNIX worked originally, and is a sensible two-stage arrangement. Currently thanks to pivot/chroot and kexec the initial Linux on the initial / can replace itself or the initial / very easily. An "initramfs" is only needed for diskless workstations, and in that case a virtual disk image can be simply downloaded and treated as the /.
Posted Jan 15, 2016 22:24 UTC (Fri)
by pizza (subscriber, #46)
[Link]
[ citation needed ] (especially as my experience says that things are vastly more reliable and robust now then they used to be..)
> The first argument is whether very complex boot situations should be supported.
Yes, they absolutely must be.
> The second argument is: assuming that very complex boot situations should be supported, whether they should be supported with a simple robust first stage boot process into a full UNIX-like environment with a full / and then continue into a much more flexible second stage boot process that among other things may "pivot" to a new / or even "kexec"s a new kernel before mounting /usr and everything else.
Um, so how is that second-stage created? Unique for every system, or is it something more generic? If it's unique, who/what creates it, and how? If it's generic, how does it differ meaningfully from what we have today with the initramfs model?
> Now what's the point of eliminating the last step, which is the simplest and most robust, because if it fails one has available a full (if small) GNU/Linux system in the final / to recover while leaving the first three steps unchanged? Very little, it is just stupid.
You're correct, it is stupid -- Because it's highly unlikely that stage 3 will succeed but stage 4 will fail, and enforcing such a separation adds complexity for very little actual gain. Furthermore, the distinction between '/' and '/usr' has blurred to the point of being pretty arbitrary.
Posted Jan 15, 2016 22:53 UTC (Fri)
by anselm (subscriber, #2796)
[Link] (20 responses)
The problem with that is that your “first-stage” Linux kernel must know how to get at “/”. This means it must have all sorts of disk controller drivers and file system implementations statically linked (not to mention the modules taking care of disk encryption, RAID, and LVM), because it can't load any modules from disk before “/” is mounted.
This is the problem that initramfs (in its various evolutionary stages) was designed to fix. It basically lets Linux distributions provide bare-bones kernels that can find arbitrary modules at runtime without actually having access to “/lib/modules” on the root partition. The alternatives are (1) putting up with a large kernel that has all sorts of stuff statically built into it that you don't need, or (2) building your own kernel that is statically targeted to your specific hardware. That's how things were in the late 20th century, and the Linux community has largely moved past this because it is wildly impractical and the modern methods, though somewhat more complicated, are also much more versatile and, in the end, easier to maintain.
Posted Jan 15, 2016 23:19 UTC (Fri)
by raven667 (subscriber, #5198)
[Link] (19 responses)
Posted Jan 16, 2016 0:29 UTC (Sat)
by anselm (subscriber, #2796)
[Link]
If you have UEFI you can in principle do away with GRUB. GRUB is more flexible and can do a few things the usual UEFI implementations don't support, but reasonably recent Linux kernels can be booted from UEFI without GRUB.
Posted Jan 16, 2016 8:37 UTC (Sat)
by walex (guest, #69836)
[Link] (17 responses)
« This is exactly the same problem has GRUB has in getting to /boot for GRUB and for the kernel loaded by GRUB to get at the "initramfs" from the image loaded by GRUB for it. There is some dispute whether the full boot process should allow for arbitrary complexity, but I hope that there is no dispute that the starting point of the boot process must be a simple, limited configuration "container"/"filetree" of some sort like /boot or "initramfs". In the UNIX world and early pre-hipster :-) Linux the kernel itself and a simple / filetree performed the function of GRUB and /boot/"initramfs". With pivot/chroot and kexec a Linux kernel can perform that function again as a simple first stage of booting which can lead to an arbitrary complexity of second stage boot. « « Well it looks like both of these comments are based on a poetteringesque deep admiration for the Microsoft Highway to Complexity and Fragility or deliberate obtuseness, because the argument above is that instead of a simple full featured standalone / like UNIX had it is much simpler and more robust to have three of them: one as /boot for GRUB (or UEFI), one as "initramfs" for a Linux kernel, and finally, one merged with /usr. Again, there is hopefully no dispute that booting needs at least one simple first stage and then one can do a complicated second stage; the question is whether the first stage should be just LILO-installer + Linux-kernel + / or should be the whole list of GRUB-installer + GRUB-kernel + /boot + Linux-kernel + "initramfs", and whether all the drivers that are needed to support a complex second stage boot should be replicated in GRUB-kernel + /boot and Linux-kernel + "initramfs" and in Linux + merged-/usr.
Posted Jan 16, 2016 9:45 UTC (Sat)
by anselm (subscriber, #2796)
[Link] (15 responses)
The fallacy here is in assuming that because something worked in the 1980s it must work today and indefinitely into the future. (We see this a lot in people who question the need for something like systemd by claiming that System-V init is the solution to all problems because, you know, it runs shell scripts.)
GRUB is the way it is because it supports a wide variety of use cases apart from simply booting a Linux kernel. Distributions like it because it covers pretty much all deployment scenarios one is likely to run into in Real Life™, and it is usually easier to maintain a single solution that does everything you need than a variety of specific solutions that are all designed to optimise a step in a system's life cycle that happens fairly rarely in the first place. (Most people don't boot their system several times per hour.) As we've mentioned, it is perfectly possible to get rid of GRUB on a UEFI-based system, because the UEFI boot loader can boot a Linux kernel directly, so this makes your “first-stage boot” that much simpler already. Mainstream Linux distributions, however, can't rely on everybody having UEFI, which is why they still use GRUB because that works everywhere.
The reason why Unix systems in former times managed to get away with a much simpler boot structure is that the range of supported disk controller hardware, file systems, etc. was a lot smaller, and therefore it was reasonable for manufacturers to distribute kernels that had everything needed to get at “/” statically built into them. (As a matter of fact, traditional Unix kernels often didn't support the idea of loadable modules in the first place.) On contemporary Linux, there is no practical way of getting rid of “initramfs” that reliably supports things like encryption, RAID, or diskless systems and does not present a maintenance nightmare compared to which initramfs is really the simpler and more versatile solution. You end up trying to build distribution kernels that have everything in them statically on the off-chance (including stuff like a DHCP client that is better left to user space), and generally distributions are very happy to be past that, and to be able to use initramfs to offer modular, maintainable kernels. This means that you're unlikely to find anyone who actually deals with that sort of thing in actual practice who will agree with you that essentially going back to “pre-hipster” Linux is anything remotely like a good idea. In any case, the approach is fairly well-understood and rarely causes problems, so this is really a tempest in a teacup.
Posted Jan 16, 2016 14:09 UTC (Sat)
by walex (guest, #69836)
[Link] (14 responses)
That's a crazy starting point, because the assumed goal here is to cover a complex boot configuration, not to have a complex boot structure for its own sake. Complex structures don't bring any benefit in themselves, except the pride of a job poorly done :-).
There is no need to assume the that a complex boot configuration requires a complex, fragile, boot structure with 4 stages instead of a simple structure like "boot a stage one Linux kernel and / and then use that to do stage two of the boot process using the full power of a Linux system to boot a bigger kernel and mount a /usr filesystem".
But GRUB in order to mount /boot makes very strong and limiting assumptions about the range of possibilities it can cope with. Can you have GRUB loading its kernel, configuration file and modules from a /boot being a BTRFS filesystem on an encrypted LVM volume on top of an MD RAID1 spanning 2 DRBD volumes? If not, why are GNU/Linux users being so limited to the range of drivers statically built into the distributed GRUB kernel? :-) What is the case for the additional complexity and fragility of having GRUB + /boot replicating in complex yet limiting way a large part of the functionality of the Linux kernel and of /? What is the case for the additional complexity and fragility for having an "initramfs" that is a statically built and customized snapshot of a large subset of / and then "pivot" to a full / or in the latter-day proposal to a full /usr?
Posted Jan 16, 2016 16:34 UTC (Sat)
by MattJD (subscriber, #91390)
[Link] (4 responses)
Regarding initramfs and /, initramfs provides what / used to provide. Now / can require several steps before it mounts (for instance, on my laptop I need to type in my encryption passphrase). An initramfs can also provide a complete recovery shell, that relies on minimal other parts of the system to work. So what you propose already basically exists. The only difference is that you want a / that comes from a partition, whereas initramfs provides a / that comes from memory. Either way they need to be maintained, and neither provide a full system.
So I don't think there is a large gap between what both sides of this conversation want. It's just that we haven't done the cleanup to get closer to the desired simpler boot sequence.
Posted Jan 18, 2016 23:45 UTC (Mon)
by walex (guest, #69836)
[Link] (3 responses)
« It comes as amazing news to me that the UNIX and Linux kernels didn't have « « GRUB (the bootloader) has exactly the same fault! Please read the manual: it can only ever boot one statically configured (with grub-install) GRUB kernel. If you upgrade the GRUB kernel you have to rerun its (static) installation. It is the GRUB kernel that spawns its shell environment, loads perhaps dozens out of several hundred (most them entirely Linux specific) possible GRUB kernel modules from /boot, and run a system initialization script that loads a Linux kernel, a toy / snapshot in the "initramfs", and then passes control the Linux kernel. These are well described and documented booting steps, and I am astonished that so many people here seem to ignore them. This ridiculous and fragile complexity can be replaced by LILO replacing the GRUB bootloader, a "first stage" Linux kernel replacing the GRUB kernel, one that mounts a / directly instead of a pointless /boot or "initramfs", loads modules and scripts from it to mount /usr from anywhere, and load optionally a new kernel chosen dynamically by a user-interaction script, and kexec it. As I just wrote in another comment, it cannot be disputed that the booting sequence must involve: #1 a bootloader that can only load one statically configured kernel, #2 a simple filesystem mounted by that kernel using statically builtin drivers and configuration, and #3 a series of drivers, scripts and other executables run dynamically by the kernel loaded in #1 from the filesystem mounted in #2. There is simply no reason to have #1 as the GRUB stage1, #2, as the GRUB stage2 kernel, #3 as the /boot filesystem, then #2 again as a Linux kernel loaded from /boot with #3 again as the "initramfs".
Like in traditional UNIX and Linux systems this can all be far more easily arranged with #1 as LILO, #2 as a "first stage" Linux kernel with just (for example!) the VESA, serial, SATA, "ext4" drivers built in, mounting / from which any further driver modules, initialization scripts, etc. can be run to support arbitrarily complex boot configurations, including optionally loading a dynamically selected "second stage" Linux kernel to boot with kexec. This may be the fourth time I try to explain how boot sequences actually work and how a traditional UNIX/Linux boot sequence works and is more robust and flexible than involving needlessly both /boot and "initramfs" as replacement for /.... Note: as always, a completely storage-less system needs a RAM disk and to download a disk image into it to setup /, but that's then the job of the bootloader like PXE.
Posted Jan 19, 2016 0:27 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
grub stage 1 loads grub stage 2. Scripts in grub stage 2 allow the choice of kernel and initramfs, loaded via firmware provided drivers. initramfs contains a subset of /. initramfs mounts the real /.
What you're proposing is:
lilo loads the boot kernel. Boot kernel mounts a / (we'll call it "boot /") that contains a subset of the real /. Scripts there allow the choice of kernel and /, loaded via boot kernel provided drivers. After kexecing a new kernel, the new kernel mounts "boot /" again and executes further scripts that mount the real /.
In your proposal you end up mounting "boot /" twice, you have to keep "boot /" synchronised with the real /, your boot kernel is significantly more complicated than grub and can't make use of any firmware-provided functionality (making it less flexible), you're using lilo which doesn't work under UEFI (ie, modern systems) and you're relying on kexec which *doesn't work* in a number of real-world cases. Your solution is *more* complicated than what we have today.
Posted Jan 19, 2016 1:12 UTC (Tue)
by MattJD (subscriber, #91390)
[Link] (1 responses)
I apologize for the confusion. I was talking about IBM compatible BIOS. BIOS doesn't have, afaik, filesystem code. It can only read from hard drives (and depending upon age, limited sized drives!). Even Microsoft has a bootloader that loads the windows kernel. Before LILO, Linux must have had a different bootloader. Or it embedded the command line and fit in the first sector of the hard drive. Other Unix either had their own bootloader, or ran on other platforms which I don't know and won't speak for their system setup code.
UEFI, for instances, does contain file system code. And UEFI doesn't require GRUB to boot Linux from a filesystem. I'd bet two pennies that if we had UEFI from the start, GRUB would not be as popular and LILO would not held out so long, assuming all else being equal.
> GRUB (the bootloader) has exactly the same fault! Please read the manual: it can only ever boot one statically configured (with grub-install) GRUB kernel. If you upgrade the GRUB kernel you have to rerun its (static) installation.
A rarer event for me, and a step required as part of my system upgrade. Countless times I'd forget to rerun LILO when testing a kernel, and break the system. GRUB is far more resilient, even when I upgrade it. Anyways, no matter what you do there is always a risk upgrading bootloaders. Many people find GRUB, even with its complexity over LILO, more robust. I also find it much easier to debug failures with GRUB and recover systems.
> <snip> dynamically selected "second stage" Linux kernel to boot with kexec.
Also, some hardware is really unhappy about kexec. As a trivial example, my desktop GPU. It does not like being double initialized from the kernel, and I suspect it would take a lot of work to make it so for little gain (this is all on the official open source driver stack).
The boot process is always a pain on a system (my favourite example? The A20 line that needs to be dealt with in real mode). Everything is about trying to make it better. I won't argue the current system is the best thing possible (I know I'll be proven wrong eventually). I do think it's very good as is.
If you want to remove GRUB, UEFI allows that. I do see the value in making cut down systems that match your current system, by initramfs (and I've manually created an initramfs, including initial boot scripts. Interesting, but not unreasonable process. I used it for several years on an encrypted LVM /).
I'm willing to be shown that whole line of thought is wrong. I suggest you try creating your desired system, including all the necessary support structure. If it is better, I'd be happy to push it forward. As of right now, I just don't see how yours is better considering my experience with the Linux boot system.
Posted Jan 19, 2016 7:47 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
To an extent. UEFI only (typically) has support for FAT, so your kernel and initramfs need to live on FAT, and the compromises around using FAT as a meaningful filesystem tend to result in it being difficult to make it /boot. Grub is, perhaps depressingly, less awful than some of the compromises you have to make to launch kernels from UEFI directly.
Posted Jan 18, 2016 0:13 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (8 responses)
No, but in your proposed non-GRUB, non-initramfs setup you can't use LILO to get your initial kernel from there, either, so this is an obvious red herring. Try harder.
An initramfs is reasonably easy to construct and keep current and, if you're providing a Linux distribution, makes it unnecessary to use a kernel that has all sorts of drivers compiled statically to ensure wide hardware coverage, most of which drivers are unnecessary for any particular system (because the hardware isn't there) but can't be removed other than by compiling a new, customised kernel. (Remember that that was a situation we used to have to deal with in the early years of Linux, and now that we no longer need to, pretty much everyone is a lot happier.) This sucks especially if you're running an enterprise-class Linux distribution like RHEL or SLES where tampering with the kernel will mean your system no longer qualifies for vendor support in case of problems.
In addition, an initramfs makes it straightforward to do various things – such as network configuration in case your actual root file system is on a remote iSCSI volume, or obtaining a pass phrase for an encrypted root file system – that the kernel isn't really set up to do. How to do this in user space in an initramfs is quite well understood, and putting the required functionality into the kernel would only add unnecessary complexity and probably bugs.
Posted Jan 18, 2016 23:02 UTC (Mon)
by walex (guest, #69836)
[Link] (7 responses)
« No, but in your proposed non-GRUB, non-initramfs setup you can't use LILO to get your initial kernel from there, either, so this is an obvious red herring. Try harder. That objection is comically stupid, because I have proposed a setup where /usr gets mounted and the second stage kernel (if needed, and it will be rarely needed) gets booted by a full (if minimal) GNU/Linux system hosted inside a first stage / root filesystem, just like it has been done for decades by people who actually understand booting sequences, and that LILO in that setup is equivalent not to the GRUB kernel but to the GRUB bootloader that loads the GRUB kernel from the GRUB root filesystem which is /boot. I think that this is the third time that I have explained this is detail for those who ignore how booting sequences work.
« « People who understand how booting sequences work understand that doing all that with a stripped down UNIX/Linux kernel and a mounted / filesystem is much easier and less fragile than packing up every time a new static "initramfs", and prompting for passwords can be done by a trivial shell script in the mounted / filesystem, and does not need to be done by the UNIX/Linux kernel, and that there is no need to snapshot the / filesystem into an "initramfs" image unless there is a totally storage-less boot situation.
Posted Jan 18, 2016 23:17 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Please, desist from insulting people who actually do the work to make booting reliable and stable.
Posted Jan 18, 2016 23:37 UTC (Mon)
by anselm (subscriber, #2796)
[Link] (3 responses)
Right, so please explain how you load your kernel from your first-stage “/” if that is encrypted. Something must find and decrypt the kernel before it can run. (GRUB can do it but you don't want to use GRUB.)
Nobody disagrees that this is possible in principle. We all did it that way on Linux 1.x, 20 years ago. The main reasons we prefer not to do it that way anymore are that (a) using an initramfs is much less hassle than compiling a new custom kernel for every single machine that we want to run Linux on, and (b) an initramfs lets us do various things in a reasonably straightforward manner that in the traditional setup are very difficult or impossible to achieve at all. In any case, it turns out that the initramfs approach, in actual practice, is a lot less brittle than you seem to assume.
OK, so “/” is encrypted (it would after all be a very bad idea to leave it unencrypted, with all the important stuff that you want to put on it). Now how do you get at that trivial shell script in the mounted “/” file system when the kernel needs the pass phrase to mount “/” in the first place?
Posted Jan 19, 2016 8:42 UTC (Tue)
by walex (guest, #69836)
[Link] (2 responses)
In exactly the same way that currently GRUB gets its modules and scripts and the new kernel in the content of an encrypted "/boot" filesystem, which also contains «important stuff that you want to put on it». :-)
As I wrote several times, disputing that there must be a statically installed bootloader and kernel mounting a simple filesystem at the start of the boot sequence from which to load another kernel, drivers, configuration files and scripts to mount anything else, is just pointless and a really stupid debating trick. Because that's how boot sequences work by necessity (the second stage is necessary because of the flexibility requirements).
The only things that can be disputed is whether mounting in sequence '/boot', "initramfs", '/', '/usr' are all necessary, and if not whether one should drop from that sequence '/boot' and "initramfs" or "/".
Posted Jan 19, 2016 8:57 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
The grub configuration and contents of /boot aren't secret and so don't need to be encrypted, so you're proposing copying much of the contents of / into /boot?
Posted Jan 19, 2016 10:05 UTC (Tue)
by anselm (subscriber, #2796)
[Link]
The issue here isn't necessity as much as convenience. As I said earlier, the Linux community used to do initramfs-free boots based on LILO for quite some time in the 1990s. We've come off that idea not because it doesn't work (it does, at least for simple scenarios), but because on the whole it is inconvenient and restrictive.
The initramfs-based approach, even though it looks more complicated at first sight, is vastly more powerful and convenient, and the added complexity is a price that most Linux distributions seem more than happy to pay. After all, Linux distribution makers are not in the business of making their life more complicated just for the sake of making it more complicated; they prefer approaches that work in a wide variety of cases and can be maintained.
If you think you can come up with a method of booting Linux systems that is better than what we have now, feel free to implement it and show it off to distribution makers. If it really is superior to the current approach, they'll be more than happy to take it on. That method worked for systemd in the basic-plumbing arena, after all …
Posted Jan 19, 2016 1:52 UTC (Tue)
by raven667 (subscriber, #5198)
[Link] (1 responses)
We can disagree without being insulting or dismissive here, some of the people here actually build distributions and know more about booting than either of us do combined.
> I think that this is the third time that I have explained this is detail for those who ignore how booting sequences work.
I think we are going to disagree about how clear your ideas have been represented by your writing, the evidence is in how few people seem to have understood your intent. Let me see if I understand what you are proposing as another valid way to boot a system.
If I understand correctly you are thinking that we should go back to a simple LILO or uEFI style boot loader that is capable of loading the kernel and passing parameters from the firmware but to disaggregate what is currently in the initramfs into /boot, presuming that the kernel will have built-in drivers equivalent to what the firmware provides to be able to access the block device that /boot lives on, or that the kernel will use the firmware APIs to read the block device. Once you can reliably access the /boot block device you can run your minimal userspace to do hardware setup and pivot into the "real" system.
> is to have a Linux kernel without needing any sorts of drivers compiled statically except those needed to mount the first-stage / filesystem from a local storage area, exactly as GRUB does it for boot,
I may be wrong I don't think GRUB has block device drivers to access /boot, it relies on the firmware built-in services to access the block device, although it does have its own minimal filesystem driver modules (BIOS or uEFI or whatever), which are loaded LILO-style using static block lists, which has the same problems as LILO but GRUB modules are updated more infrequently than the kernel so is less of a problem in practice.
> plus optional kexec
As mjg59 has said elsewhere, and he has more operational experience with this feature than I do, it is a neat feature and great when it works, it's not something that is so reliable that you'd want to depend on it for the common case of everyday booting. In any event, if you are going to boot a Linux kernel, it might as well be the "real" one.
Posted Jan 19, 2016 7:55 UTC (Tue)
by mjg59 (subscriber, #23239)
[Link]
It does, but they're not typically used in BIOS or UEFI environments. I point this out merely for precise technical accuracy rather than it altering your point in any way!
Posted Jan 16, 2016 19:44 UTC (Sat)
by mjg59 (subscriber, #23239)
[Link]
Posted Jan 18, 2016 23:32 UTC (Mon)
by dirtyepic (guest, #30178)
[Link]
I always love it when the loonies preload their arguments like this and I know I can just skip the rest of the post and all the replies. Thanks big guy!
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
different people disagree on what constitutes a "minimal, streamlined" base.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
How does the kernel obtain the passphrase to decrypt an encrypted /? The network configuration to mount a / over iscsi? Interact with the TPM in order to attest system state to the user?
»I hope that there is no dispute that the starting point of the boot process must be a simple, limited configuration "container"/"filetree" of some sort like /boot or "initramfs"
»Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
2) Have /var, /home and /etc be separate partitions, and introduce additional complexity (how do you know where /etc is if /etc isn't already there?)
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Linux kernel is booted mounting /dev/sda1 as /, which contains a fully functional but "vanilla" / filesystem.
Anyhow in the current practice GRUB's root (first-stage root) /boot is inside an unencrypted partition distinct from that containing / (second/third stage root) and someone could replace anyhow its contents with suitably "improved" kernel images or "initramfs" snapshot images. I know people who understand boot processes and who keep /boot (only) on a USB storage device they carry with them at all times because they reckon that an encrypted / with an unencrypted /boot is not giving them peace of mind.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
«
The idea that very complex boot configurations can be made to "just work" by using very complex boot processes seems very stupid to me, and easy of recovery from issues seems of paramount importance.
Because otherwise if one has a very complex and brittle (because of several complex and rigid stages) boot process if it breaks the fix is the usual "Microsoft technology": reinstall. It is amazing how often recent distributions can only be realistically fixed, because of great love of "clever" complexity, by reinstalling from scratch.
«
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Consider the alternative of LILO loading a "first" stage Linux kernel mounting a / on the local disk, which then has a full environment to do the job currently done by GRUB with /boot and by the Linux kernel with the "initramfs" before pivoting.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
your “first-stage” Linux kernel must know how to get at “/”.
»This means it must have all sorts of disk controller drivers and file system implementations statically linked (not to mention the modules taking care of disk encryption, RAID, and LVM), because it can't load any modules from disk before “/” is mounted.
»you can't get away without an initramfs, which makes the initramfs the simple to load system (since it's pulled from /boot) that has all the tools,
»
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
The reason why Unix systems in former times managed to get away with a much simpler boot structure
is that the range of supported disk controller hardware, file systems, etc. was a lot smaller, and therefore it was reasonable for manufacturers to distribute kernels that had everything needed to get at “/” statically built into them.
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
When BIOS was how x86 booted, GRUB was necessary to read filesystems, as no other code in the system provided that.
»code in the system»
to «to read filesystems
». Try to imagine: before GRUB (or LILO) happened UNIX and Linux kernel could be easily booted, and mount /usr from encrypted filesystems, from iSCSI volumes, etc. How could this happen if GRUB «GRUB was necessary to read filesystems, as no other code in the system provided that
»?LILO was mentioned previously, but LILO had one large fault: if you upgraded your kernel you had to rerun LILO, or everything broke
»Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Correct me if I'm wrong, but kexec wipes out the previous system state. To build such a system would require extra work to make kexec into a proper bootloader replacement. Considering GRUB has all the drivers I need to just boot my preferred kernel and reliably load an initial /, why would I replace the simpler GRUB with a more complex system?
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Can you have GRUB loading its kernel, configuration file and modules from a /boot being a BTRFS filesystem on an encrypted LVM volume on top of an MD RAID1 spanning 2 DRBD volumes?
What is the case for the additional complexity and fragility for having an "initramfs" that is a statically built and customized snapshot of a large subset of / and then "pivot" to a full / or in the latter-day proposal to a full /usr?
Preparing for a merged /usr in Debian
Can you have GRUB loading its kernel, configuration file and modules from a /boot being a BTRFS filesystem on an encrypted LVM volume on top of an MD RAID1 spanning 2 DRBD volumes?
»An initramfs is reasonably easy to construct and keep current and, if you're providing a Linux distribution, makes it unnecessary to use a kernel that has all sorts of drivers compiled statically to ensure wide hardware coverage
»sorts of drivers compiled statically
except those needed to mount the first-stage / filesystem from a local storage area, exactly as GRUB does it for boot, but unnecessarily. Having mounted / the stripped down Linux kernel can then load whichever modules and run whichever scripts may be needed to mount the /usr filesystem, or to load a new dynamically selected kernel from any storage medium whatever, with kexec and reboot into it.an initramfs makes it straightforward to do various things – such as network configuration in case your actual root file system is on a remote iSCSI volume, or obtaining a pass phrase for an encrypted root file system – that the kernel isn't really set up to do. How to do this in user space in an initramfs is quite well understood
»Preparing for a merged /usr in Debian
I.e. not you?
Preparing for a merged /usr in Debian
That objection is comically stupid, because I have proposed a setup where /usr gets mounted and the second stage kernel (if needed, and it will be rarely needed) gets booted by a full (if minimal) GNU/Linux system hosted inside a first stage / root filesystem
The proposed setup, which is the historical UNIX setup plus optional kexec, is to have a Linux kernel without needing any sorts of drivers compiled statically except those needed to mount the first-stage / filesystem from a local storage area
and prompting for passwords can be done by a trivial shell script in the mounted / filesystem, and does not need to be done by the UNIX/Linux kernel
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
The only things that can be disputed is whether mounting in sequence '/boot', "initramfs", '/', '/usr' are all necessary, and if not whether one should drop from that sequence '/boot' and "initramfs" or "/".
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian
Preparing for a merged /usr in Debian