LWN: Comments on "Preparing for a merged /usr in Debian"

Preparing for a merged /usr in Debian

raliscenda — Wed, 23 Nov 2016 07:34:05 +0000

>Unfortunately the result of symlinking things, causes developers who hard code paths to have portability problems, going to more traditionally laid out systems.

Your assumptions are that GNU/Linux is only one favour of UNIX like system developers have to support. For sure all BSD based systems does not have plan to merge /usr in near future, so portability issue stay with us for a while.

Preparing for a merged /usr in Debian

trenton — Sat, 23 Jan 2016 19:06:59 +0000

That's why merging /usr in all popular distributions is particularly important. Only after that it will be possible to change shebangs in scripts from #!/bin/bash to #!/usr/bin/bash. After some (long) period of time it will be possible to remove /{bin,sbin,lib} top level directories, as shebangs will no longer point on them, reducing top-level clutter and achieving separation of concerns, as you proposed.

Preparing for a merged /usr in Debian

apoelstra — Wed, 20 Jan 2016 17:27:57 +0000

I think it'd have been better to say "use >" (which would indeed be better for readability than « or ») rather than "do what everyone else does". There are a bunch of quoting styles used on LWN, and my feeling is that «...» is easier to read than most of them, since these characters are distinctive and never used for anything but quoting.

Preparing for a merged /usr in Debian

jond — Wed, 20 Jan 2016 14:22:09 +0000

It's not a language issue, it's a website-issue. On LWN, people quote with ">", because it gets marked up nicely. Anything else is harder to read (and the OP has not even been consistent in using U+00BB etc.)

Preparing for a merged /usr in Debian

anselm — Wed, 20 Jan 2016 09:00:18 +0000

Linux kernel is booted mounting /dev/sda1 as /, which contains a fully functional but "vanilla" / filesystem.

This is essentially what happens with initramfs, except that the initramfs is loaded using the boot loader/system firmware, and can in principle come from anywhere (including a remote server). Your approach forces you to have the first-stage “/” available locally as an actual file system on a storage medium, which sucks if you want to have a diskless system.

You still need to ensure that whatever is on the first-stage “/” matches the rest of the system, just like with an initramfs. The difference is that the process of making initramfses is fairly well-understood and there are good tools for it already.

Anyhow in the current practice GRUB's root (first-stage root) /boot is inside an unencrypted partition distinct from that containing / (second/third stage root) and someone could replace anyhow its contents with suitably "improved" kernel images or "initramfs" snapshot images. I know people who understand boot processes and who keep /boot (only) on a USB storage device they carry with them at all times because they reckon that an encrypted / with an unencrypted /boot is not giving them peace of mind.

Two points:

Your approach doesn't help these people because your first-stage “/” can't be encrypted either, since the Linux kernel has no way of getting at a pass phrase to decrypt the first-stage “/”. They will have to put their first-stage “/” on a removable storage device just like they do now with “/boot”.
It is quite possible for GRUB to use an encrypted “/boot” (see, e.g., here), although it is not what distributions do by default.

Preparing for a merged /usr in Debian

mjg59 — Tue, 19 Jan 2016 19:38:12 +0000

If /dev/sda1 is used only for boot and then gets dropped, it's actually identical to an initramfs except none of the tooling to keep it in sync with the rest of the system exists and you can't use it for netbooting so it's precisely as complicated but less flexible and nobody's written the software that you'd need to make it work. And you still don't have a way to choose a kernel unless you're using kexec, which isn't an option. The reason we do things the way we do is that it works better than what you're proposing.

Preparing for a merged /usr in Debian

walex — Tue, 19 Jan 2016 19:25:35 +0000

It seems that I need to explain again the basics of how the boot process works and how UNIX/Linux sytems wor k in repect of the mount and pivot_root system calls, I'll try to keep it short:

It is impossible to avoid using a static kernel and a simple root filesystem to boostrap the process of getting to a complex-configuration / or /usr or anything else.
The only choice is which "static kernel" and "simple root" to use.
Currently the "static kernels" and "simple root" are the GRUB kernel (stage2) plus its root filesystem /boot and the Linux kernel it loads plus its stage 1 "simple root", the "initramfs" instantiated from the associated static snapshot loaded from /boot.
They can both be replaced with the Linux kernel and a / filesystem.
That / filesystem for stage 1 booting can be but does not need to be the stage 2 / filesystem anymore than the GRUB kernel /boot root needs to part of the stage 2 / or /usr filesystem or the "initramfs" snapshot image need to be stored on those stage 2 / or /usr filesystems.

The outline boot sequence can therefore well be:

Linux kernel is booted mounting /dev/sda1 as /, which contains a fully functional but "vanilla" / filesystem.
Linux kernel loads from it modules, runs scripts, does whatever it takes to unencrypt and prepare for mounting /dev/sda2 which contains /etc, /bin, /var, /home, ... /usr.
Then something like mount /dev/mapper/sda2 /newroot and pivot_root /newroot. In simpler times that would be instead something like mount /dev/mapper/sda2 /, and I have done something like that many times in the distant past. and occasionally in the recent past (e.g. after boot from live-CD)
Linux kernel remounts the "vanilla" root in /dev/sda1 as /boot :-).

That the above, which seems to me basic system knowledge, needs explaining again and again in detail is quite frustrating; as I wrote a few times already, the first-stage boot / (or kernel) does not need to be the very same as the the second-stage / (or kernel), and indeed in the current arrangement there are another two.

Plus there is mount --bind for an alternative option.

Anyhow in the current practice GRUB's root (first-stage root) /boot is inside an unencrypted partition distinct from that containing / (second/third stage root) and someone could replace anyhow its contents with suitably "improved" kernel images or "initramfs" snapshot images. I know people who understand boot processes and who keep /boot (only) on a USB storage device they carry with them at all times because they reckon that an encrypted / with an unencrypted /boot is not giving them peace of mind.

Preparing for a merged /usr in Debian

paulj — Tue, 19 Jan 2016 16:57:42 +0000

Ah wikipedia has some more information: https://en.wikipedia.org/wiki/Quotation_mark

«This is common in France apparently»

Preparing for a merged /usr in Debian

paulj — Tue, 19 Jan 2016 16:54:30 +0000

That's not bespoke at all, that's the normal way of quoting in some countries (Germany or France and a few others?). Also common is „This is a quote”, but I can't remember where, and probably lots more.

Not all the world is anglo-phile.

Preparing for a merged /usr in Debian

nix — Tue, 19 Jan 2016 12:39:38 +0000

Linked-in initramfses are a fragility *reduction*: even if your / is completely buggered, you still have a userspace running, and you can easily build in an emergency shell and recovery tools: all the bootloader needs is to be able to find the kernel image, and it's found the entirety of early userspace too. This has saved my bacon on multiple occasions, because among other things it means that the fsck used to fsck / and /usr is built in to the kernel image. This also means you can identify when changes in fsck, mount and the like have broken something critical by simply booting a known-working kernel, which will automatically contain a known-working fsck :)

Now linked-in initramfses are, admittedly, not what distros use, but they still work (and if they stop working people like me will yell bloody murder so they should keep working!)

Other benefits of using an initramfs include that the terrible hack whereby fsck has to work on mounted filesystems and then you sometimes have to crash-reboot is simply gone: fsck of filesystems needed by early boot, be that /, / and /usr, or whatever, is just like fsck of everything else. Plus, of course, it means that those of us using setups you consider intolerably unusual such as RAIDed / can use them without effort. Heck, I've suddenly switched an existing system from RAID to RAID-partially-over-NBD in the past when a sudden disk failure left me bereft on one machine but with a superfluity of storage on another: without an early userspace, this would have been an entirely hopeless quest without support in the *kernel* for such a baroque configuration, which is unlikely in the extreme.

This was not something I did recently, either -- it was way back in 2005. This configuration has been useful for over a decade now. Recent architecture astronautics it is not.

Further history of Unix of /usr

jond — Tue, 19 Jan 2016 12:09:20 +0000

> https://people.debian.org/~stapelberg/docs/systemd-depend...

I'm not sure if you read that before you posted it, but it basically debunks the myth that systemd has too many dependencies.

Preparing for a merged /usr in Debian

jond — Tue, 19 Jan 2016 11:56:26 +0000

Your bespoke quoting style «...» is quite hard to read. Please consider doing what everyone else does to keep the conversation easy to read.

Preparing for a merged /usr in Debian

anselm — Tue, 19 Jan 2016 10:05:04 +0000

The only things that can be disputed is whether mounting in sequence '/boot', "initramfs", '/', '/usr' are all necessary, and if not whether one should drop from that sequence '/boot' and "initramfs" or "/".

The issue here isn't necessity as much as convenience. As I said earlier, the Linux community used to do initramfs-free boots based on LILO for quite some time in the 1990s. We've come off that idea not because it doesn't work (it does, at least for simple scenarios), but because on the whole it is inconvenient and restrictive.

The initramfs-based approach, even though it looks more complicated at first sight, is vastly more powerful and convenient, and the added complexity is a price that most Linux distributions seem more than happy to pay. After all, Linux distribution makers are not in the business of making their life more complicated just for the sake of making it more complicated; they prefer approaches that work in a wide variety of cases and can be maintained.

If you think you can come up with a method of booting Linux systems that is better than what we have now, feel free to implement it and show it off to distribution makers. If it really is superior to the current approach, they'll be more than happy to take it on. That method worked for systemd in the basic-plumbing arena, after all …

Preparing for a merged /usr in Debian

mjg59 — Tue, 19 Jan 2016 09:02:35 +0000

> The solution that exists and has existed for decades and has worked well for decades is to mount am ordinary '/' filesystem at first-stage boot and use that to mount '/usr'

That's certainly a thing you could do, but now you have two choices:

1) Have /var, /home and /etc be part of /, and so be unencrypted
2) Have /var, /home and /etc be separate partitions, and introduce additional complexity (how do you know where /etc is if /etc isn't already there?)

Both of these choices are bad, and so your solution is a bad solution and has not worked well for decades.

Preparing for a merged /usr in Debian

mjg59 — Tue, 19 Jan 2016 08:57:26 +0000

> In exactly the same way that currently GRUB gets its modules and scripts and the new kernel in the content of an encrypted "/boot" filesystem, which also contains «important stuff that you want to put on it». :-)

The grub configuration and contents of /boot aren't secret and so don't need to be encrypted, so you're proposing copying much of the contents of / into /boot?

Preparing for a merged /usr in Debian

walex — Tue, 19 Jan 2016 08:53:31 +0000

«Things that grub doesn't do. And nor does the kernel. But an initramfs does. So we use the solution that exists, and we'll continue doing so until somebody dislikes it enough to come up with an alternative that works better.»

The solution that exists and has existed for decades and has worked well for decades is to mount am ordinary '/' filesystem at first-stage boot and use that to mount '/usr' (encrypted over iSCSI using TPM if necessary) for second stage boot. If you think that is more complicated and fragile than to mount first '/boot', then "initramfs", then '/', then '/usr', and that in such a sequence the clever move is to drop '/', good luck.

Preparing for a merged /usr in Debian

walex — Tue, 19 Jan 2016 08:42:37 +0000

«OK, so “/” is encrypted (it would after all be a very bad idea to leave it unencrypted, with all the important stuff that you want to put on it). Now how do you get at that trivial shell script in the mounted “/” file system when the kernel needs the pass phrase to mount “/” in the first place?»

In exactly the same way that currently GRUB gets its modules and scripts and the new kernel in the content of an encrypted "/boot" filesystem, which also contains «important stuff that you want to put on it». :-)

As I wrote several times, disputing that there must be a statically installed bootloader and kernel mounting a simple filesystem at the start of the boot sequence from which to load another kernel, drivers, configuration files and scripts to mount anything else, is just pointless and a really stupid debating trick. Because that's how boot sequences work by necessity (the second stage is necessary because of the flexibility requirements).

The only things that can be disputed is whether mounting in sequence '/boot', "initramfs", '/', '/usr' are all necessary, and if not whether one should drop from that sequence '/boot' and "initramfs" or "/".

Preparing for a merged /usr in Debian

mjg59 — Tue, 19 Jan 2016 07:55:11 +0000

> I may be wrong I don't think GRUB has block device drivers to access /boot

It does, but they're not typically used in BIOS or UEFI environments. I point this out merely for precise technical accuracy rather than it altering your point in any way!

Preparing for a merged /usr in Debian

mjg59 — Tue, 19 Jan 2016 07:47:37 +0000

> UEFI, for instances, does contain file system code. And UEFI doesn't require GRUB to boot Linux from a filesystem. I'd bet two pennies that if we had UEFI from the start, GRUB would not be as popular and LILO would not held out so long, assuming all else being equal.

To an extent. UEFI only (typically) has support for FAT, so your kernel and initramfs need to live on FAT, and the compromises around using FAT as a meaningful filesystem tend to result in it being difficult to make it /boot. Grub is, perhaps depressingly, less awful than some of the compromises you have to make to launch kernels from UEFI directly.

Preparing for a merged /usr in Debian

raven667 — Tue, 19 Jan 2016 01:52:04 +0000

> That objection is comically stupid ... people who actually understand booting sequences ...

We can disagree without being insulting or dismissive here, some of the people here actually build distributions and know more about booting than either of us do combined.

> I think that this is the third time that I have explained this is detail for those who ignore how booting sequences work.

I think we are going to disagree about how clear your ideas have been represented by your writing, the evidence is in how few people seem to have understood your intent. Let me see if I understand what you are proposing as another valid way to boot a system.

If I understand correctly you are thinking that we should go back to a simple LILO or uEFI style boot loader that is capable of loading the kernel and passing parameters from the firmware but to disaggregate what is currently in the initramfs into /boot, presuming that the kernel will have built-in drivers equivalent to what the firmware provides to be able to access the block device that /boot lives on, or that the kernel will use the firmware APIs to read the block device. Once you can reliably access the /boot block device you can run your minimal userspace to do hardware setup and pivot into the "real" system.

> is to have a Linux kernel without needing any sorts of drivers compiled statically except those needed to mount the first-stage / filesystem from a local storage area, exactly as GRUB does it for boot,

I may be wrong I don't think GRUB has block device drivers to access /boot, it relies on the firmware built-in services to access the block device, although it does have its own minimal filesystem driver modules (BIOS or uEFI or whatever), which are loaded LILO-style using static block lists, which has the same problems as LILO but GRUB modules are updated more infrequently than the kernel so is less of a problem in practice.

> plus optional kexec

As mjg59 has said elsewhere, and he has more operational experience with this feature than I do, it is a neat feature and great when it works, it's not something that is so reliable that you'd want to depend on it for the common case of everyday booting. In any event, if you are going to boot a Linux kernel, it might as well be the "real" one.

Preparing for a merged /usr in Debian

MattJD — Tue, 19 Jan 2016 01:12:00 +0000

> It comes as amazing news to me that the UNIX and Linux kernels didn't have «code in the system» to «to read filesystems». Try to imagine: before GRUB (or LILO) happened UNIX and Linux kernel could be easily booted, and mount /usr from encrypted filesystems, from iSCSI volumes, etc. How could this happen if GRUB «GRUB was necessary to read filesystems, as no other code in the system provided that»?

I apologize for the confusion. I was talking about IBM compatible BIOS. BIOS doesn't have, afaik, filesystem code. It can only read from hard drives (and depending upon age, limited sized drives!). Even Microsoft has a bootloader that loads the windows kernel. Before LILO, Linux must have had a different bootloader. Or it embedded the command line and fit in the first sector of the hard drive. Other Unix either had their own bootloader, or ran on other platforms which I don't know and won't speak for their system setup code.

UEFI, for instances, does contain file system code. And UEFI doesn't require GRUB to boot Linux from a filesystem. I'd bet two pennies that if we had UEFI from the start, GRUB would not be as popular and LILO would not held out so long, assuming all else being equal.

> GRUB (the bootloader) has exactly the same fault! Please read the manual: it can only ever boot one statically configured (with grub-install) GRUB kernel. If you upgrade the GRUB kernel you have to rerun its (static) installation.

A rarer event for me, and a step required as part of my system upgrade. Countless times I'd forget to rerun LILO when testing a kernel, and break the system. GRUB is far more resilient, even when I upgrade it. Anyways, no matter what you do there is always a risk upgrading bootloaders. Many people find GRUB, even with its complexity over LILO, more robust. I also find it much easier to debug failures with GRUB and recover systems.

> <snip> dynamically selected "second stage" Linux kernel to boot with kexec.
Correct me if I'm wrong, but kexec wipes out the previous system state. To build such a system would require extra work to make kexec into a proper bootloader replacement. Considering GRUB has all the drivers I need to just boot my preferred kernel and reliably load an initial /, why would I replace the simpler GRUB with a more complex system?

Also, some hardware is really unhappy about kexec. As a trivial example, my desktop GPU. It does not like being double initialized from the kernel, and I suspect it would take a lot of work to make it so for little gain (this is all on the official open source driver stack).

The boot process is always a pain on a system (my favourite example? The A20 line that needs to be dealt with in real mode). Everything is about trying to make it better. I won't argue the current system is the best thing possible (I know I'll be proven wrong eventually). I do think it's very good as is.

If you want to remove GRUB, UEFI allows that. I do see the value in making cut down systems that match your current system, by initramfs (and I've manually created an initramfs, including initial boot scripts. Interesting, but not unreasonable process. I used it for several years on an encrypted LVM /).

I'm willing to be shown that whole line of thought is wrong. I suggest you try creating your desired system, including all the necessary support structure. If it is better, I'd be happy to push it forward. As of right now, I just don't see how yours is better considering my experience with the Linux boot system.

Preparing for a merged /usr in Debian

mjg59 — Tue, 19 Jan 2016 00:27:44 +0000

We currently have this:

grub stage 1 loads grub stage 2. Scripts in grub stage 2 allow the choice of kernel and initramfs, loaded via firmware provided drivers. initramfs contains a subset of /. initramfs mounts the real /.

What you're proposing is:

lilo loads the boot kernel. Boot kernel mounts a / (we'll call it "boot /") that contains a subset of the real /. Scripts there allow the choice of kernel and /, loaded via boot kernel provided drivers. After kexecing a new kernel, the new kernel mounts "boot /" again and executes further scripts that mount the real /.

In your proposal you end up mounting "boot /" twice, you have to keep "boot /" synchronised with the real /, your boot kernel is significantly more complicated than grub and can't make use of any firmware-provided functionality (making it less flexible), you're using lilo which doesn't work under UEFI (ie, modern systems) and you're relying on kexec which *doesn't work* in a number of real-world cases. Your solution is *more* complicated than what we have today.

Preparing for a merged /usr in Debian

walex — Mon, 18 Jan 2016 23:45:43 +0000

«When BIOS was how x86 booted, GRUB was necessary to read filesystems, as no other code in the system provided that.»

It comes as amazing news to me that the UNIX and Linux kernels didn't have «code in the system» to «to read filesystems». Try to imagine: before GRUB (or LILO) happened UNIX and Linux kernel could be easily booted, and mount /usr from encrypted filesystems, from iSCSI volumes, etc. How could this happen if GRUB «GRUB was necessary to read filesystems, as no other code in the system provided that»?

«LILO was mentioned previously, but LILO had one large fault: if you upgraded your kernel you had to rerun LILO, or everything broke»

GRUB (the bootloader) has exactly the same fault! Please read the manual: it can only ever boot one statically configured (with grub-install) GRUB kernel. If you upgrade the GRUB kernel you have to rerun its (static) installation.

It is the GRUB kernel that spawns its shell environment, loads perhaps dozens out of several hundred (most them entirely Linux specific) possible GRUB kernel modules from /boot, and run a system initialization script that loads a Linux kernel, a toy / snapshot in the "initramfs", and then passes control the Linux kernel. These are well described and documented booting steps, and I am astonished that so many people here seem to ignore them.

This ridiculous and fragile complexity can be replaced by LILO replacing the GRUB bootloader, a "first stage" Linux kernel replacing the GRUB kernel, one that mounts a / directly instead of a pointless /boot or "initramfs", loads modules and scripts from it to mount /usr from anywhere, and load optionally a new kernel chosen dynamically by a user-interaction script, and kexec it.

As I just wrote in another comment, it cannot be disputed that the booting sequence must involve: #1 a bootloader that can only load one statically configured kernel, #2 a simple filesystem mounted by that kernel using statically builtin drivers and configuration, and #3 a series of drivers, scripts and other executables run dynamically by the kernel loaded in #1 from the filesystem mounted in #2.

There is simply no reason to have #1 as the GRUB stage1, #2, as the GRUB stage2 kernel, #3 as the /boot filesystem, then #2 again as a Linux kernel loaded from /boot with #3 again as the "initramfs".

Like in traditional UNIX and Linux systems this can all be far more easily arranged with #1 as LILO, #2 as a "first stage" Linux kernel with just (for example!) the VESA, serial, SATA, "ext4" drivers built in, mounting / from which any further driver modules, initialization scripts, etc. can be run to support arbitrarily complex boot configurations, including optionally loading a dynamically selected "second stage" Linux kernel to boot with kexec.

This may be the fourth time I try to explain how boot sequences actually work and how a traditional UNIX/Linux boot sequence works and is more robust and flexible than involving needlessly both /boot and "initramfs" as replacement for /....

Note: as always, a completely storage-less system needs a RAM disk and to download a disk image into it to setup /, but that's then the job of the bootloader like PXE.

Preparing for a merged /usr in Debian

anselm — Mon, 18 Jan 2016 23:37:09 +0000

That objection is comically stupid, because I have proposed a setup where /usr gets mounted and the second stage kernel (if needed, and it will be rarely needed) gets booted by a full (if minimal) GNU/Linux system hosted inside a first stage / root filesystem

Right, so please explain how you load your kernel from your first-stage “/” if that is encrypted. Something must find and decrypt the kernel before it can run. (GRUB can do it but you don't want to use GRUB.)

The proposed setup, which is the historical UNIX setup plus optional kexec, is to have a Linux kernel without needing any sorts of drivers compiled statically except those needed to mount the first-stage / filesystem from a local storage area

Nobody disagrees that this is possible in principle. We all did it that way on Linux 1.x, 20 years ago. The main reasons we prefer not to do it that way anymore are that (a) using an initramfs is much less hassle than compiling a new custom kernel for every single machine that we want to run Linux on, and (b) an initramfs lets us do various things in a reasonably straightforward manner that in the traditional setup are very difficult or impossible to achieve at all. In any case, it turns out that the initramfs approach, in actual practice, is a lot less brittle than you seem to assume.

and prompting for passwords can be done by a trivial shell script in the mounted / filesystem, and does not need to be done by the UNIX/Linux kernel

OK, so “/” is encrypted (it would after all be a very bad idea to leave it unencrypted, with all the important stuff that you want to put on it). Now how do you get at that trivial shell script in the mounted “/” file system when the kernel needs the pass phrase to mount “/” in the first place?

Preparing for a merged /usr in Debian

dirtyepic — Mon, 18 Jan 2016 23:32:25 +0000

>Because the Very Important People who have decided that merging is Just Like Microsoft Does It And Therefore Is Good have forgotten

I always love it when the loonies preload their arguments like this and I know I can just skip the rest of the post and all the replies. Thanks big guy!

Preparing for a merged /usr in Debian

mjg59 — Mon, 18 Jan 2016 23:31:37 +0000

> In exactly the same way GRUB currently prompts for the passphrase to mount an encrypted /boot, the same way GRUB currently gets the network configuration to mount /boot over iSCSI, and using the same way GRUB uses to interact with the TPM to attest system state to the user.

Things that grub doesn't do. And nor does the kernel. But an initramfs does. So we use the solution that exists, and we'll continue doing so until somebody dislikes it enough to come up with an alternative that works better.

Preparing for a merged /usr in Debian

Cyberax — Mon, 18 Jan 2016 23:17:16 +0000

> just like it has been done for decades by people who actually understand booting sequences
I.e. not you?

Please, desist from insulting people who actually do the work to make booting reliable and stable.

Preparing for a merged /usr in Debian

walex — Mon, 18 Jan 2016 23:14:52 +0000

«How does the kernel obtain the passphrase to decrypt an encrypted /? The network configuration to mount a / over iscsi? Interact with the TPM in order to attest system state to the user?»

In exactly the same way GRUB currently prompts for the passphrase to mount an encrypted /boot, the same way GRUB currently gets the network configuration to mount /boot over iSCSI, and using the same way GRUB uses to interact with the TPM to attest system state to the user. Equivalently all the same ways all those things are currently done by the Linux kernel booted by GRUB to mount an encrypted "initramfs", to get the network configuration to mount an "initramfs" over iSCSI, and for the "init" processor in the "initramfs" to attest the system state to the user. :-).

Isn't that totally obvious? Didn't I write very clearly that «I hope that there is no dispute that the starting point of the boot process must be a simple, limited configuration "container"/"filetree" of some sort like /boot or "initramfs"»

If you dispute that, this discussion is pointless like most discussions with the members of the poetteringista tendency, because evidently the understanding of how booting works and the role of the / filesystem in UNIX/Linux systems has been completely lost.

Preparing for a merged /usr in Debian

walex — Mon, 18 Jan 2016 23:02:55 +0000

«Can you have GRUB loading its kernel, configuration file and modules from a /boot being a BTRFS filesystem on an encrypted LVM volume on top of an MD RAID1 spanning 2 DRBD volumes?»

No, but in your proposed non-GRUB, non-initramfs setup you can't use LILO to get your initial kernel from there, either, so this is an obvious red herring. Try harder.

That objection is comically stupid, because I have proposed a setup where /usr gets mounted and the second stage kernel (if needed, and it will be rarely needed) gets booted by a full (if minimal) GNU/Linux system hosted inside a first stage / root filesystem, just like it has been done for decades by people who actually understand booting sequences, and that LILO in that setup is equivalent not to the GRUB kernel but to the GRUB bootloader that loads the GRUB kernel from the GRUB root filesystem which is /boot. I think that this is the third time that I have explained this is detail for those who ignore how booting sequences work.

«An initramfs is reasonably easy to construct and keep current and, if you're providing a Linux distribution, makes it unnecessary to use a kernel that has all sorts of drivers compiled statically to ensure wide hardware coverage»

The proposed setup, which is the historical UNIX setup plus optional kexec, is to have a Linux kernel without needing any sorts of drivers compiled statically except those needed to mount the first-stage / filesystem from a local storage area, exactly as GRUB does it for boot, but unnecessarily. Having mounted / the stripped down Linux kernel can then load whichever modules and run whichever scripts may be needed to mount the /usr filesystem, or to load a new dynamically selected kernel from any storage medium whatever, with kexec and reboot into it.

«an initramfs makes it straightforward to do various things – such as network configuration in case your actual root file system is on a remote iSCSI volume, or obtaining a pass phrase for an encrypted root file system – that the kernel isn't really set up to do. How to do this in user space in an initramfs is quite well understood»

People who understand how booting sequences work understand that doing all that with a stripped down UNIX/Linux kernel and a mounted / filesystem is much easier and less fragile than packing up every time a new static "initramfs", and prompting for passwords can be done by a trivial shell script in the mounted / filesystem, and does not need to be done by the UNIX/Linux kernel, and that there is no need to snapshot the / filesystem into an "initramfs" image unless there is a totally storage-less boot situation.

Preparing for a merged /usr in Debian

Cyberax — Mon, 18 Jan 2016 22:45:28 +0000

Personally, I always feel that / on Unix is somehow untidy. Too many top-level directories, not enough clear separation of concerns.

Moving most of the / into /usr definitely helps to clean it up. Of course, we'll have to use compat symlinks for the next 10 years or so, but there's at least a hope of getting them removed eventually.

Preparing for a merged /usr in Debian

mcortese — Mon, 18 Jan 2016 22:35:39 +0000

After the merge, /usr will include all and only the data provided by the distribution, with one notable exception: /usr/local. Somebody will then advocate moving it to /. Just imagine the symmetry in having a /local hierarchy that mirrors /usr: for example /local/bin would be the locally-installed equivalent of distribution-installed /usr/bin, and so on...

Preparing for a merged /usr in Debian

anselm — Mon, 18 Jan 2016 00:13:08 +0000

Can you have GRUB loading its kernel, configuration file and modules from a /boot being a BTRFS filesystem on an encrypted LVM volume on top of an MD RAID1 spanning 2 DRBD volumes?

No, but in your proposed non-GRUB, non-initramfs setup you can't use LILO to get your initial kernel from there, either, so this is an obvious red herring. Try harder.

What is the case for the additional complexity and fragility for having an "initramfs" that is a statically built and customized snapshot of a large subset of / and then "pivot" to a full / or in the latter-day proposal to a full /usr?

An initramfs is reasonably easy to construct and keep current and, if you're providing a Linux distribution, makes it unnecessary to use a kernel that has all sorts of drivers compiled statically to ensure wide hardware coverage, most of which drivers are unnecessary for any particular system (because the hardware isn't there) but can't be removed other than by compiling a new, customised kernel. (Remember that that was a situation we used to have to deal with in the early years of Linux, and now that we no longer need to, pretty much everyone is a lot happier.) This sucks especially if you're running an enterprise-class Linux distribution like RHEL or SLES where tampering with the kernel will mean your system no longer qualifies for vendor support in case of problems.

In addition, an initramfs makes it straightforward to do various things – such as network configuration in case your actual root file system is on a remote iSCSI volume, or obtaining a pass phrase for an encrypted root file system – that the kernel isn't really set up to do. How to do this in user space in an initramfs is quite well understood, and putting the required functionality into the kernel would only add unnecessary complexity and probably bugs.

Preparing for a merged /usr in Debian

raven667 — Sat, 16 Jan 2016 22:55:10 +0000

> Keep the ability to have distinct block device containers for / and /usr but merge the /bin /lib directories into the /usr hierarchy and container. Leaving in the / hierarchy only /etc, /dev, /sys, ...; the main purpose of this would be to shorten the default $PATH and $LD_LIBRARY_PATH.

Its not just to shorten the $PATH, it's also so that _all_ of the system binaries, which are interdependent due to shared library and data file dependencies, are mounted at the same time in a self-consistent way with the same mount options.

> Leave the / and /usr hierarchies as they are with distinct /bin etc. directories because what matters is mounting a single block device.

If you require all the binaries to be on the root filesystem then when having many systems with a shared binary directory it precludes the host specific variable data from being on a single filesystem (etc, var,home) , unless you add bind mounts to the mix.

Preparing for a merged /usr in Debian

mjg59 — Sat, 16 Jan 2016 19:49:29 +0000

> go back to booting directly an initial Linux kernel that mounts an initial / filesystem

How does the kernel obtain the passphrase to decrypt an encrypted /? The network configuration to mount a / over iscsi? Interact with the TPM in order to attest system state to the user?

Preparing for a merged /usr in Debian

mjg59 — Sat, 16 Jan 2016 19:44:25 +0000

Any solution that depends on kexec is a solution that doesn't actually work. It's amazingly unreliable on real platforms.

Preparing for a merged /usr in Debian

MattJD — Sat, 16 Jan 2016 16:34:40 +0000

The four stages you describe come because of other limitations of the interface available to the kernel to boot more complicated systems. When BIOS was how x86 booted, GRUB was necessary to read filesystems, as no other code in the system provided that. LILO was mentioned previously, but LILO had one large fault: if you upgraded your kernel you had to rerun LILO, or everything broke (and I did that to myself several times on and older laptop). UEFI provides what GRUB required, just like u-boot provides for ARM what GRUB does on x86 (though you can use GRUB in both cases afaik). That takes one stage out. I bet GRUB is still around because UEFI systems don't provide a common user interface for choosing what to boot, while GRUB does. Distributions go UEFI->GRUB so when there is a problem they can give instructions that are more likely to work across all machines.

Regarding initramfs and /, initramfs provides what / used to provide. Now / can require several steps before it mounts (for instance, on my laptop I need to type in my encryption passphrase). An initramfs can also provide a complete recovery shell, that relies on minimal other parts of the system to work. So what you propose already basically exists. The only difference is that you want a / that comes from a partition, whereas initramfs provides a / that comes from memory. Either way they need to be maintained, and neither provide a full system.

So I don't think there is a large gap between what both sides of this conversation want. It's just that we haven't done the cleanup to get closer to the desired simpler boot sequence.

Preparing for a merged /usr in Debian

walex — Sat, 16 Jan 2016 14:32:16 +0000

different people disagree on what constitutes a "minimal, streamlined" base.

Indeeed and I think that we can all agree that having a 4 stage boot process with 4 different "root" filesystems (/boot, "initramfs" snapshot, /, /usr) is too complex and fragile.

Given that the end goal is having /usr mounted, that cannot be removed, so the alternatives are:

Drop / and keep both /boot and the "initramfs", then mount the final /usr.
Drop /boot and the "initramfs" and go back to booting directly an initial Linux kernel that mounts an initial / filesystem, and then if needed kexec a final kernel, and then mount by whatever means the final /usr.

To me dropping / and keeping the far more complex and fragile /boot and "initramfs" stages seems solving the wrong problem at best.

Preparing for a merged /usr in Debian

walex — Sat, 16 Jan 2016 14:09:39 +0000

The reason why Unix systems in former times managed to get away with a much simpler boot structure

That's a crazy starting point, because the assumed goal here is to cover a complex boot configuration, not to have a complex boot structure for its own sake. Complex structures don't bring any benefit in themselves, except the pride of a job poorly done :-).

There is no need to assume the that a complex boot configuration requires a complex, fragile, boot structure with 4 stages instead of a simple structure like "boot a stage one Linux kernel and / and then use that to do stage two of the boot process using the full power of a Linux system to boot a bigger kernel and mount a /usr filesystem".

is that the range of supported disk controller hardware, file systems, etc. was a lot smaller, and therefore it was reasonable for manufacturers to distribute kernels that had everything needed to get at “/” statically built into them.

But GRUB in order to mount /boot makes very strong and limiting assumptions about the range of possibilities it can cope with. Can you have GRUB loading its kernel, configuration file and modules from a /boot being a BTRFS filesystem on an encrypted LVM volume on top of an MD RAID1 spanning 2 DRBD volumes? If not, why are GNU/Linux users being so limited to the range of drivers statically built into the distributed GRUB kernel? :-)

What is the case for the additional complexity and fragility of having GRUB + /boot replicating in complex yet limiting way a large part of the functionality of the Linux kernel and of /?

What is the case for the additional complexity and fragility for having an "initramfs" that is a statically built and customized snapshot of a large subset of / and then "pivot" to a full / or in the latter-day proposal to a full /usr?

Preparing for a merged /usr in Debian

anselm — Sat, 16 Jan 2016 09:45:13 +0000

The fallacy here is in assuming that because something worked in the 1980s it must work today and indefinitely into the future. (We see this a lot in people who question the need for something like systemd by claiming that System-V init is the solution to all problems because, you know, it runs shell scripts.)

GRUB is the way it is because it supports a wide variety of use cases apart from simply booting a Linux kernel. Distributions like it because it covers pretty much all deployment scenarios one is likely to run into in Real Life™, and it is usually easier to maintain a single solution that does everything you need than a variety of specific solutions that are all designed to optimise a step in a system's life cycle that happens fairly rarely in the first place. (Most people don't boot their system several times per hour.) As we've mentioned, it is perfectly possible to get rid of GRUB on a UEFI-based system, because the UEFI boot loader can boot a Linux kernel directly, so this makes your “first-stage boot” that much simpler already. Mainstream Linux distributions, however, can't rely on everybody having UEFI, which is why they still use GRUB because that works everywhere.

The reason why Unix systems in former times managed to get away with a much simpler boot structure is that the range of supported disk controller hardware, file systems, etc. was a lot smaller, and therefore it was reasonable for manufacturers to distribute kernels that had everything needed to get at “/” statically built into them. (As a matter of fact, traditional Unix kernels often didn't support the idea of loadable modules in the first place.) On contemporary Linux, there is no practical way of getting rid of “initramfs” that reliably supports things like encryption, RAID, or diskless systems and does not present a maintenance nightmare compared to which initramfs is really the simpler and more versatile solution. You end up trying to build distribution kernels that have everything in them statically on the off-chance (including stuff like a DHCP client that is better left to user space), and generally distributions are very happy to be past that, and to be able to use initramfs to offer modular, maintainable kernels. This means that you're unlikely to find anyone who actually deals with that sort of thing in actual practice who will agree with you that essentially going back to “pre-hipster” Linux is anything remotely like a good idea. In any case, the approach is fairly well-understood and rarely causes problems, so this is really a tempest in a teacup.

Preparing for a merged /usr in Debian

walex — Sat, 16 Jan 2016 09:27:46 +0000

shoving the contents of /{s,}bin and /lib into /usr will, somewhat paradoxically, *decrease* the need for an initramfs:

This and other comments make me suspect that a lot of the discussion above is based on some big misunderstanding of the situation, and possible changes, and in part because the article above seems a bit confusing to me, and I'll try to clarify:

A "filesystem" is a tree of directories and files which is stored in its entirety in a container which is as a rule a "block device" (there are special cases but let's skip them).
A "hierarchy" in UNIX is a tree of directories and files that have a conventional organization, for example one where at the top there are /bin, /etc, include, lib, doc, /src etc. subtrees, because the UNIX convention is to store files by type.
Traditional UNIX had first one hierarchy under /, then a second one was added under /usr, and then a third one was added under /usr/local; the hierarchies were nested inside each other instead of being mounted alongside each other, as for example /core, /usr, /local.
Eventually differently-shaped sub-hierarchies were added under /var, and at different points in time under /usr/X11, /usr/net etc.
Traditionally the / and /usr hierarchy were independent filesystems, where the / hierarchy contained the core tools needed to get to the "block device" containing the /usr hierarchy and mount it.

Now nearly all existing distributions still support having distinct block device containers for the / and /usr (and /usr/local and /var) hierarchies, even if some in that case mount / and /usr at the same time, which requires all the GRUB and Linux drivers needed to mount both / and /usr to be present in both GRUB's root filesystem /boot and in the Linux first-stage root filesystem the "initramfs".

The "merged-usr" proposal can be interpreted in different ways:

Keep the ability to have distinct block device containers for / and /usr but merge the /bin /lib directories into the /usr hierarchy and container. Leaving in the / hierarchy only /etc, /dev, /sys, ...; the main purpose of this would be to shorten the default $PATH and $LD_LIBRARY_PATH.
Remove the ability to have distinct block device containers for / and /usr enforcing the assumption that whichever root block device is mounted at boot time contains both the / and /usr hierarchies. This can come in two flavours:
- Leave the / and /usr hierarchies as they are with distinct /bin etc. directories because what matters is mounting a single block device.
- Also merge the /bin /lib etc. directories into the /usr hierarchy because if they enforced to be in the same container this is an opportunity to reduce the length of the default paths.