|
|
Log in / Subscribe / Register

Poettering: Fitting Everything Together

[Posted May 9, 2022 by corbet]

Lennart Poettering designs his ideal desktop operating system in great detail:

First and foremost, I think the focus must be on an image-based design rather than a package-based one. For robustness and security it is essential to operate with reproducible, immutable images that describe the OS or large parts of it in full, rather than operating always with fine-grained RPM/dpkg style packages. That's not to say that packages are not relevant (I actually think they matter a lot!), but I think they should be less of a tool for deploying code but more one of building the objects to deploy.

to post comments

Poettering: Fitting Everything Together

Posted May 9, 2022 21:48 UTC (Mon) by jonesmz (subscriber, #130234) [Link] (12 responses)

I normally love systemd related tools, but I can't stand the current trend of moving everything towards images and containers. So for this I wholeheartedly disagree.

Poettering: Fitting Everything Together

Posted May 9, 2022 22:06 UTC (Mon) by stumbles (guest, #8796) [Link]

Times I wish this site had up votes.

Poettering: Fitting Everything Together

Posted May 10, 2022 2:38 UTC (Tue) by pabs (subscriber, #43278) [Link]

I think there is room for both types of system management. You can even use packages to build read-only images.

Poettering: Fitting Everything Together

Posted May 10, 2022 11:59 UTC (Tue) by XTerminator (subscriber, #59581) [Link]

Don't turn this into facebook, or youtube, but I Upvote this comment

Poettering: Fitting Everything Together

Posted May 10, 2022 12:37 UTC (Tue) by nix (subscriber, #2304) [Link] (2 responses)

Yabbut he's not really arguing that nothing should be package-based, and *that* is the real problem with a lot of containerized systems: you can't do like you do on a desktop, of saying "ok install this package [magic happens]; now I'm doing stuff with that installed without having to restart everything". Unfortunately the only thing Lennart says about this is the "system extension images", which, well, I suppose they'd work but they do feel awfully clunky. They'd need to have a lot of tooling wrapped around them.

In particular, what if you upgrade a bunch of stuff -- which builds and installs a system extension image -- start working with it, and then want to upgrade something else? Do you end up with *two* images installed at once? If the images are "purely additive", does this mean you can't add an image that updates something already present in another running image? 'cos the inability to upgrade something you just installed without removing it first (doing what to running programs?) feels like a fairly harsh restriction, unless you use a nix-like model in which every installed program goes into a new subdir and is symlinked into place, which a) obviates most of the need for this whole scheme and b) still means you have to replace symlinks that already exist.

The other horrible limitation (which is explicitly documented) is the inability to fix bugs in the core image without rebuilding the whole thing: to pick an example not at random that I just hit (but with a packaged system so fixing it was easy): some new package you want to build into an image needs a slightly newer version of pyqt-5 and oops on this crazy system of discourse pyqt-5 is built into the core image and now you *can't upgrade pyqt-5 at all*, not even slightly, not even purely to fix bugs, without upgrading the core image, which seems to involve a reboot (and, more importantly, a complete dynamic state loss, which is what people really dislike about reboots). Given that we have things like libabigail to prove that the ABI is unbroken, and given that we don't *want* to prevent behaviour changes (a bugfix is a behaviour change), this all feels like it might be nailing things down a bit too much. Why do we want to prevent bugfixes anyway?

(Note: right now the stuff I'm working on often involves upgrading *kernel* packages, but in ways that don't require a reboot to test out the changes. Presumably this would be even more impossible in the new world.)

Packages are horribly complicated and probably not the right way to install software, though distributing them that way and using them to build install images does still feel right (and though it explodes the test matrix, I haven't seen that systems with single huge official core images are notably less buggy).

Large parts of this new model seem really nice, in particular the ability to actually verify and enforce that things haven't changed dammit until you want them to... but doing the change when you do want it to should be easy and nondisruptive. Perhaps if there was better tooling around the whole image thing -- letting you replace arbitrary parts of the image without rebuilding it, then replace those replacements and generally fiddle with them until you're happy, then (say) merge some of those parts back into the main image at reboot time when you're happy with them, like you can with QEMU CoW images, recomputing all the fs-verity data as needed -- it might be adequate for not only the boring "nothing needs to change" workflows but also the systems-developer workflows that, uh, constitute a tiny portion of the world's computing. Um.

(My ideal world would allow you to have an actual *tree* of things you're experimenting with, so you can flip between an image (built additively from package installs and who knows what else atop some earlier set of images) using change A and one using change B (and C and D...), and then when you're happy with, say, A, merge change A back into the root image at next boot and throw B and its descendants away. I can already do this with text editors, why I can't I do this with the distro! Plus if these things were all persistent it would give us the effect of fs-level snapshots for distro upgrade testing, but on the level of the *entire system*.)

I do wonder if what we actually need is some sort of fs-verity improvement so that you can replace verity fses' content and update their integrity metadata without needing a reboot -- or some way to switch the system into a developer mode, disable integrity enforcement, hack on the root image, recompute everything, and reactivate, all without rebooting, transparently, while the package manager was running: this would probably remove any need for all this image stuff but keep the same benefits. But of course if you have that, an attacker can do it too -- but an attacker could presumably reinstall the base image in Lennart's model as well if they had that much power: and if they didn't, they probably just need to ask the user in an official-seeming dialogue and then they've got it -- or the user can't grant that power, and I don't want to use a system like that. Does this make the whole fs-verity idea useless? Does it mean that the massively disruptive reboot is actually necessary to warn people that a major verity-adjusting change has been installed? (I doubt it: non-geeks' machines spontaneously reboot all the time and nobody ever thinks "oh no I bet that means an attacker has replaced the core OS image": it's just disruptive without benefit, that's all).

Poettering: Fitting Everything Together

Posted May 10, 2022 13:22 UTC (Tue) by walters (subscriber, #7396) [Link]

> The other horrible limitation (which is explicitly documented) is the inability to fix bugs in the core image without rebuilding the whole thing

We put a *lot* of work in the rpm-ostree side to enable exactly this. In particular, it is a first class operation to do a local override of just e.g. the kernel - we then trigger a client-side rebuild of the initramfs, etc. We don't ship an rpm-ostree release (or really a Fedora release) unless this functionality works.

Poettering: Fitting Everything Together

Posted May 10, 2022 16:25 UTC (Tue) by kleptog (subscriber, #1183) [Link]

The way I understand it, the core image would be very minimal, just enough to boot the system and start other images. You certainly wouldn't want anything like pyqt-5 on it.

Additionally, applications/services should not be based on this image at all, but off in a private namespace. There's no reason for applications to access the filesystem space of the host system, that would just complicate upgrading. Whatever application uses pyqt-5 would be based off a different image which could be updated at any time without affecting the host.

This sounds a lot like containers, but it's only isolating the filesystem, they still share the same PID and network namespace for example. More like a fancy chroot. And not a tree, but a forest.

Really, the only thing you can't easily upgrade on the fly is the kernel. Even the init process could re-exec itself into a new root image without affecting the rest of the system.

Poettering: Fitting Everything Together

Posted May 10, 2022 13:17 UTC (Tue) by kleptog (subscriber, #1183) [Link] (4 responses)

In the beginning we just had files. make && make install. But then we found that that was unmaintainable and upgrades were hard. So we created packaging so we could manage groups of related files as a block, installing, uninstalling and upgrading all together.

Now our systems are bigger. My development system has ~3000 Debian packages installed, and that's not counting the hundreds of packages via other packaging systems (npm/pypi/etc). I didn't even manually install a tiny fraction of those, they're mostly dependencies of others. I don't know what most of them do, I'm not a computer. I want to work at a higher level of abstraction. I want to be able to say: "upgrade application X" and have it happen without worrying about it accidentally breaking something else. I don't care how many hundreds (or thousands) of packages are impacted, that's the computer's job.

That's the problem all these container abstractions are trying to solve: to be able to install/uninstall/upgrade collections of packages (not files) and configuration in an reliable way without interfering with others. They don't replace packages: they build on top of them.

There are other ways of doing this, I believe Nix tries to solve a similar problem. But container images are a currently widespread well-understood and well-supported mechanism to doing this. And they also remove the problem of building applications for multiple distributions; I don't even need to know if an application was built for Debian or Redhat, it will work either way perfectly.

Maybe it's not the best solution, I'm sure it will be refined as we go along. But you surely agree it's a real problem that needs solving?

Poettering: Fitting Everything Together

Posted May 10, 2022 14:02 UTC (Tue) by andi8086 (guest, #153876) [Link] (3 responses)

For this reason, there are package groups in arch linux for example. In my opinion, Poetterings ideas are against long standing working design principles. systemd is as ugly in my opinion. The more fine granular the pieces are, the more reusable and the easier individual components can be exchanged and fixed. Now inventing something like package groups again is completely unnecessary. Things like container images are a waste of memory, thus waste of natural resources and hard to work with if offline. Individual packages can also be transfered on USB stick easily and quickly. But hey, as long as I don't have to use it, he can design what he wants :D

The problem for non-working applications is another one. It is the problem of unclean APIs. As long as every Hienz and Cunz defines his own API a little different, we get such problems. And then every application ships it's own glibc or what? This cant be! In my opinion Mister Poettering should leave Linux alone and maybe define something new, like Potterix.

Poettering: Fitting Everything Together

Posted May 10, 2022 14:52 UTC (Tue) by farnz (subscriber, #17727) [Link]

Poettering is working on his own thing. It just happens that it's based on the Linux kernel and glibc, and as such people are free to take Poettering's things and use them in Debian, Fedora, and other Linux-based distros.

And because the problems he's tackling are ones that no-one else has done a better job of (so far), his ideas get adopted by other people working on the Linux family of OSes. A lot of his ideas don't actually get adopted by anyone else, because they're simply not the best versions of those ideas, and distros prefer the versions that work.

Poettering: Fitting Everything Together

Posted May 11, 2022 11:18 UTC (Wed) by MatejLach (guest, #84942) [Link] (1 responses)

> Poetterings ideas are against long standing working design principles. systemd is as ugly in my opinion.

I don't agree with this particular post of his and his ideas around it, but I was there before Arch switched over to systemd and it was an ugly ad-hoc script kind of a mess. Compared to that, writing systemd definitions and generally its command and syntax structure being predictable is a whole lot better imo.

Personally I think he has good ideas and bad ones, don't really get the 'hate everything he does' thing.

Poettering: Fitting Everything Together

Posted May 11, 2022 16:54 UTC (Wed) by Wol (subscriber, #4433) [Link]

The trouble with Poettering, is he expects documented interfaces to behave AS DOCUMENTED.

The sendmail mantra of "be liberal in what you accept, be strict in what you send" was widely abused as "who gives a shit what you send the other end will handle it".

So when Poettering replied "if you send me shit I'll dump it on the floor" he got all the flak for everyone else's bugs. Never mind that he did his best to fix those bugs.

That's why PulseAudio was such a mess - it was very strict in what it accepted, but it sat on top of ALSA which was a liberal cesspit.

And there are too many people out there who can't get over the fact that distros shipped PulseAudio on top of broken ALSA.

Cheers,
Wol

Poettering: Fitting Everything Together

Posted May 11, 2022 10:42 UTC (Wed) by MatejLach (guest, #84942) [Link]

I echo your sentiment.

What annoys me especially is how the assertion seems to be that moving to the likes of flatpak doesn't change the experience in any way.

On something like the PinePhone Pro any random flatpak app takes about 4x as long to launch as its native counterpart does. To me this is not a great experience, not even going into all the other disadvantages since this is something we should universally agree as being bad.

I hear snaps are even worse in this regard btw.

Poettering: Fitting Everything Together

Posted May 9, 2022 22:04 UTC (Mon) by wiltaylor (guest, #145007) [Link] (4 responses)

I actually think the nix approach is better. Go for reproducible builds over huge images or you will just recreate the same problems docker has but with os images.

Poettering: Fitting Everything Together

Posted May 9, 2022 22:32 UTC (Mon) by areilly (guest, #87829) [Link] (1 responses)

I think that the image approach is silly, and definitely not what I want, but I'm not most users. I still build everything from source.

One idea that sounds intriguing though, was the Fuchsia one, where applications are all packaged with all of their dependencies (so the runtime environment is known), but the installation process involves a system-wide deduplication cache of all package objects, so dependencies that are already on the system are not re-loaded, not taking up space, and not wasting virtual memory space when running. It's package loading as web-page loading. Dependencies age out naturally, just like a browser cache.

Obviously that also has issues: there are different trust relationships that need to be sorted out, but technically it's really cool, I think.

Poettering: Fitting Everything Together

Posted May 10, 2022 8:37 UTC (Tue) by nedrichards (subscriber, #23295) [Link]

It's a good concept - and very similar conceptually to the ostree based systems (that use file level dedupe) on Linux (e.g. Endless OS, Fedora Silverblue, Flatpak).

Poettering: Fitting Everything Together

Posted May 10, 2022 0:12 UTC (Tue) by bartoc (guest, #124262) [Link] (1 responses)

The nix approach makes reliable integration testing extremely difficult.

Poettering: Fitting Everything Together

Posted May 10, 2022 1:33 UTC (Tue) by sbaugh (guest, #103291) [Link]

In what way does the Nix approach make reliable integration testing difficult? I can't think of anything, except maybe "package /nix/store/A could break abstractions and touch /nix/store/B which isn't one of its dependencies", but that's both easy to work around and basically impossible to do accidentally.

Poettering: Fitting Everything Together

Posted May 9, 2022 22:55 UTC (Mon) by donbarry (guest, #10485) [Link] (23 responses)

In an field in which Poettering is often visionary, he falls flat on his face here. This is essentially a pessimistic abdication of the concept of moving forward with planned, versioned APIs to form a common basis on which our software lives.

"Throw everything into an image" is certainly the paradigm promoted by those who put the mad rush to get something out above security, OS lifecycle, maintainability, and all those other tasks you'd prefer to be done by "other" people but care not to contribute to yourself - I use personal pronouns but by this I really mean corporate interests, which deeply shape the individual responses.

It is the rejection of even the term "Software Engineering," much less "Computer Science," for unfettered pragmatism, meaning OS anarchy. One could almost imagine the polite but corrosive comments of the greater visionaries like Edsger Dijkstra, were he still around, to such improvisatory and unplanned methods. "...informality is the hallmark of the Mathematical Guild, whose members —like poor programmers—derive their intellectual excitement from not quite knowing what they are doing and prefer to be thrilled by the marvel of the human mind (in particular their own ones)."

Poettering: Fitting Everything Together

Posted May 10, 2022 0:53 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (4 responses)

What evidence is there that image-based distribution approaches are less secure or maintainable than traditional ones? Having worked on both, there's a bunch of ways in which image-based distributions make it easier to deploy and test security updates and a bunch of corner cases that package-based ones can fall short on.

Poettering: Fitting Everything Together

Posted May 10, 2022 13:44 UTC (Tue) by ddevault (subscriber, #99589) [Link] (3 responses)

The key difference is that image-based distribution generally relies on the goodwill of the upstream developers to ship software that doesn't contain (adware | spyware | rootkits | nonfree software), whereas having a package distribution in the middle can prevent these with a dispassionate third-party who advocates for the user's rights even when in conflict with vendor desires.

Image-based distributions also cannot effectively plan around the integration into and operation of the entire system - only a distribution can. A complex system is (usually) an insecure system.

Poettering: Fitting Everything Together

Posted May 10, 2022 13:55 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

> The key difference is that image-based distribution generally relies on the goodwill of the upstream developers to ship software

Why? The proposal doesn't include shipping upstream software as is. The images are built from distribution packages in the first place. As is the case today with Fedora CoreOS, Silverblue etc.

Poettering: Fitting Everything Together

Posted May 10, 2022 13:58 UTC (Tue) by ddevault (subscriber, #99589) [Link] (1 responses)

Perhaps for these examples (which I still don't like because of the increased complexity, and increased complexity means decreased security), but how else is the goal of "easier deployments" achieved?

Poettering: Fitting Everything Together

Posted May 10, 2022 14:13 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link]

> Perhaps for these examples (which I still don't like because of the increased complexity, and increased complexity means decreased security), but how else is the goal of "easier deployments" achieved?

The examples are the only relevant ones since there noone has said we should bypass distributions here. The proposal explicits covers how various distribution changes are relevant to making this feasible at all and how it helps with deployments.

Poettering: Fitting Everything Together

Posted May 10, 2022 7:59 UTC (Tue) by mezcalero (subscriber, #45103) [Link] (17 responses)

APIs are great – for developing stuff. ABIs are great too, for combining components reasonably safely and quickly. I am not convinced though that the APIs/ABIs alone are also the answer to the problem of delivery and deployment of software. What is good for the developer or system builder is not necessarily also great for delivery and deployment of projects. Not distinguishing between these concepts originates in the hacker-centric PoV inherent to traditional Linux, but I think is something we should revisit as we take Linux elsewhere.

Note that my proposal specifically uses RPMs or debian packages, to build images from and then deliver them as such. It thus doesn't depart from a world view where stable APIs/ABIs are good for development and system builders, but it does depart from the idea that RPMs/deb packages are the ultima ratio of pushing code onto the deployment.

Lennart

Poettering: Fitting Everything Together

Posted May 10, 2022 12:43 UTC (Tue) by nix (subscriber, #2304) [Link] (5 responses)

Well... the packages are still there, right? You're just pushing around the *result of installing a bunch of them* in some consistent state you want to replicate, rather than pushing around the packages and hoping the package installer independently replicates a consistent state at every destination machine (thin chance of that if any of them involve config edits).

Poettering: Fitting Everything Together

Posted May 11, 2022 11:23 UTC (Wed) by mezcalero (subscriber, #45103) [Link] (4 responses)

exactly. in my model images are built from packages. (Though you'd usually drop the packaging metadata in the resulting image that is deployed, as the value of that once you deal with images is questionable.)

Previously: RPM/deb is a deployment method onto the final system.
In my model: RPM/deb is a build method for the image and the image is the deployed onto the final system.

Lennart

Poettering: Fitting Everything Together

Posted May 12, 2022 13:07 UTC (Thu) by cortana (subscriber, #24596) [Link]

> exactly. in my model images are built from packages. (Though you'd usually drop the packaging metadata in the resulting image that is deployed, as the value of that once you deal with images is questionable.)

The metadata is actually quite handy to keep around for security scanning. That said, I wish there was a standard way to push the metadata about installed packages/modules/what have you into the container image metadata, rather than relying on keeping all the metadata in the container image itself.

Poettering: Fitting Everything Together

Posted May 16, 2022 18:49 UTC (Mon) by jonesmz (subscriber, #130234) [Link] (2 responses)

I'm really not seeing how this is any different from having a list of packages to install on the target system. It sounds like "dnf install ${SOME_LIST}" with extra steps.

Poettering: Fitting Everything Together

Posted May 17, 2022 16:18 UTC (Tue) by bluca (subscriber, #118303) [Link]

Yeah! It could be easily done on ZFS!

Poettering: Fitting Everything Together

Posted May 17, 2022 20:27 UTC (Tue) by raven667 (subscriber, #5198) [Link]

The extra steps are to create some atomicity around the whole group of packages, either the whole lot is correct or it isn't which can be verified, rather than the install or upgrade transaction being at the individual file and package level it's for the whole image. People have done work with snapshots and btrfs or zfs to get similar properties, this is another more general way to go about it.

Poettering: Fitting Everything Together

Posted May 10, 2022 13:53 UTC (Tue) by k3ninho (subscriber, #50375) [Link] (1 responses)

Hey Lennart, thank you for your efforts over the past 20+ years to make my computer work better. And thank you also for your patience with people who disagreed with you, no matter how horrid they were.

>I am not convinced though that the APIs/ABIs alone are also the answer to the problem of delivery and deployment of software.
Where do you fit tools in 'Software Bill of Materials' space for logging source and build provenance, or extending into capture of test results that speak to the promised capability of the packages? Debs have to be signed by a Debian Developer, confirmed at install time and ideally they transit over TLS-encrypted tunnels, but I only know that they promised ABI compatibility and not any story about the testing process run through or the specific measurement/observations taken of a packaged build.

K3n.

Poettering: Fitting Everything Together

Posted May 11, 2022 11:28 UTC (Wed) by mezcalero (subscriber, #45103) [Link]

As mentioned, in my model there would be four parties: developers, package builders, image builders, deployment. I would reuse the first to roles from the RPM/deb model, but then, the image builder takes the packages, combines them, builds an image, and then signs the result, which is then deployed. It's the job of the image builder to provide a manifest of what's in the image, building on the data the packager already provided.

Poettering: Fitting Everything Together

Posted May 10, 2022 19:31 UTC (Tue) by mrugiero (guest, #153040) [Link] (5 responses)

Hi Lennart. Overall I liked your vision, but I worry it would lead to much increased memory and disk space usage compared to the current state of affairs, which could be problematic for weaker/older hardware. Did you think of any measures to counteract this?

Poettering: Fitting Everything Together

Posted May 10, 2022 19:33 UTC (Tue) by mrugiero (guest, #153040) [Link] (2 responses)

Plus, I forgot to ask, how much of an effect do you think it would have in the lifetime of an SSD? It does look like A/B updates would write more than single package ones to me.

Poettering: Fitting Everything Together

Posted May 10, 2022 21:10 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

A half-way decent SSD?

I think some magazine did a "hammer it to death" test, and even writing to the SSDs 24/7 it took how long to kill them? 18 months? Three years? I really can't remember, but as I say the test was hammering them 24/7 and it was a looonnnnnngggggg time.

Rotating rust would probably fail quicker ...

Cheers,
Wol

Poettering: Fitting Everything Together

Posted May 10, 2022 21:22 UTC (Tue) by mrugiero (guest, #153040) [Link]

For the sake of argument, let's assume low end.
Thanks for the answer BTW, I think that's probably good enough.

Poettering: Fitting Everything Together

Posted May 11, 2022 11:32 UTC (Wed) by mezcalero (subscriber, #45103) [Link] (1 responses)

Given that other popular OSes (ChromeOS) work like this and are very widely deployed I am pretty sure hardware is just fine with this. In fact, am immutable OS should have written patters that should help lifetime of storage (i.e. not random write access for small stuff all the time, but instead seldom, large writes on OS updates only). So no, I didn't test it, but given what I am proposing here isn't precisely new stuff (other people built the hard stuff like dm-verity, I am just using them in this model) and is probably even more widely deployed already than traditional Linux is, I really don't see a problem here.

Lennart

Poettering: Fitting Everything Together

Posted May 12, 2022 16:45 UTC (Thu) by mrugiero (guest, #153040) [Link]

OK, that sounds satisfying. Thanks for your answer, Lennart.

Poettering: Fitting Everything Together

Posted May 10, 2022 21:30 UTC (Tue) by cyba (guest, #45803) [Link] (2 responses)

Lennart, you seem to ignore LVM completely. I don't use partition tables on my disks (they are all just PVs). LVs are much more flexible than partitions.

Poettering: Fitting Everything Together

Posted May 11, 2022 4:27 UTC (Wed) by edomaur (subscriber, #14520) [Link] (1 responses)

LV are much flexible, but they don't automate well so it's easier to keep using partition table (I did the same after having lost some sanity trying to work with LV in scripts)

Poettering: Fitting Everything Together

Posted May 29, 2022 11:04 UTC (Sun) by cyba (guest, #45803) [Link]

[sorry for late reply]

I'm not sure what problems with automation you have in mind, but I've been using LVM exclusively for many years (and I'm using LVs together with dm-raid and dm-crypt) and never had any problems that could be solved by using partition tables instead.

Poettering: Fitting Everything Together

Posted May 10, 2022 0:37 UTC (Tue) by flussence (guest, #85566) [Link] (33 responses)

This approach works great for something with the size and release cadence of OpenWRT; it's like changing the tyres and oil on a car.

However, it doesn't work so well on regular distros which are more the scale of a house, unless you're in the business of hoarding housing and renting it out.

Poettering: Fitting Everything Together

Posted May 10, 2022 0:56 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (31 responses)

It works fine in that model. The distribution becomes infrastructure - it's intended to be immutable for the lifetime of the image being deployed, and it's not where most of your applications live. Instead your applications are deployed via something like Flatpak and are broadly agnostic to the underlying infrastructure. This is, roughly, where MacOS is now - the OS partition is read-only and cryptographically verified - and somehow it's still a platform that people enthusiastically write code for.

Poettering: Fitting Everything Together

Posted May 10, 2022 4:51 UTC (Tue) by flussence (guest, #85566) [Link] (24 responses)

Yes, I've experienced macOS before. It was the only time ever out of three dozen devices that something flat out wouldn't connect to my home WPA2 network, and when I opened the control panel to try and diagnose it the entire window froze. I felt sorry enough for the owner of that thing that I just turned the security off while they were around.

I'd say something along the lines of "I hope Linux doesn't end up like that", but let's be real here, Android did all this right in 2012. Everyone else is playing catch-up.

Poettering: Fitting Everything Together

Posted May 10, 2022 5:00 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (23 responses)

And yet millions of people choose to use macOS on a daily basis, despite there not being the same level of market lock-in that encouraged people to use Windows in the 90s. So, sure, it's an OS that has bugs and pain points (like literally every other OS), but it's also a clear demonstration that a mass-market acceptable end-user OS can be produced with an immutable system partition.

Poettering: Fitting Everything Together

Posted May 10, 2022 7:08 UTC (Tue) by gfernandes (subscriber, #119910) [Link] (22 responses)

"Millions of people choose..." should most certainly NOT be the benchmark here.

The Mac is a boutique device. Does everyone crave a Ferrari? I'm sure they do. But the Common Man settles for a middle of the road Ford.

Which tend to still be repairable, for those who wish to do so. Quite unlike that Ferrari that perhaps once you laid your hands on, you'd probably be bashing it for being as unrepairable as a lump of iron.

So careful where you set your benchmark.

Poettering: Fitting Everything Together

Posted May 10, 2022 7:15 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (21 responses)

The discussion isn't about whether people should buy Macs. It's about whether it's possible to produce user-facing mass-market consumer devices running an immutable OS image. MacOS demonstrates that the answer is yes.

Poettering: Fitting Everything Together

Posted May 10, 2022 9:38 UTC (Tue) by rsidd (subscriber, #2582) [Link] (20 responses)

I scrolled and didn't find the answer, so asking: what happens if the immutable OS image has a bug?

I suppose there is a system update that would involve a reboot. I hardly ever reboot my Linux laptop. Would having an immutable image mean I would need to reboot every time there is a userland security update?

Poettering: Fitting Everything Together

Posted May 10, 2022 12:21 UTC (Tue) by mezcalero (subscriber, #45103) [Link] (19 responses)

If the bug is in the base OS image, then yes, the way to activate the next, newer version is to reboot the system.

If there's a bug in a portable service, or nspawn container, it's sufficient to restart those specifically. Similar, if you have a flatpak or OCI app on top, you restart that.

It's like this works on ChromeOS, Android and suchlike: if you update apps, the apps are restarted, if you update the OS itself, the system is restarted.

Lennart

Poettering: Fitting Everything Together

Posted May 10, 2022 12:22 UTC (Tue) by mezcalero (subscriber, #45103) [Link] (2 responses)

And of course, whoever prepares the base OS image should batch fixes, so that you don't have to reboot for every tiny bug fixlet. I figure ChromeOS and the lot won't push out updates more often than once per week...

Lennart

Poettering: Fitting Everything Together

Posted May 10, 2022 16:23 UTC (Tue) by johill (subscriber, #25196) [Link]

Actually it used to be every 6 weeks, now every 4 weeks (I believe).

They also have A/B images, of course.

But they only have a browser, so everything is actually in a single image - apart from the Android apps you have installed, of course.

Poettering: Fitting Everything Together

Posted May 10, 2022 19:36 UTC (Tue) by mrugiero (guest, #153040) [Link]

This reminds me of patch Tuesday on the Windows world. I think it's mostly a reasonable system.

Poettering: Fitting Everything Together

Posted May 10, 2022 12:48 UTC (Tue) by nix (subscriber, #2304) [Link] (15 responses)

That feels like a workaround for a shortcoming in fs-verity, frankly. What benefit does this have (other than "fs-verify can't do that, sorry") over some hypothetical system where you could update the verity info and restart affected installed apps *without* rebooting?

I mean, requiring a reboot brings no benefits at all that I can see, other than if the kernel changed: we already have the technology to restart apps affected by updates in numerous package managers, and the reboot *certainly* doesn't make people think "oh my machine just rebooted, I need to see if the OS image was replaced by a hostile attacker". End-user non-geek machines autoreboot all the time. They never know why and they certainly can't investigate: they just think of this as being computers being computers and annoyingly and unhelpfully throwing their work away. They think this is normal. Why on earth are we considering turning even *more* systems into something like this, when we should be *fixing* the problem?

It seems to me that the need for a reboot is purely a hack around a technical limitation. We should fix that rather than giving up and requiring reboots willy-nilly because /usr/bin/ls needed a bugfix.

Poettering: Fitting Everything Together

Posted May 10, 2022 21:51 UTC (Tue) by khim (subscriber, #9252) [Link] (4 responses)

The whole thing is just so much cargo-culting it's not funny.

Yes, all successful OSes have fixed base which is updated regularly (the infamous patch Tuesday in the Windows world).

Does it make them popular? Well… no.

What does? SDK.

All popular OSes are popular because there millions of apps available for them and for that to happen you have to have an SDK: an ability to actually write and deploy an app for the whole range of systems (including both 10 years old and latest and greatest ones) with full support for all the latest bells and whistles (Visual Studio 2022 still supports Windows XP!).

And if your OS is based on a stable and API-preserving base then it becomes natural to put it on a separate read-only partition.

But this attempt to take a bunch of packages and turn them into a single image? It's like an attempt to build a car by playing with different designs but without ever attempting to add wheels to it.

What's the point of all that if you still break ABIs every half-year or, maybe, even more often?

Poettering: Fitting Everything Together

Posted May 12, 2022 5:40 UTC (Thu) by rqosa (subscriber, #24136) [Link] (3 responses)

> you still break ABIs every half-year or, maybe, even more often

The problem of "no stable API for the developers to target" has been solved for a long time by now on desktop Linux distros: the end user's PC can have >=1 versions of >=1 multiple Flatpak runtimes installed at the same time, and the developers then target whichever ones they choose.

(Well, except that many commercial desktop-app/game developers have found a different solution: they just target the Steam runtime SDK instead. Either way, though, that problem has already been solved on desktop Linux.)

Poettering: Fitting Everything Together

Posted May 19, 2022 11:09 UTC (Thu) by mrugiero (guest, #153040) [Link] (2 responses)

That's everything but a decent solution really. "We aren't competent enough to keep a stable API for applications so just ship an OS with every app" is not a solution.
What happens when people target different distributions because of their preferences, with different versions, etc. Bloat is what happens. Wasted RAM because shared libraries aren't shared anymore. Wasted disk because the OS is not shared anymore. Longer boot times (OK OK that depends on implementation, but snap does that) because we mount the squashfs images on boot to pretend the apps actually launch as fast as native.
All of those solutions are embarrassing hacks.
Even WinSxS at least limits how many versions of the runtime you need to have on your computer.
It feels like the kernel community is wasting a lot of effort in keeping syscalls compatible if userspace will break everything anyway.

Poettering: Fitting Everything Together

Posted May 21, 2022 0:08 UTC (Sat) by raven667 (subscriber, #5198) [Link] (1 responses)

> That's everything but a decent solution really. "We aren't competent enough to keep a stable API for applications so just ship an OS with every app" is not a solution.

Okay...that perfect world where all commonly used libraries have stable ABI just doesn't exist, and given the widely distributed nature of Linux-based operating systems can probably never exist. We don't even have the base platform project cohesion that the BSDs enjoy for the core os let alone at the level that desktop apps need. While better stability is a goal to continuously strive for, out of the hundreds of distinct projects that make up a full desktop system, we do not have the man hours or authority to have long term, Windows-like binary compatibility of the whole stack.

Tools like flatpak are a response to the world as it actually is and what is achievable, not the unattainable perfection we wish it could be. Yes, we aren't "competent" enough for full stack compatibility without dragging around large runtimes, but there is no level of incentive or force that can be applied to all the developers who have graciously made the os to change that in an open system.

Poettering: Fitting Everything Together

Posted May 21, 2022 12:05 UTC (Sat) by mrugiero (guest, #153040) [Link]

There we agree. But call things by their name: a kludge. The problem hasn't "been solved for a long time". Closer to the truth would be to say it's unsolvable due to the nature of Linux desktops, so we chose to create a different problem instead.

Poettering: Fitting Everything Together

Posted May 10, 2022 23:41 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> It seems to me that the need for a reboot is purely a hack around a technical limitation. We should fix that rather than giving up and requiring reboots willy-nilly because /usr/bin/ls needed a bugfix.

But what if libz needs a bugfix? How do you find all the software that depends on it? What about static linking? What about GNOME itself that likely links it for some of the very core services?

I guess it'd be OK to have some sort of metainformation along with updates about services/apps that need to be restarted. Some updates can also be marked as "safe without restart" if they just tweak some utility.

And in practice with Windows and macOS you need a restarting update perhaps every 2-3 months. Right now my uptime on macOS is 45 days and I typically install patches right away.

Poettering: Fitting Everything Together

Posted May 11, 2022 11:44 UTC (Wed) by mezcalero (subscriber, #45103) [Link] (8 responses)

Updating the core system without reboot is unreliable madness if you ask me. It *mostly* works on RPM/deb based systems but that's mostly luck, and there are various components you cannot replace without reboot. kernel, dbus, and so on. Noone could possibly test all combinations of all library version upgrades with each other. It's not atomic, and in a world with IPC interfaces or where shared libraries need static data on disk that might vanish just a terrible concept. Moreover, there's no established way how we could even restart non-system stuff on package updates. So it's a terrible mess the way it is. We mostly get away with this because traditional Linux is so nerd-focussed, and if something doesn't work nerds are smart enough to know what you do.

But if you claim that complex system updates without reboot are a stellar, excellently working feature of the status quo ante, and I can just tell you that you haven't done your research. So, I sympathize with the goal, but as long as Linux is this hairy ball of stuff that is not isolated at all, I think updates without reboot are not achievable. if you want to support that, minimize the interface, put everything in a sandbox, so that you can start making guarantees about the stability of the holes in that sandbox, and reduce interdependencies, so that you address the combinatorial explosion of the potential update paths you'd have to test.

Or in other words: I am genuinely not interested in supported updates of core OS components without reboots. I am absolutely interested in making this work for portable services, nspawn containers, flatpak stuff, oci stuff, i.e. high-level, isolated stuff. But system components that integrate deeply into the OS? nope.

Sorry if that's disappointing.

Lennart

Poettering: Fitting Everything Together

Posted May 11, 2022 13:44 UTC (Wed) by bluca (subscriber, #118303) [Link] (6 responses)

Yes, this is pretty much the conclusion that everybody who seriously looks at this problem comes to. Beyond restarting basic system services (using things like https://github.com/liske/needrestart ) you can't really go - if your current Gnome session uses a library that gets update, how do you restart it without completely disrupting the logged in user? You don't. If you have to restart your GUI session, you might as well reboot.

There might be an argument to get smarter on what _type_ of reboot one does to squeeze as much time as possible out of the downtime, and an image-based OS would facilitate that a lot: eg, only /usr was updated, but firmware/kernel are the same? Only restart userspace (can we do that? Go to emergency.target and re-exec systemd?). Kernel was updated but not the firmware? Go via kexec.

There's also an argument for bringing back session restore after reboot in GNOME and friends. IIRC that was a thing once. The biggest annoyance about a reboot is losing current state, at least for me, and that would help a lot to mitigate it.

Poettering: Fitting Everything Together

Posted May 11, 2022 14:47 UTC (Wed) by fenncruz (subscriber, #81417) [Link] (1 responses)

>There's also an argument for bringing back session restore after reboot in GNOME and friends. IIRC that was a thing once. The biggest annoyance about a reboot is losing current state, at least for me, and that would help a lot to mitigate it.

It seems that apps themselves are re-implementing that. For me Firefox can reopen with last session and vscode reopens the open windows (and nicely keeps track of unsaved files, i guess by saving them somewhere else). I just need to find a terminal that saves state as well and it would be mostly there for my day-to-day life.

Poettering: Fitting Everything Together

Posted May 11, 2022 15:10 UTC (Wed) by bluca (subscriber, #118303) [Link]

True, most electron-based or browser-based applications do that. A lot of stuff doesn't though IIRC - terminals, editors, documents

Poettering: Fitting Everything Together

Posted May 12, 2022 4:25 UTC (Thu) by pabs (subscriber, #43278) [Link]

needrestart-session exists, for prompting the user to restart apps. It needs fixing for cgroupsv2 but it works fine, although it could do with some extra features.

For user services in GNOME sessions, it is definitely possible to restart most of them. There are some that can't be, like things run by gnome-session, or the gnome-shell processes and other related process.

I use this script for restarting parts of GNOME and root stuff that wasn't automatically restarted:

sudo needrestart -v
sudo bash -i
needrestart
for pid in `needrestart -b | tail -n +2 | cut -d ' ' -f2 | cut -d= -f2 | tr , '\n'` ; do
ps $pid
echo
systemctl --no-pager --user status $pid ||
systemctl --no-pager status $pid
PID=$pid bash -i
done
needrestart

In the inner shell I do either `kill $PID` or `systemctl --user restart foo.service` or `systemctl --user restart foo.target` as appropriate. eg dbus services can just be killed, systemd user services get restarted, systemd user services that are part of GNOME (especially gsd-*) often need a target restart instead of a service restart. A lot of this could be automated, especially if things switch from dbus to systemd user services, and if systemd gets some knowledge of restartability.

Poettering: Fitting Everything Together

Posted May 14, 2022 2:03 UTC (Sat) by patrakov (subscriber, #97174) [Link] (2 responses)

On some hardware (in particular, everything with an AMD APU), kexec is not viable. See e.g. https://lore.kernel.org/all/CADnq5_MdLTLvVdwFQJxuRaQcQFNk.... Besides, kexec is currently not compatible (for Microsoft compliance reasons, not for technical ones) with Secure Boot.

Poettering: Fitting Everything Together

Posted May 14, 2022 2:16 UTC (Sat) by mjg59 (subscriber, #23239) [Link]

kexec works fine with lockdown, it just requires that the kexec target be signed with a trusted key

Poettering: Fitting Everything Together

Posted May 18, 2022 11:26 UTC (Wed) by flussence (guest, #85566) [Link]

IME the amdgpu driver doesn't reset things properly for standalone GPUs either. I've got a boot script that just underclocks mine, because sometimes the fan(!) doesn't get reset to working state at reboot.

Poettering: Fitting Everything Together

Posted May 12, 2022 4:15 UTC (Thu) by pabs (subscriber, #43278) [Link]

needrestart has a solution for restarting dbus without rebooting. Of course it restarts a lot of other stuff too, like systemd init, the display manager and various other things. It would surprise me if it weren't possible to modify dbus to restart itself, similar to how systemctl daemon-reexec works. The Linux kernel has live patching, so that can be updated without reboot in some cases. Intel are even working on live firmware/microcode updates. All or almost all of systemd can be restarted these days. Even lots of parts of the GNOME desktop (gsd-* for eg) can be restarted. Of course not everything is safe to restart at all points in time. It would be nice if services could have a way to export that info, so that needrestart knows that no audio is playing right now, so it is safe for it to restart pipewire.

Poettering: Fitting Everything Together

Posted May 10, 2022 8:38 UTC (Tue) by bojan (subscriber, #14302) [Link] (5 responses)

I have been using a Mac for a while now (work) and upgrades/updates are painfully slow. I can never tell what is actually happening underneath. Updates are delivered rarely, probably because users would go nuts if they had to wait for them to finish more often.

On the other hand, upgrades of my two Fedora machines using dnf have been fast and painless for quite some time now. Patching daily (hourly?) is a breeze. Reverting to previous packages is a breeze too. Picking what I do and do not want is simple.

I don't know - delivering everything in big lumps that are supposed to suit everyone scares me. If I'm not mistaken, packages have checksums for /usr files already, so not sure why those are no good.

Poettering: Fitting Everything Together

Posted May 10, 2022 8:49 UTC (Tue) by danieldk (guest, #27876) [Link] (1 responses)

> On the other hand, upgrades of my two Fedora machines using dnf have been fast and painless for quite some time now. Patching daily (hourly?) is a breeze. Reverting to previous packages is a breeze too. Picking what I do and do not want is simple.

These are not inherent properties of package or image-based systems. OSTree uses a git-like store, requiring only fetching of changed objects. Updates on Fedora Silverblue and Fedora IOT have been very fast for me, both are OSTree-based systems.

> If I'm not mistaken, packages have checksums for /usr files already, so not sure why those are no good.

The traditional package model does not work great with immutable system volumes (which increase security) or atomic updates/rollback (though you can get some of that with file system snapshots).

Poettering: Fitting Everything Together

Posted May 10, 2022 11:46 UTC (Tue) by bojan (subscriber, #14302) [Link]

I am glad to hear it is just MacOS that sucks here. Because every time there is an update there, I can say goodbye to my laptop for about an hour.

Poettering: Fitting Everything Together

Posted May 10, 2022 8:50 UTC (Tue) by anselm (subscriber, #2796) [Link] (1 responses)

I have been using a Mac for a while now (work) and upgrades/updates are painfully slow. I can never tell what is actually happening underneath. Updates are delivered rarely, probably because users would go nuts if they had to wait for them to finish more often.

IIUC in the proposed model this won't be as much of an issue because you can update the “B” system while you're using the “A” system and vice versa, so the actual window during which the system is unusable can be quite brief.

Poettering: Fitting Everything Together

Posted May 11, 2022 12:39 UTC (Wed) by Baughn (subscriber, #124425) [Link]

TBH, the entire system feels a lot like "Duct tape papering over the fact that we're not running nixos".

Which has other problems, and this article isn't about it. But I truly wonder which approach will win out, in the end.

Poettering: Fitting Everything Together

Posted May 12, 2022 12:11 UTC (Thu) by njs (subscriber, #40338) [Link]

Like the blog post notes, ChromeOS is probably the closest existing OS to what Lennart's proposing, and ChromeOS upgrades are stupid fast. Like a few seconds at worst. And it makes sense... the actual upgrade is just flipping a pointer from old-image to new-image.

macOS upgrades are horrifically slow, and I don't understand why. But it's something silly they're doing, not a general problem with immutable OSes.

Poettering: Fitting Everything Together

Posted May 10, 2022 13:19 UTC (Tue) by eduperez (guest, #11232) [Link]

> This approach works great for something with the size and release cadence of OpenWRT,

OpenWrt has a nice "image builder", but it also has a package manager; and I do not think OpenWrt would be as popular as it is now, if users had to build and install a new image each time they required to use a different set of packages.

Poettering: Fitting Everything Together

Posted May 10, 2022 1:56 UTC (Tue) by Russell (guest, #1453) [Link] (39 responses)

Hope he doesn't leverage systemd to push this onto us all... But I think he will.

Poettering: Fitting Everything Together

Posted May 10, 2022 2:09 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (38 responses)

systemd can't magically change the way distributions build their distribution. If it happens then it's going to be because the people doing the work want it to happen - some people may choose to do that (and, for instance, I can certainly imagine someone in Debian choosing to build images based on this architecture), but the idea that Lennart can commit some code to a repo and then compel a number of volunteer led projects to engage in dramatic changes to their build systems ascribes a degree of power to him that he genuinely doesn't have (I mean, he could barely beat me in an arm-wrestling contest, and I have about the most pathetic arms imaginable)

Poettering: Fitting Everything Together

Posted May 10, 2022 6:38 UTC (Tue) by johannbg (guest, #65743) [Link] (4 responses)

Never arm wrestle with a German. They possess superior arm strength from most human beings from drinking all that beer...

Poettering: Fitting Everything Together

Posted May 10, 2022 17:12 UTC (Tue) by mpr22 (subscriber, #60784) [Link] (3 responses)

Have you ever been to an English town centre on a Saturday night?

Poettering: Fitting Everything Together

Posted May 10, 2022 19:50 UTC (Tue) by johannbg (guest, #65743) [Link] (2 responses)

Yes I actually have on more than one occation but those are just amateur nights compared to the Germans, The Germans drink beer on everyday that ends with an Y. What the Brits call a pint, the Germans call a shot and it's not just Arm wrestling they are good at but also Handball and have you ever seen a Brit throw a handball nope never all they can do, at best is try to kick a ball or hold it and run meanwhile the Germans have the best handball division in the world, the Handball-Bundesliga.

Poettering: Fitting Everything Together

Posted May 10, 2022 21:11 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

I thought Germans drank beer by the Stein. That's two pints at a time ...

Cheers,
Wol

Poettering: Fitting Everything Together

Posted May 11, 2022 8:55 UTC (Wed) by anselm (subscriber, #2796) [Link]

I thought Germans drank beer by the Stein. That's two pints at a time ...

That depends entirely on where you are in Germany. If you want the two-pint stein experience you need to go to Bavaria. Around Cologne, on the Rhine, a large beer is 0.2 litres, but they keep bringing you new ones until you explicitly signal that you have had enough.

Poettering: Fitting Everything Together

Posted May 10, 2022 10:35 UTC (Tue) by lobachevsky (subscriber, #121871) [Link]

Also, distributions can choose to not include these tools when building systemd and some will, at least for a time, choose not to. Just because that stuff exists in a repo, doesn't mean people will be forced to or casual users even able to use it on stable distros.

As a rough estimate: portabled arrived in Debian only on bullseye, homed, repartd, userdbd will only come with bookworm, newer components might not make bookworm+1. This really is something that most people will need to strongly opt in, i.e. build their own systemd package, to be able to use.

Poettering: Fitting Everything Together

Posted May 10, 2022 12:54 UTC (Tue) by nix (subscriber, #2304) [Link] (1 responses)

> the idea that Lennart can commit some code to a repo and then compel a number of volunteer led projects to engage in dramatic changes to their build systems ascribes a degree of power to him that he genuinely doesn't have (I mean, he could barely beat me in an arm-wrestling contest, and I have about the most pathetic arms imaginable)

I suddenly understand why so many ARM-based systems use image-based installation workflows! I didn't know Cambridge was such a hotspot of arm-wrestling... oh god last time I was there I had an arm-wrestling contest with my sister (she won, of course). ... maybe it's something in the water?

Poettering: Fitting Everything Together

Posted May 11, 2022 9:02 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

> I didn't know Cambridge was such a hotspot of arm-wrestling

Oh, this was in a bar in Porto Alegre, so not sure we can blame the Cambridge water for this one (and I hadn't lived in Cambridge for over 7 years at that point)

Poettering: Fitting Everything Together

Posted May 11, 2022 1:35 UTC (Wed) by JanC_ (guest, #34940) [Link] (24 responses)

Let’s be fair: it’s not about his arm-wrestling prowess, but about the financial/economic power of his employer, as well as about the dependencies of other projects on his projects.
Denying that he has those (and maybe other) powers behind him isn’t intellectually correct.

Poettering: Fitting Everything Together

Posted May 11, 2022 1:40 UTC (Wed) by mjg59 (subscriber, #23239) [Link] (23 responses)

Plenty of other people working for equally (if not more) powerful companies who could have done (or could still do) the same. Other projects weren't compelled to adopt dependencies on his work.

> Denying that he has those (and maybe other) powers behind him

"maybe others"?

Poettering: Fitting Everything Together

Posted May 11, 2022 3:36 UTC (Wed) by JanC_ (guest, #34940) [Link] (22 responses)

There aren’t really any more powerful companies in this particular space AFAICT, and certainly there aren’t many. But that’s not all that important; any company that owns a relevant project has some power.

Most of this influence isn't explicit, but it’s still there, and the same applies to you too: once you get a certain fame, you also get the power that comes with it…

But in that earlier reply you were trying to ridicule the power people like you two have, as if it’s about physical arm wrestling, which it really isn’t. Don’t be a hypocrite about that.

Poettering: Fitting Everything Together

Posted May 11, 2022 3:54 UTC (Wed) by mjg59 (subscriber, #23239) [Link] (21 responses)

A company having power due to it owning a relevant component doesn't explain how that component became relevant to begin with. systemd has relevance because it solved a number of problems that weren't previously solved - Upstart was the first real attempt to handle that but had some fundamental design issues (notably https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/4...) that never got fixed, and the decision making bodies in most major distributions independently decided to adopt it as the default init. What power did Lennart have to force this?

Poettering: Fitting Everything Together

Posted May 11, 2022 13:01 UTC (Wed) by johannbg (guest, #65743) [Link] (20 responses)

"the decision making bodies in most major distributions independently decided to adopt it as the default init."

Which is arguably is the reason why no distribution other than upstart/chromeOS properly integrated it into it's distribution and proves the point that it does not matter what those "decision making bodies" decide if there is no one willing to do all the (hard) work based on that decision.

Back in the day ChromeOS ( I assume it's still using upstart ) was willing to make the transition after I would have converted the entire distro as a proof of concept and the value been seen, which would have taken a good 3+ months probably a year of my life since I would be doing in my spare time. ( which I backed out of, not enough sacrifies of someone's lifespan for a "maybe" ) then there would be some transition issues on existing devices in the wild to be solved on top of that.

Anyway the point I'm trying to make here is that it's arguably thanks ( or not so thanks ) to us that did all the legwork integrating systemd into distribution that got systemd so widely adopted. Lennart, Kay, Zbyszek etc. they were busy writing code and preaching to the choire on conferences which should come as a no surprise since they were better suited for that task.

Poettering: Fitting Everything Together

Posted May 19, 2022 11:28 UTC (Thu) by mrugiero (guest, #153040) [Link] (19 responses)

> Which is arguably is the reason why no distribution other than upstart/chromeOS properly integrated it into it's distribution and proves the point that it does not matter what those "decision making bodies" decide if there is no one willing to do all the (hard) work based on that decision.

RHEL 6 shipped Upstart and Ubuntu did for several releases. In fact, it only migrated to systemd because having Debian adopt it by default made it extra hard for them to go against the mainstream.

Poettering: Fitting Everything Together

Posted May 19, 2022 11:44 UTC (Thu) by seyman (subscriber, #1172) [Link] (18 responses)

> RHEL 6 shipped Upstart and Ubuntu did for several releases.

Let's keep in mind that RHEL (and Fedora, its upstream) used upstart in its sysvinit-compatibility mode. No effort was made to convert sysvinit scripts to upstart-native jobs. The work needed to integrate upstart with the distribution was kept to an absolute minimum.

Poettering: Fitting Everything Together

Posted May 19, 2022 14:44 UTC (Thu) by mrugiero (guest, #153040) [Link] (15 responses)

> Let's keep in mind that RHEL (and Fedora, its upstream) used upstart in its sysvinit-compatibility mode. No effort was made to convert sysvinit scripts to upstart-native jobs. The work needed to integrate upstart with the distribution was kept to an absolute minimum.

I see. But still, it means it had two of the biggest players (and THE biggest player and the one that ended up backing systemd in the end) behind it, so clearly if systemd prevailed for RHEL it was because they judged it more useful. I don't think they'd just waste the effort they did already willingly.
Besides, it makes sense that effort to migrate to native jobs was kept to a minimum when the documentation of Upstart still states the format is not considered stable. Compared that to systemd, most if not all .unit files you wrote at the time of the Debian drama still work in current systemd. Doing that amount of work twice or more doesn't sound like a reasonable expectation.

Poettering: Fitting Everything Together

Posted May 19, 2022 19:19 UTC (Thu) by Wol (subscriber, #4433) [Link] (3 responses)

> I see. But still, it means it had two of the biggest players (and THE biggest player and the one that ended up backing systemd in the end) behind it, so clearly if systemd prevailed for RHEL it was because they judged it more useful. I don't think they'd just waste the effort they did already willingly.

The story as I heard it is that Upstart, while much better than SysVInit, had several big design flaws. Which was one (big) reason why RedHat didn't really convert, they couldn't see much benefit from it.

And whether it was upstream not wanting to fix the flaws, or downstream deciding it wasn't worth the effort trying, Poettering decided it was simpler just to start from scratch.

And where he scores is clean, simple design. So systemd winning out was a bit of a no-brainer, really, ...

Cheers,
Wol

Poettering: Fitting Everything Together

Posted May 20, 2022 14:24 UTC (Fri) by seyman (subscriber, #1172) [Link] (2 responses)

> And whether it was upstream not wanting to fix the flaws, or downstream deciding it wasn't worth the effort trying, Poettering decided it was simpler just to start from scratch.

IIRC, Lennart & co wanted to contribute to upstart. Canonical demanded they sign their CLA, Red Hat Legal said 'no' and systemd was born.

Poettering: Fitting Everything Together

Posted May 20, 2022 14:29 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

> IIRC, Lennart & co wanted to contribute to upstart. Canonical demanded they sign their CLA, Red Hat Legal said 'no' and systemd was born.

This is correct and CLA is the reason why the upstart developer didn't contribute to it after leaving Canonical as well. Refer to

https://www.reddit.com/r/linux/comments/1vpyle/comment/ce...

Poettering: Fitting Everything Together

Posted May 20, 2022 23:20 UTC (Fri) by mrugiero (guest, #153040) [Link]

I read about this at the time, but I forgot. I know a lot of people also complained about the use of CLA for Mir.

Poettering: Fitting Everything Together

Posted May 19, 2022 19:53 UTC (Thu) by johannbg (guest, #65743) [Link] (10 responses)

Red Hat was not particularly behind it so I'm not sure where that nonsense comes from...

Poettering: Fitting Everything Together

Posted May 19, 2022 21:22 UTC (Thu) by mrugiero (guest, #153040) [Link] (9 responses)

Not in the making. I mean in terms of being on board with it. Investing effort in it. Possibly evaluating investing money, assuming they didn't?

Poettering: Fitting Everything Together

Posted May 19, 2022 23:56 UTC (Thu) by johannbg (guest, #65743) [Link] (8 responses)

Given that I'm the one that handle most of the integration of systemd into Fedora either through my own "feature" proposals ( which more or less all dealt with components in the hundreds ) or behind the scenes for such proposal and spent ca 8 hours a day unpaid by Red Hat ( but got "Fedora" sponsored to BRNO for sd hackfests on few occations ) doing that work for years ( along with few others from the Fedora Community, not Red Hat ) I would say the answer to Red Hat investing in it would relatively be little to none.

And we did not get any "special" treatment pushing systemd into Fedora in fact what we got was the joy of having to work *twice* as hard than any feature proposal ever to be pushed into Fedora still to this day.

Fedora was not even the first distribution to ship systemd and the only Red Hat employee that was working on the project in the beginning was Lennart, maybe Harald as well. Frequently we tried to correc that systemd was not ( and still is not ) "a Red Hat project" and Fedora not being the first distro to use it but you know "the internet" where people read and think moronix and mashdot are reliable source of information and opinions are "facts".

And the reason why systemd got adopted in distribution is the collaborated work us distribution integrators did to achieve that. ( From Fedora,(Open)Suse, Arch, the embedded space etc ) And the reason why upstart ( with native upstart jobs ) never got adopted outside ChromeOS/Ubuntu was because no one was willing to do the required leg work to do that.

Writing an functional system and service management framework for everyone is hard, getting that framework integrated into distributions is harder...

Poettering: Fitting Everything Together

Posted May 20, 2022 15:25 UTC (Fri) by mrugiero (guest, #153040) [Link] (7 responses)

Thanks for the detailed answer. I was aware Arch migrated to systemd before pretty much all of the other popular distros (and, of the two families I actively used, it's the only that made a proper complete migration instead of just using the LSB support), but I was unaware of most of the other parts.

Regarding leg work, it's pretty much the problem with all the open source community, right? Everyone wants to be entitled to an opinion on how to run projects, but almost nobody wants to put the hours.

Poettering: Fitting Everything Together

Posted May 20, 2022 17:54 UTC (Fri) by johannbg (guest, #65743) [Link] (6 responses)

Arch went a bit overboard with it which created the systemd cult of "migrate" everything to systemd which infested Fedora at one point requesting everything should get migrated which is not always the right thing to do since obviously there always will be projects that will be simpler to use and provide better solutions to certain tasks/problems then systemd ever will.

Poettering: Fitting Everything Together

Posted May 20, 2022 19:17 UTC (Fri) by johannbg (guest, #65743) [Link] (5 responses)

Now one thing that I have noticed that the collective ignorance on the internet has not realized is the fact that systemd is reusing the same principals that IBM`s System Resource Controller used.

What is this IBM System Resource Controller you speak of and what was it used for?
IBM's SRC is an init system that was written for AIX back in 1992 and was the first init system to introduce the concept of using supervision semantic in a fully contained framework.

So be it systemd or for that matter Solaris SMF, they are just reusing the same principals that were used in last century so if you ( the reader ) intends to write an init system ( writing in rust seems to be popular these days ) you might want to look into those three init system ( as opposed to the alphabet inits as in jinit, minit, pinit, etc ) then try to meet up with Lennart and simply ask him what he would change in systemd now a decade later if he had to do it all over again ( or if he reads this, he goes hey that's a talk I could make on <insert some conference> )' and maybe just maybe you might be able to write a better init system which will out perform and thus eventually will replace systemd and get adopted by various distributions or even by the *BSD's if licenced correctly.

Poettering: Fitting Everything Together

Posted May 20, 2022 19:39 UTC (Fri) by mrugiero (guest, #153040) [Link] (4 responses)

I've read about that SRC and about SMF not too long ago. It's the rule rather than the exception that "novel" solutions are just things that were already invented and widely used in certain niches but everyone else chose to ignore when they were actually new. "The cloud" was invented by IBM as well IIRC. But I guess IBM is boring or something.

I don't think you mean precisely me writing an init system, but a very basic (toy) one would be an interesting learning project really (not one for real world use of course). I'm personally quite comfortable with systemd as it is, and that choice of Arch (which may or may not be wrong for some cases, but I don't think I stumbled upon those) is the main reason I stick to it.

Related, did you at some point write something like a blog post detailing the things you're telling me here? Could you send me a link? Or are you willing to write something in the future if not already? I think it would be a valuable resource for further discussion with others.

Poettering: Fitting Everything Together

Posted May 20, 2022 20:46 UTC (Fri) by johannbg (guest, #65743) [Link] (3 responses)

> Related, did you at some point write something like a blog post detailing the things you're telling me here? Could you send me a link? Or are you willing to write something in the future if not already? I think it would be a valuable resource for further discussion with others.

People that do work dont have time to write about things + I tried blogging in my early years sucked at it, so that would be no.

> IBM is boring or something.

IBM has never been boring, just not easily approachable, there is an barrier of entry.

Poettering: Fitting Everything Together

Posted May 20, 2022 23:25 UTC (Fri) by mrugiero (guest, #153040) [Link] (2 responses)

> People that do work dont have time to write about things + I tried blogging in my early years sucked at it, so that would be no.

Some people do work and write from time to time (Lennart for example). Regarding sucking, well, that happens. I wouldn't be a good writer either.
I guess I could try to follow your discussions in the mailing lists if I want to try and get something clean tho.

> IBM has never been boring, just not easily approachable, there is an barrier of entry.

Don't take it too literal. I meant mostly in the eye of geeks. I never see them say things like "look at this cool thing IBM did".

Poettering: Fitting Everything Together

Posted May 21, 2022 13:24 UTC (Sat) by johannbg (guest, #65743) [Link] (1 responses)

> Don't take it too literal. I meant mostly in the eye of geeks. I never see them say things like "look at this cool thing IBM did".

Yeah I dont know why that is, it's like people are unaware of IBM's history and origin which dates all the way back to 1880's and the fact they are using IBM's inventions in their every day life like for example when they use a card with magnetic stripe or use an ATM, gaming consoles etc. it's even been supporting Linux since ca 1998.

Unlike Google, Facebook, people can take great pride working at IBM since arguably IBM is the living, breathing manifestation of computer technology and it's history.

Poettering: Fitting Everything Together

Posted May 21, 2022 14:13 UTC (Sat) by mpr22 (subscriber, #60784) [Link]

The IBM in a lot of people's heads is the corporate behemoth with shenanigans characterised along the lines of your 'upgrade' consisting of a field service engineer coming out, removing the performance limiter, and changing the model number badge on the chassis.

Poettering: Fitting Everything Together

Posted May 19, 2022 21:08 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Ubuntu converted a few services to native upstart, but it was all kinda brittle. Especially in interactions with udev.

The other problem was that upstart was hacky. It used horrible ptrace-based code to follow the double-forking to keep track of the PIDs of the services. Very few people wanted to depend on _that_.

Poettering: Fitting Everything Together

Posted May 20, 2022 23:25 UTC (Fri) by mrugiero (guest, #153040) [Link]

Oh, so that's what ptrace was used for.

Poettering: Fitting Everything Together

Posted May 11, 2022 3:39 UTC (Wed) by Russell (guest, #1453) [Link] (4 responses)

systemd can make it difficult for distributions and competing projects by disrupting there dependencies and interfaces. This rhetoric about freedom of choice is nothing but gaslighting, ignoring the other dirty ways systemd attacks any competition. eg. depended on udev, you now have to maintain a fork...

Poettering: Fitting Everything Together

Posted May 11, 2022 3:47 UTC (Wed) by mjg59 (subscriber, #23239) [Link] (2 responses)

> depended on udev, you now have to maintain a fork...

Why? You can build udev out of the systemd source without it having a runtime dependency on systemd as init.

Poettering: Fitting Everything Together

Posted May 11, 2022 4:06 UTC (Wed) by Russell (guest, #1453) [Link] (1 responses)

But can you get a change that benefits something other than systemd?

Poettering: Fitting Everything Together

Posted May 11, 2022 6:13 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

What sort of change?

Poettering: Fitting Everything Together

Posted May 11, 2022 12:44 UTC (Wed) by johannbg (guest, #65743) [Link]

I dont see the argument here, those competing projects will just have to write things from scratch that work equal or better than systemd/udev to be replaced by distro and it's a decade dedication of hardwork in doing so.

And someday some project will but as far as I have seen thus far, no existing system and service manageframework/init is doing that, they all seem to be trying to avoid having to do all that work but instead are trying to re-use individual components of systemd, udev etc. then complain if they cant use it.

Poettering: Fitting Everything Together

Posted May 10, 2022 3:32 UTC (Tue) by wtarreau (subscriber, #51152) [Link] (7 responses)

For using both types of systems, I would say that three's no one-size fits all.

Using immutable images on appliances, remote devices, and infrastructure components that need to be always-on and really secure is both convenient and effective as it seriously limits the ability for an intruder to change components, plus it's easy to restart from a known state, and you know that upgrades always turn a stable system into a new stable system (but they usually require a reboot).

However, having tried to use this in the past on some user-facing devices (monitoring stations, web client, and lab machines), it was a nightmare. You constantly miss something and having to update your distro's description, rebuild one and reboot on it is painful, and you end up with a writable+executable directory mounted from a ramfs somewhere, where you place all your extra stuff that you were too lazy to install on a new image.

I would say, use immutable for unattended production, and use packages where there are users.

Poettering: Fitting Everything Together

Posted May 10, 2022 3:37 UTC (Tue) by mjg59 (subscriber, #23239) [Link]

There's no inherent incompatibility here - an immutable image that provides infrastructure with a limited set of guarantees about which interfaces will remain stable, plus writable storage that contains additional apps and their dependencies (potentially with some degree of enforced sandboxing) is a model that has been demonstrated to work on a range of mass market consumer devices, including general purpose computers.

Poettering: Fitting Everything Together

Posted May 10, 2022 8:06 UTC (Tue) by mezcalero (subscriber, #45103) [Link] (5 responses)

If you read the blog story you'll find that there are multiple approaches presented to make the immutable system behave in a less immutable fashion without losing its good properties for hacker type people, who quickly want to modify/add something in locally. Modularity matters, and hackability does too. I don't think it's in conflict with the basic premise of an immutable OS base image and a full trust chain.

Lennart

Poettering: Fitting Everything Together

Posted May 10, 2022 10:38 UTC (Tue) by edomaur (subscriber, #14520) [Link]

Rancher did some work like that with RancherOS, but it has been deprecated in favor of K3OS. Anyway, an interesting aspect of the system was that you could choose between various OS personalities.

Poettering: Fitting Everything Together

Posted May 10, 2022 18:10 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

What about using an overlay FS on top of /usr?

This would allow developers to quickly customize stuff in /usr, while providing an easy way to roll back to 'factory' settings. Or to create a diff of the local changes, etc.

Poettering: Fitting Everything Together

Posted May 10, 2022 22:27 UTC (Tue) by walters (subscriber, #7396) [Link] (2 responses)

Yes, it works really well: https://github.com/ostreedev/ostree/pull/212/commits/6748...

However, overlayfs doesn't support mutating the lower dir - it can't magically solve the merge problem when the base image changes. The only correct way to handle that is to capture changes individually and re-union them.

Poettering: Fitting Everything Together

Posted May 10, 2022 22:35 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> The only correct way to handle that is to capture changes individually and re-union them.

If there's no real difference between modifying stuff in the overlay and in the base system, then why bother? Sure, it's fine to have a tool to meld together the base OS and the overlay changes, but it's not really essential.

Poettering: Fitting Everything Together

Posted May 13, 2022 13:29 UTC (Fri) by wtarreau (subscriber, #51152) [Link]

Note that one of the benefits sought from using R/O FS is to prevent too easy a replacement of what you have on your FS. Of course one could always bind-mount a binary on top of another one (and I've done that many times for debugging) but when dealing with additions it's anthoer story. This *does* give great confidence about what is running, not in the sense of file-signing but rather in terms of "it's so difficult to run with non-standard stuff that I'm quite certain it's still genuine". Dealing with remote system management, writing replicable procedures from a copy in the lab to deploy on 150 hosts etc works pretty well in such cases.

Poettering: Fitting Everything Together

Posted May 10, 2022 3:36 UTC (Tue) by pothos (subscriber, #116075) [Link] (3 responses)

Regarding the problem of the changing "usrhash=" kernel parameter that makes the TPM2 PCR hash vary and break the LUKS unlocking: I think it would be interesting search for solutions to have this value be stored elsewhere in the unified EFI kernel image (How about the .osrel section?) instead of passing it as kernel parameter.

A missing piece in the proposed OS design is that the A/B update mechanism doesn't update the /etc config files that got created on first boot. This is sometimes needed for software that has required values in its /etc config file that get incompatible between versions. In general it's good to keep these files aligned with what the OS under /usr would create on a fresh install - of course unless the user touched these files. To migrate user changes the A/B update needs 3-way merge of configuration files in /etc, taking the old /usr factory file and the new /usr factory file into account (to my knowlegde this is done in rpm-ostree as used by Fedora CoreOS).

Poettering: Fitting Everything Together

Posted May 10, 2022 8:16 UTC (Tue) by mezcalero (subscriber, #45103) [Link] (1 responses)

Moving the usrhash= value elsewhere doesn't really change things: we must include it in the data to verify during SecureBoot authentication. The kernel command line is as good as the .osrel PE section there. Having signed PCR hashes isn't a big problem: the logic in the TPM2 allows for that, it's just a matter of hacking support for this into systemd-cryptenroll/systemd-cryptsetup.

systemd has concepts for running update services on boot that are triggered by /usr/ having been updated. See ConditionNeedsUpdate= in the docs. It allows you to run services whenever /usr/ has been updated, to catch up with changes in /usr/ and apply them to /etc/ or /var/. Note that in an ideal world things like that wouldn't be necessary though, because everything is declarative. For example, systemd-tmpfiles + systemd-sysusers data is declarative in the sense that they describe the intended outcome, and not the way to get there. This makes updates relatively easy and safe: check if the declaration matches what is so far in place, and adjust it if necessary. Otherwise skip the operation, in an idempotent fashion. That means, tmpfiles/sysusers can run on every single boot, but most of the time they'll not do anything but just exit quickly because everything is already order – until you apply an update, in which case they might readjust/add some stuff here or there.

But no, we don't do 3-way merges of config files. If this is necessary, then implement that via a ConditionNeedsUpdate= service. I think the problem is not as pressing in a world where /etc/ is basically empty though, as proposed in this model: if there's no redundant, default config in /etc/, then there's no need to merge that with newer versions on update.

Lennart

Poettering: Fitting Everything Together

Posted May 12, 2022 16:51 UTC (Thu) by dbnichol (subscriber, #39622) [Link]

I think the handling of /etc and /var on update is the dirty secret of image based systems. We use ostree at Endless and have a collection of gross ConditionNeedsUpdate services that run on update to fix various things gone wrong on earlier releases or handle migration of existing settings. I often have to tell new developers that they need to consider the update case and not just booting a new image with their proposed change in it.

These are the type of things that are done in the package world with maintainer scripts. While a lot of the things in maintainer scripts (e.g., restart services, rebuild the ldconfig cache) are unnecessary in the image based world, there are many legitimate things that happen in maintainer scripts that would need to get reimplemented as post-boot services. I've often thought about a more generic system that runs things like maintainer scripts installed in a directory via a ConditionNeedsUpdate service.

It would be nice if packages/applications handled these cases (empty /etc and /var or migration of old settings/data) on their own, but that's not currently the world we live in.

Poettering: Fitting Everything Together

Posted May 12, 2022 13:30 UTC (Thu) by njs (subscriber, #40338) [Link]

Another way to fix this issue, though maybe more of a long term thing, would be to fix the software so it knows its own defaults and /etc is only used for local overrides.

Poettering: Fitting Everything Together

Posted May 10, 2022 4:48 UTC (Tue) by weberm (guest, #131630) [Link] (17 responses)

Lennart can dreamingly put together all the systems he wants until they start using the systemd gravity leverage to force it all upon us. "But it's for the greater good, everybody!"

I get so angry with such a politicized systemd usage.
"Well, we didn't consider your corner case or system design or use of previously-existing-and-normal-but-now-deprecated-features, you're SOL, adapt!"

Next up: systemd won't boot a system that hasn't been updated by systemd-sysupdate (f u, RAUC et al users, why don't you yell at RAUC et al to drop their features and use systemd-sysupdate under the hood?!); who cares about A/R systems, you gotta go A/B; storage space is cheap anyways, eh?! Ooooh you're not following our ideal concept of signing or immutability? Get lost, "legacy" system! Oh you're not EFI? or have TPM2? "Obviously security is weakened on your platform and beginning with systemd vX we will no longer run on your inferior platform!"

If F/OSS was the equivalent of a direct democracy empowerment movement, systemd is the complete opposite. A hopefully benevolent entity pushing their world-view on everybody. Now you say, but people willingly adapt this based on its merit! Well, you adapt one tool and you're in a world of hurt until you adapt the rest. There's a mass psychology at play as well, one that systemd can't just "innocently" ignore and keep using their position as leverage at the same time.

The kindest way to describe systemd folks' development is to say "they're on a mission", but so was, e.g., the catholic church as well.

Poettering: Fitting Everything Together

Posted May 10, 2022 5:46 UTC (Tue) by motk (subscriber, #51120) [Link]

systemd derangement syndrome is real, folks.

Poettering: Fitting Everything Together

Posted May 10, 2022 7:31 UTC (Tue) by HenrikH (subscriber, #31152) [Link] (14 responses)

You anti-systemd trolls are so tiresome. All of these ideas of Lennart using some kind of leverage (that he does not even possess) to force any forms of change upon the masses is just all in your head.

Had you even bothered to read the actual blog post:

I have no illusions, general purpose distributions are not going to adopt this model as their default any time soon, and it's not even my goal that they do that. The above is my personal vision, and I don't expect people to buy into it 100%, and that's fine. However, what I am interested in is finding the overlaps, i.e. work with people who buy 50% into this vision, and share the components.

Poettering: Fitting Everything Together

Posted May 10, 2022 10:25 UTC (Tue) by weberm (guest, #131630) [Link] (13 responses)

I did bother to read the post. Given I professionally bake systemd-centered distributions I think I hardly qualify as systemd troll either.

That being said, I still stand by my points, given I had to adapt our systems multiple times in both distro architecture on the macro and micro scale to adapt changes that depart from long-standing designs (especially the essence of the UNIX koans) with no benefit to us because systemd deliberately chooses to reduce its compatibility surface to "those who follow our mission".

If you cannot discern the technological side of looking at each proposal / blog post / systemd-X tool / "module" in isolation on one hand and the political impact of acknowledging that none of these work without the full monolith and specific requirements on the run-time environment on the other, I kindly ask you to reconsider your authority to judge criticism and/or personal reflection on systemd's missionary agenda as trolling.

Poettering: Fitting Everything Together

Posted May 10, 2022 12:18 UTC (Tue) by bluca (subscriber, #118303) [Link] (6 responses)

When you post nonsense like this:

> Next up: systemd won't boot a system that hasn't been updated by systemd-sysupdate

You should hardly be surprised to be called out for trolling

Poettering: Fitting Everything Together

Posted May 10, 2022 13:00 UTC (Tue) by weberm (guest, #131630) [Link] (5 responses)

When Lennart posts stuff like:
> (Oh, and in case you wonder, all of this only works with distributions that have completed the /usr/ merge. On legacy distributions that didn't do that and still place parts of /usr/ all over the hierarchy the above won't work, since merging /usr/ trees via overlayfs is pretty pointess if the OS is not hermetic in /usr/.)
(from: https://0pointer.net/blog/testing-my-system-code-in-usr-w... )

...you should hardly be surprised that I expect systemd to require compliance to their world-view in order to use a feature.

Poettering: Fitting Everything Together

Posted May 10, 2022 13:09 UTC (Tue) by bluca (subscriber, #118303) [Link] (4 responses)

Apart from the fact that merged-usr is a good and common-sense approach that has nothing to do with systemd, that is simply clarifying what should be a no-brainer: you can't have an hermetic /usr (and use tools that deal with hermetic /usr) if you ship stuff outside of it. Nothing to do with "world-view", just basic logic.

Poettering: Fitting Everything Together

Posted May 11, 2022 4:20 UTC (Wed) by weberm (guest, #131630) [Link] (3 responses)

So your "basic logic" demands a merged /usr to provide an overlay mount?
It's your "basic logic" that prevents you seeing a use for extending an existing /usr, no matter whether it includes previously-/ or not?

Mind expanding on that? On how I, the user, or I, the developer, or I, the distributor can't possibly think of perfectly valid, sane and working use-cases without a merged /usr? because of your "basic logic"? Why would I need a merged-/usr to add a view to, say, gdb onto an otherwise r/o system installation; what in your "basic logic" prevents that from working on a "legacy" system aside of "world-view" and "mission"?

Poettering: Fitting Everything Together

Posted May 11, 2022 7:03 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

If your libraries are spread between /usr and /lib, you can't atomically swap them or use features like btrfs snapshots. You'll either have to snapshot the whole root volume, or manage two separate trees (perhaps with a bind mount aliasing).

Doable, but far less robust in practice.

Poettering: Fitting Everything Together

Posted May 11, 2022 9:34 UTC (Wed) by weberm (guest, #131630) [Link] (1 responses)

If my package properly installs under DESTDIR != /, my libraries won't be in /.
Doable, and as robust in practice while your stuff doesn't want to install in /.
That's the kind of limited imagination of use-cases that makes it seem like choosing to arbitrarily limit the theoretical capabilities for the visitors of the same circus as the one you happen to run.

Poettering: Fitting Everything Together

Posted May 11, 2022 10:34 UTC (Wed) by bluca (subscriber, #118303) [Link]

You are completely missing the point. Go read the blog post again, especially the part around a single partition format used for host, guest, extensions, applications. Also worth noting that in 2022 there is not a _single_ valid reason why somebody would want a split-usr system, apart from "hurr durr change bad". There's a reason every distro that actually matters has already switched or at worst is in the process of doing so.

Poettering: Fitting Everything Together

Posted May 10, 2022 13:37 UTC (Tue) by ovitters (guest, #27950) [Link] (5 responses)

> I think I hardly qualify as systemd troll either.

If you don't want to qualify as a troll, why is your initial post so full of trolling?

Example:
> until they start using the systemd gravity leverage to force it all upon us

It's the same repeated statements for many years. The first distributions to default to systemd were released around 10 years ago. And still there's this conspiracy talk about being forced to do things.

> That being said, I still stand by my points

So you're not trolling and you're trolling. Great.

Poettering: Fitting Everything Together

Posted May 11, 2022 5:41 UTC (Wed) by weberm (guest, #131630) [Link] (4 responses)

I'll use an analogy to show what I mean with systemd gravity leverage:

Suppose linus decides linux drops support for a file-system and the official git mirror of linux does so. Now suppose you have installed using a dropped file-system. Your distribution's kernel of choice dropped support for it, so .. SOL?

With regards to the developers of said file-systems, this is linux gravity leverage being used to corner them out. No conspiracy involved here. Also you as a user aren't forced to up-date, right? So you're not forced to do anything drastic like, change the setup of your box, are you?

Now suppose there's no technical reasons for dropping that file-system support, but rather ideological ones.

If this analogy doesn't help you understand what I'm talking about, you're free to continue calling me a "troll", of course.

Poettering: Fitting Everything Together

Posted May 11, 2022 6:57 UTC (Wed) by Wol (subscriber, #4433) [Link] (3 responses)

What's that phrase? "Survival of the fittest"?

You are describing reality here. Shit happens. Deal with it. It's nothing special to do with systemd. Or linux. Or computing ...

Troll or not, you're railing against life doing what life does. Tough. (I know, I do the same sometimes, but it's a waste of time ...)

(Oh, and we've been talking about "pets and cattle" - if it's a pet system then you damn well *should* upgrade. Just buy a new hard drive and copy it across. If it's a cattle system, who cares. Just roll out a new image.)

Cheers,
Wol

Poettering: Fitting Everything Together

Posted May 11, 2022 9:40 UTC (Wed) by weberm (guest, #131630) [Link] (2 responses)

I AM dealing with it. I was just sharing a sentiment that the way especially the systemd folks communicate and randomly castrate the theoretical compatibility and usability of features outside of their own mission/mindset makes me angry. Sure, I was ranting. The real time waste is/was defending and explaining my position, and I will stop that right after the next paragraph.

There are other "survivors" that do a far better job to be mindful of the people they bewilder along the way, or leave behind, or take a better approach to being less exclusive and more compatible, or actually be modular with stable APIs in between their nodes so 3rd party can replace bits and pieces.
With victory comes a certain responsibility, but first you gotta realize that indeed you have won and can stop fighting now and be more inclusive with your users.

Poettering: Fitting Everything Together

Posted May 11, 2022 12:06 UTC (Wed) by mezcalero (subscriber, #45103) [Link] (1 responses)

Please make a list of the supposedly numerous cases where we cut people off their favourite feature. I mean fuck it, you can even still build udev from the systemd tree and use it outside of it, even though I think it's a bad idea. We have been pretty good at maintaining compat and support the status quo ante.

I am not sure how you can be so upset that we design certain new features with only modern systems in mind. Yes, sysext stuff, and so on only makes sense with a merged /usr/ tree. But it's a *new* tool, so you can as well stay with the old mechanisms if you hate merged /usr/ so much.

You imply we cut people off perfectly well working stuff all the time on a whim. That's just simply not the case. So please, list all the cases where we willfully cut people off. I am curious what you'll come up with.

Lennart

Poettering: Fitting Everything Together

Posted Jun 15, 2022 11:44 UTC (Wed) by nix (subscriber, #2304) [Link]

There seems to have been no response to this. Odd that. Maybe the list is soooo huge it's still being compiled? ;)

Poettering: Fitting Everything Together

Posted May 10, 2022 7:58 UTC (Tue) by smurf (subscriber, #17840) [Link]

Yeah. sure they're on a mission. A mission to throw out (some of) the existing sub-optimal tools from ancient history and replace them with something that actually works for you instead of something you have to fight against (up to and including succumbing to Stockholm Syndrome (sorry, no offense to anybody who dislikes systemd intended, but I can't help to think along these lines)).

Likewise here. The "immutable base OS image" idea isn't new (even disregarding Android). However, to "factory reset" a laptop after it's been in the hands of, well, whoever, is basically impossible on any current "packaged" desktop Linux installation. Ditto atomic upgrades. Ditto thwarting most Evil Maid scenarios. Ditto just plugging your stick with your home directory (with the Flatpaks containing the three not-commonly-installed applications you need) into basically any Linux desktop and, well, that's it, you're done, log in start working.

It should not be.

Poettering: Fitting Everything Together

Posted May 10, 2022 5:35 UTC (Tue) by iabervon (subscriber, #722) [Link] (3 responses)

I like the idea of an immutable /usr which consists of the contributions from all of the packages that are presently installed. Rather than writing files to /usr or removing them from /usr, installation would simply change which package (versions) contribute, so there's no possibility of files left behind or misattribution of files to packages.

I think containers and images get a bad reputation from starting from a base image that's who-knows-what and getting the initial running state from doing whatever install steps to it. I want system installs to be that you start with a blank filesystem and a package manager outside of it installs packages into it; a purely package-based installation without any bootstrapping weirdness where your package manager and its dependencies are in your system when you first start running it.

Poettering: Fitting Everything Together

Posted May 10, 2022 7:11 UTC (Tue) by WolfWings (subscriber, #56790) [Link] (1 responses)

Honestly I could see a system where everything (and I mean EVERYTHING) in /usr and several other locations is just symlinks into RO mounts elsewhere.

Then system infrastructure packages are just (possibly compressed) read-only filesystems mounted somewhere and the only package management is updating the symlinks in an updated /usr copy and swapping it into place.

Which is more or less a Linux-esque variant of Windows WinSxS in a lot of ways I realize. Just with RO filesystems instead of just ACL-locked directories.

In theory it could even be taken to the point where the /usr versions could be swapped out on the fly, and the old ones could be kept around to allow a near-instant "Whoops, nevermind!" revert of updates.

Poettering: Fitting Everything Together

Posted May 10, 2022 10:31 UTC (Tue) by atnot (guest, #124910) [Link]

This is basically how NixOS works, except it adds reproducible buildings and a bunch of linker hacks etc. to make everything work deterministically.

Poettering: Fitting Everything Together

Posted May 10, 2022 7:58 UTC (Tue) by pabs (subscriber, #43278) [Link]

Debian kind of supports this, using the DPKG_ROOT variable for .deb maintainer scripts.

https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap

Poettering: Fitting Everything Together

Posted May 10, 2022 8:14 UTC (Tue) by JMB (guest, #74439) [Link]

I think that the views differ enormously when switching targets.
When in industry I used a lot of techniques which were important in these use cases
while I would never use them on the desktop of an expert doing creative work
(not saying that people are not experts when working in industry - but not all are
computer experts working 8+ hours with computers incl. their configuration and
the need of improving efficiency with computers - so using e.g. mice instead of
keyboards, or even touchpads not suited for qualified/creative work).

Maybe I never liked Poettering's views and it may just be this different points of view.
I created images and put them on workstation in industry twenty years ago - but when
I see my systems they may not differ a lot - but they differ - and for a reason.

And I still want my base being a distribution built from a fixed set of libraries which
are only switched by compatible ones fixing bugs without changing its functionality.
What makes sense for the kernel should be the same for libraries - new functionality
would mean new functions - nerver altering old ones to lose compatibility.
Rolling kernels don't harm - currently no one will like to apply this on libraries,
as library developers have a different view - which makes the desktop not a nice place.
But using fresh and stable versions of applications which I need to have current and
with full functionality - not being able to wait several years (GCC, TeXLive, you
name it) - but built on the fixed set of libraries. No snap, not flatpak - packages of
the used distribution (and yes, due to my experience I prefer deb packages over rpm ones).

These, snap and flatpak and alike, are not appropriate for systems I want to use - and images are
used in my work only to create backups - i.e. seldom taken snaps ... and frequent incremental
backups for working fields which change a lot.
So I use happily ext4 and would not wait for anything BTRFS and friends have to offer
(which are sure important for some use cases), but rely on robustness and recoverable errors.
Using fixed sized partitions (much larger than needed) and not using LVMs - and also
no RAIDs. I know them, but no benefit but lots of problems if applied to my desktop systems.

One size fits all won't ever work ... and while industrial use cases are in focus with the LTS releases,
they are silly to apply on desktops (LTS kernels are a really bad idea to be used on desktops - but
may be appropriate for smartphone and servers as these are typically mere industrial products).
I hope that people will learn this - otherwise Red Hat/Canonical etc. may bring their taste as
something everyone needs ... or has to use in the end.
And normal desktop users are forced to test industrial products for free (not being paid) but
have to pay the price to have no longer choices - use GNOME, use flatpak/snap, use systemd,
use Wayland - and alternatives need additional care in infrastructure which not even Debian
can provide - and I understood the struggling to get all maintainers to do additional work
e.g. for different system management/booting infrastructures.
But having only one - which is magically the best - is nothing I would call reasonable.
This is not a world of free users - but a world of companies getting everything for free
what many people improved in their spare time.
And I discovered bugs when employed in IT business well known IT giants would not even
think to be possible and first reacted that these bugs could not exist as I was the only one
reporting them - and this happened several times.

Both worlds are important and both can benefit from each other - the current vision of one united suit
(or better uniform) may not end the way those creators had in mind.
And I am not yet inclined to give names of political systems to that directions ...
but everyone may know which direction resembles which political system, right?

Poettering: Fitting Everything Together

Posted May 10, 2022 12:09 UTC (Tue) by scientes (guest, #83068) [Link] (3 responses)

> Hermetic /usr/

Yes, please.

> 14. Things should be reasonably democratic and hackable. It should be easy to fork an OS, to modify an OS and still get reasonable cryptographic protection. Modifying your OS should not necessarily imply that your "warranty is voided" and you lose all good properties of the OS, if you so will.

The first generation of security keys were divested from the companies that designed them, but still connected to the mainframe and expensive (something Tim Ansell commented on https://tomu.im/fomu.html ). However there are some that can actually store keys, and log you into something other than the mainframe, but they still probably should just ship with TPM2 chips and something similar to usb mass storage interface......

> systemd-sysusr

Lennart Poëttering is quoted saying "SELinux is a great technology, but I don't understand it." The original UNIX #5 from Bell Labs (except for tiny 16-bit UID/GID) can actually do absolutely everything if you:

1. Let go of any preconceived notions of how a filesystem must store UIDs and GIDs (I spoke with Eric W. Biederman in Oakland with Mark Shuttleworth and while he awknowledged that UIDs and GIDs are sticky, and they are, he definitely did not have any preconceived notions that filesystems must store them in any specific way.)

2. Turn the Group table into a Directed Acyclic Graph.

Poettering: Fitting Everything Together

Posted May 10, 2022 12:55 UTC (Tue) by jem (subscriber, #24231) [Link] (2 responses)

>However there are some that can actually store keys, and log you into something other than the mainframe, but they still probably should just ship with TPM2 chips and something similar to usb mass storage interface......

Source, please? Or are you just spreading FUD?

Poettering: Fitting Everything Together

Posted May 12, 2022 13:02 UTC (Thu) by scientes (guest, #83068) [Link] (1 responses)

I had some of those first keys, and then are just basically random-number-generators, like the WoW tokens. Thus they have to coordinate with the corresponding liniar random-number-generator of the main-frame, and cannot be used with PAM or other offline authorization methods. It was REALLY stupid to start a company selling security when the thing you are selling just has to come back to you......incredible delusions of grandeur, but that is exactly what the first generation was. The RSA keys and smartcards were actually a thing, but these things are really stupid and over-priced, although FIDO2 actually could be made to work.

Poettering: Fitting Everything Together

Posted May 12, 2022 15:41 UTC (Thu) by jem (subscriber, #24231) [Link]

>The RSA keys and smartcards were actually a thing, but these things are really stupid and over-priced, although FIDO2 actually could be made to work.

Is a ~50 USD Yubikey capable of RSA and ECC over-priced? It is not a toy, nor stupid.

Getting FIDO2 to work with Google (for example) is child's play.

Question for mjg59 and mezcalero on image sizes

Posted May 10, 2022 12:22 UTC (Tue) by smoogen (subscriber, #97) [Link] (5 responses)

What is the envisioned expected image sizes for these A/B images? We get about 2 kernel updates a week to even the stable trees (with many of them listed as must update), so a secure aware distro may be looking at needing to build and set up for 'pulling' a kernel and a N {mega,giga}byte signed image every week since the kernel and image are 'paired'. I expect that the image would need to be 'minimalized' but is there an expected size and transfer mechanism?

My first thought is the need for 'yet another partition' which is basically a download scratch space to get things down to correctly checksum and build out either A/B partitions or the various 'flatpaks' and other stuff needed to layer ontop of the A/B images.

Question for mjg59 and mezcalero on image sizes

Posted May 10, 2022 12:34 UTC (Tue) by mezcalero (subscriber, #45103) [Link] (2 responses)

The `systemd-sysupdate` logic doesn't address the issue of minimizing images, it just manages the files/partitions/directory trees on disks, it then invokes an external tool (currently just a fancy wrapper around libcurl) for the downloads. My idea would be to make that pluggable to some degree, so that external stuff can be swapped in instead. i.e. my own `casync` tool would be one candidate there, but rsync might be fine too, or zsync or whatnot. I see the data transfer question separate from the local version management/candidate management question. The latter, with its semantics around the discoverable partition spec, boot loader spec and so on really belongs in systemd itself I think (and is what systemd-sysupdate implements). The transfer thing probably not so much.

So depending how good the download tool you use is, results will be worse or better. The plain HTTP download of course will be expensive for large base OS images... I think tools like casync could be much much better there, in particular if they acquire some basic understanding for the payloads to optimize "cut points", i.e. if they know how a squashfs is put together, they can increase the chance of large reuse. People could even use that highly optimized disassembling chromeos download back-end if they like, which should certainly give better results than RPM update downloads I guess.

But if you want hard real-life data, I can't give you that...

Question for mjg59 and mezcalero on image sizes

Posted May 10, 2022 14:02 UTC (Tue) by smoogen (subscriber, #97) [Link]

No thanks.. that gives me an idea of what to look at.

Question for mjg59 and mezcalero on image sizes

Posted May 10, 2022 15:46 UTC (Tue) by pablotron (subscriber, #105494) [Link]

Tangential, but Raymond Chen wrote a series of posts a couple of years ago which update distribution methods that Microsoft has tried, what they settled on, and why:

https://devblogs.microsoft.com/oldnewthing/20200213-00/?p...

Question for mjg59 and mezcalero on image sizes

Posted May 10, 2022 16:24 UTC (Tue) by kerneis-anssi (guest, #158435) [Link] (1 responses)

The blog post mentions:

> In my model it would be up to the image builders to minimize shipping overly redundant disk images, by making good use of suitably composable system extensions.

That's where a filesystem like Fuchsia's blobfs would be handy: https://fuchsia.dev/fuchsia-src/concepts/filesystems/blobfs

Question for mjg59 and mezcalero on image sizes

Posted May 11, 2022 12:09 UTC (Wed) by mezcalero (subscriber, #45103) [Link]

Yes, I'd *love* to have blobfs on Linux. That would be stellar, and fit perfectly into my view of the world.

Lennart

Poettering: Fitting Everything Together

Posted May 10, 2022 14:04 UTC (Tue) by dutch_wolf (guest, #93834) [Link] (5 responses)

Though I do like this idea of immutable /usr I do think in practice some things still need to be figured out, like what to do with new hardware, sure e.g. intel and amd are these days pretty good with upstreaming their stuff in a timely matter but can often still require fairly new kernels or user space (mesa mostly), for user space it might be possible to push it into a sysext but that won't work for the kernel. This will need to be solved somehow (in other words how can we make sure people can use their shiny new hardware while their favourite distro is always a couple of kernel version behind?), another issue is that depending on what gets into sysext vs base image vs application images the images can get huge so some kind of delta update would be needed (and lets not forget disk size requirements!)

I do think that the above issues (hardware support and download size) are solvable and look forward to an immutable future!

(as a tangent how good is systemd-homed with large homes, currently have about ~3.5TiB of data (working on photo/video eats space...))

Poettering: Fitting Everything Together

Posted May 10, 2022 20:04 UTC (Tue) by mrugiero (guest, #153040) [Link] (4 responses)

On the same lines, I wonder if this scheme would allow for rolling-releases in a reasonable manner. As stated it looks more like something for slower paced distros.

Poettering: Fitting Everything Together

Posted May 11, 2022 13:33 UTC (Wed) by dutch_wolf (guest, #93834) [Link] (3 responses)

Although probably not in the exact manner envisioned I still think that rolling distros can take advantage of (some of) the tech, hermetic /usr/ for example would mean the package manage would only really need to care about /usr/ (and maybe /opt/) and would thus need less "magic" to deal with /etc/, also there is no reason the new "image" can't be build local (in way like we do with initrds right now). So going for a bit of spitballing right now a rolling release build on this infrastructure might look like this:

- full system updates (e.g. equivalent of `pacman -Syu`) install in the B partition (assuming currently booted in A) and become active on next reboot
- install of new applications (e.g. equivalent of `pacman -S [package name]`) would install in an overlay/sysext (potentially pulling them in into the "base" image on next update)

Optionally there might be a couple of different types of overlay/sysext in this scenario e.g.

- temp: lost on next reboot (will not be added to "base" on update, stored on tempfs)
- semi-temp: only lost on next full update (not lost on reboot, stored on disk)
- permanent: next full update will be added to base (stored on disk)

So yes though rolling distros won't (probably) have truly immutable "images" for /usr/ there are still ways they can take advantage of the tech developed here

Poettering: Fitting Everything Together

Posted May 11, 2022 13:49 UTC (Wed) by bluca (subscriber, #118303) [Link] (2 responses)

The reason for having the image built remotely by default is trust and signing . While having user-owned local keys is important for many user cases, for certain use cases and threat models it's important to avoid having the private key locally available.

Poettering: Fitting Everything Together

Posted May 12, 2022 14:03 UTC (Thu) by dutch_wolf (guest, #93834) [Link]

As I said I was just spitballing, and true there are definitely use cases where it makes sense that the image is build remote and truly immutable. but I think there are places/use cases where it makes sense to drop the immutable requirement and still make use of (some) the same infrastructure.

Poettering: Fitting Everything Together

Posted May 12, 2022 16:50 UTC (Thu) by mrugiero (guest, #153040) [Link]

There's also the part of testing it all together to avoid unexpected behavior. That would be lost IMO. I'm not entirely sure it's as valuable if you'd be installing a lot of packages in the traditional way.
Although it can be useful if worked in a layered way, e.g. splitting the core system and libs and putting those in the immutable /usr and higher level apps as packages. Kinda like suggested with the flatpak approach but avoiding flatpak.

Why remote attestation?

Posted May 10, 2022 15:47 UTC (Tue) by JesseW (subscriber, #41816) [Link] (16 responses)

I was surprised to see in the list of goals a mention of remote attestation (#4), and more surprised to see no further mention or explanation of why it was included. The only uses of remote attestation that I'm aware of undermine the ability of the directly-present user of a computer to determine how it processes data it receives (i.e. various DRM fantasies), and maintaining that ability seems pretty central to the point of having a FOSS system. Presumably I'm missing something?

Why remote attestation?

Posted May 10, 2022 16:20 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (12 responses)

There's various limitations that make remote attestation less useful in the general case (eg, nothing stops people from selling additional TPMs that you can plug in via USB, stick your own arbitrary measurements into and then perform attestation from there), but it's still useful in more specific cases. For example, it's viable to do this for a corporate fleet of machines to ensure that anyone accessing sensitive customer data is doing so from a machine that's running an up to date OS, but it's also practical to do this between a laptop and a phone to ensure that your system is in a good state before you type any passwords into it.

Why remote attestation?

Posted May 10, 2022 16:57 UTC (Tue) by mikebenden (guest, #74702) [Link] (11 responses)

So, then, it's about enabling corporate IT to enforce ever more draconian surveillance and lock-down policies on their hapless vict^h^h employees, who must, per policy, be treated like idiots, lest they actually get some useful work done...

That, and DRM fantasies.

To the grandparent: IOW: No, I don't think you're missing anything... :)

Why remote attestation?

Posted May 10, 2022 17:00 UTC (Tue) by mjg59 (subscriber, #23239) [Link] (2 responses)

In my case, it's actually about imposing /less/ monitoring software and restrictions on the endpoints, because I've got greater confidence that a machine that booted in a good state is going to continue being in a good state than I do if I have no insight into the state it booted into in the first place.

Why remote attestation?

Posted May 11, 2022 8:13 UTC (Wed) by taladar (subscriber, #68407) [Link] (1 responses)

I think you are confusing known state with good state.

Why remote attestation?

Posted May 11, 2022 8:27 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

It's true that a known state that I believe to be good is not necessarily a good state - a previously undisclosed vulnerability may allow that state to be modified into an untrustworthy state. But that's true of pretty much any form of monitoring I may perform, so my choices are either assume some level of risk or just not let people use computers at all, and since the latter is not an option I end up choosing the former. I can make no statements whatsoever about a machine that's in an unknown state, but I can assert that a machine in a known state is, at least, not in a set of known bad states.

Why remote attestation?

Posted May 10, 2022 17:45 UTC (Tue) by farnz (subscriber, #17727) [Link] (1 responses)

No - the core of remote attestation is to let someone outside the machine verify that the machine state matches what's expected. One use case for this is related to FIDO's CTAP, allowing an authenticator on a separate device (such as your phone, under your control) to only send authentication data to the laptop if it's in the state you expect.

That, in turn, allows you to have an authenticator device that will send your password to your laptop if and only if the laptop is in the state you expect; if your laptop has been tampered with, your device will refuse to send the password, and it's up to you to determine if the laptop is still trustworthy (e.g. you know you tampered with it deliberately), or if you want to treat it as compromised.

Similar applies in the IT situation - my employer ideally doesn't want me to get access to confidential internal data (such as GDPR-protected personal information) if I'm not on a device they can trust. One way to get that trust is to validate that the software I'm running is the expected OS, and that the OS is being careful with the certificate store such that only trusted clients can send a high-trust identity to their systems (and, of course, only release confidential data to high-trust identities). The other is to insist that I run a teetering stack of endpoint detection software that's intended to differentiate "malicious" activity from "legitimate" activity, but which has a whole bunch of false positives because, as a software engineer, I do things that most users don't (like run unsigned binaries produced by my local compiler).

Why remote attestation?

Posted May 10, 2022 19:23 UTC (Tue) by Wol (subscriber, #4433) [Link]

And this is the problem when too many people do not (want to) understand the difference between *owner* and *user*.

I am provided with a laptop by my employer for my work. Yes I curse sometimes beccause I can't install software I need, but it's not my place to just install anything and everything.

Now if it's *my* *own* laptop, restrictions on what I can do are (mostly) unacceptable. But even then, I want to know that some cracker hasn't got onto my machine and installed malware ...

Cheers,
Wol

Why remote attestation?

Posted May 10, 2022 22:19 UTC (Tue) by roc (subscriber, #30627) [Link] (5 responses)

For companies that handle data important to me, I sincerely hope that they are surveilling their employees and locking down systems to prevent unauthorized access/modification of that data.

Why remote attestation?

Posted May 11, 2022 5:29 UTC (Wed) by oldtomas (guest, #72579) [Link] (4 responses)

Me? I'd very much prefer they had employees they trust. Actually, when I choose companies to have business with, I try to make this a criterion, as far as my capabilities go (not very far, alas).

It's interesting how someone's wish can be someone else's dystopia.

Why remote attestation?

Posted May 11, 2022 5:47 UTC (Wed) by roc (subscriber, #30627) [Link] (3 responses)

"Hire employees you trust" doesn't scale. No-one can be trusted to never make a mistake and get hacked. Even if you set that aside, once you have hundreds of people there's no way you can trust all of them all the time.

Why remote attestation?

Posted May 11, 2022 6:52 UTC (Wed) by Wol (subscriber, #4433) [Link]

Desperate people do desperate things ...

How many times have frauds been carried out by reliable, trusted people who got themselves into trouble and couldn't see any other way out? These are the sort of people who (a) you would (and should) hire, and (b) wouldn't normally have been any trouble at all!

As the saying goes, "trust but verify". And having seen a colleague jailed for fraud, there's no way I would have expected them to be untrustworthy. With hindsight it was obvious, but things just changed slowly over time ...

Cheers,
Wol

Why remote attestation?

Posted May 11, 2022 8:16 UTC (Wed) by taladar (subscriber, #68407) [Link] (1 responses)

The problem is that you are just as likely (or possibly more likely) for the central IT department to have incompetent people who enforce running the same known vulnerable system for months.

Why remote attestation?

Posted May 11, 2022 8:29 UTC (Wed) by mjg59 (subscriber, #23239) [Link]

If you have incompetent central IT then allowing individuals to manage machines as they see fit is unlikely to result in significantly better outcomes.

Why remote attestation?

Posted May 10, 2022 16:55 UTC (Tue) by excors (subscriber, #95769) [Link]

> The only uses of remote attestation that I'm aware of undermine the ability of the directly-present user of a computer to determine how it processes data it receives (i.e. various DRM fantasies)

From a different perspective, it can be used to let you decide whether to send data to a remote computer based on how that computer is going to process the data. E.g. a company's VPN may allow remote employees to access sensitive information which is necessary for their job, but first it might want to check that the connection is coming from a company-provided computer (not from some attacker who's social-engineered the password and stolen the OTP token from an employee, which I suspect is not terribly hard), and that the computer hasn't been infected with a keylogger or whatever. Remote attestation is (at least in theory) a secure way to do that verification. (Obviously it can't protect against all possible attacks, but it can do better than 2FA alone at preventing unauthorised access.)

Apparently Windows can already do that - the VPN client can use Azure's Conditional Access (https://docs.microsoft.com/en-us/windows/security/identit...) which optionally uses the Health Attestation Service (https://docs.microsoft.com/en-us/windows/security/threat-...) which uses the TPM hardware to verify some configuration details.

There are also cases where there's no directly-present user, e.g. if you're using a cloud computing service you might want to verify that the cloud provider hasn't sneakily disabled the memory encryption that protects your application's data from the hypervisor, before you send sensitive information to your application.

Why remote attestation?

Posted May 11, 2022 12:16 UTC (Wed) by mezcalero (subscriber, #45103) [Link] (1 responses)

If your company runs a fleet of systems (nodes in a cloud cluster, cashier systems in a grocery chain, cell base stations in the wild, ticket selling machines for a transport authority) it is essential that the nodes authenticate to the central authority and vice versa. Exchanging credit card data for example, with a system that might be hacked is a bad idea, so you want to check first if its in a good state, and that's what remote attestation gives you.

On a classic desktop this is less relevant (though I think the interactive devices that are cashier systems or ticket selling systems are quite close to a desktop), but I am sure it still matters: i.e. if you are a big corporation and want to give out laptops to each employee it might make sense to verify they aren't hacked before you allow them to join your corporate network. Not every employee will constantly watch their laptop all their time, hence providing offline security and remote attestation is a great way to lock down the whole thing a bit, and make it harder to exploit it.

Lennart

Thanks, this makes sense!

Posted May 11, 2022 13:20 UTC (Wed) by JesseW (subscriber, #41816) [Link]

Thanks to everyone (particularly Lennart) for the replies and discussion. I now know more about the possible usecases for remote attestation, which is what I was looking for. mjg59's comments about bypass hardware dongles is also quite interesting!

Poettering: Fitting Everything Together

Posted May 10, 2022 15:49 UTC (Tue) by jd (guest, #26381) [Link]

Ok, so they're suggesting a default image is loaded onto the computer. I can buy that. It's not an astonishing way to do things. Installing extra stuff as packages is basically how Microsoft does things, and although I disagree with copying what other people do (the ideal is to learn from it and then to determine what would be an improvement), it would certainly simplify things. Microsoft's SNMP is installed as a package onto the pre-built default image, for example.

We can do better than that, much better than that - Microsoft's approach is now getting old and is cumbersome, it shouldn't be hard to improve on.

Mint comes in three flavours depending on the GUI you want, and Ye Olde Tyme installers used to ask if you wanted a development setup, desktop setup or workstation setup. It wouldn't be hard to have 9 standard images.

There's no reason why you couldn't do the underlying operations on those standard images at the package level, there's no reason why the DVD couldn't have a whole load of packages on it, the base install just freezes the partition in the state where certain packages are already installed.

Poettering: Fitting Everything Together

Posted May 11, 2022 5:13 UTC (Wed) by cytochrome (subscriber, #58718) [Link]

Very interesting ideas.

How difficult would it be to develop a proof of concept?

Encrypting home directories with per-user key

Posted May 12, 2022 0:19 UTC (Thu) by jhhaller (guest, #56103) [Link]

If one has tried to use NFS-v4 with Kerberos authentication and encryption, the place where encrypted home directories breaks down is with daemons, or user crontabs. It requires having a persistent secret (token) to refresh the Kerberos ticket, stored on the system, or accessible from the system using data in the system. systemd-cred (and systemd timers) are one obvious choice for daemons, but the daemon home directory will then have to be automatically decrypted when the system comes out of suspend.The same problem applies to encrypted local home filesystems. It's a catch-22, if one wants data to be accessible when not suspended without logging in, it can't really be protected from prying eyes in a hotel room. Perhaps these aren't as big of an issue on laptops as they are on servers, but still incomplete functionality.

Image-based vs. declarative

Posted May 13, 2022 20:32 UTC (Fri) by civodul (guest, #58311) [Link]

Much of the focus is on "images"--for the OS, for the home directory.

Images are a byproduct. I find the declarative approach more promising: declare the OS config, declare the home environment config, and use tools to instantiate those configs. Guix System and Guix Home follow that approach, and similarly for NixOS.

For OS deployments just like for software packages, users have more to gain by having the source than by having nothing but the byproduct.


Copyright © 2022, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds