LWN: Comments on "14 years of systemd"

Still don't handle remote boot correctly

farnz — Thu, 06 Mar 2025 10:22:08 +0000

Can you describe how? It's clearly not obvious to lee_duncan how to get at the post-pivot filesystem contents from a daemon started pre-pivot, and documentation on how to do that might clear up their confusion.

Still don't handle remote boot correctly

bluca — Wed, 05 Mar 2025 21:40:43 +0000

> Open-iscsi is unlike any of these implementations, in that it daemon needs to access post-pivot sysfs, a post-pivot database, and post-pivot configuration files. The pre-pivot daemon has their own copies of these things.

And? That's trivial to get

Still don't handle remote boot correctly

lee_duncan — Wed, 05 Mar 2025 19:43:10 +0000

Excellent grepping skills on github, but only one of those projects is actually a filesystem daemon! Seems funny that so many other projects (like a web server serving up git as a filesystem) use the "ROOT STORAGE DAEMONS" loophole to keep their daemons running.

Open-iscsi is unlike any of these implementations, in that it daemon needs to access post-pivot sysfs, a post-pivot database, and post-pivot configuration files. The pre-pivot daemon has their own copies of these things.

This kind of push-back is why, in my experience, some see systemd as less than cooperative. I just see it as an immovable object I have to go around.

Lack of understanding of fundamentals even after 14 years

dtardon — Wed, 05 Mar 2025 12:53:17 +0000

> systemd does not have or propose a model of service states as Poettering and many others confuse service states with process states. The reason why the "socket activation concept" was loved is that usually socket state matches service state better than process, as when a socket is ready usually the service behind it is ready or soon to be so.

So maybe you could enlighten those among us who don't know what service states are? And why they are better than the status quo?

> Since systemd aims to manage system states and the UNIX architecture does not have good facilities for connecting unrelated (by forking) processes

UNIX might not, but Linux does. It's called cgroups and systemd makes heavy use of it.

> and there is no model of service states systemd has become a huge and effectively monolithic (despite being technically split into 150 closely related executables) general service state multiplexer

IOW, because systemd doesn't have a model of service states, it's become a service state multiplexer... Sorry, but this sentence makes no sense. (And the bit about the split to separate executables being just technical is pure bullshit. It just shows that you've no idea what you're talking about.)

> where it must eventually manage all system aspects itself using a wild and complex variety of service state wrappers,
> so for example it must manage package installs,

It doesn't.

> network interfaces,

It doesn't either. networkd does, but that's completely unrelated to PID1 and there's no information sharing between them.

> filesystem mounts itself etc. to ensure that their state is well defined before starting services that depend on those.

I'm eager to learn how service states help to manage dependencies between services and mounts without tracking mounts.

Still don't handle remote boot correctly

bluca — Sat, 01 Mar 2025 12:25:54 +0000

> Yes, sadly, that's the same response I have gotten in the past. An architecture document that, to my knowledge, nobody has followed.

This is not true, and even a cursory search on GH shows plenty of real world examples:

https://github.com/search?q=%22argv%5B0%5D%5B0%5D+%3D+%27...

Still don't handle remote boot correctly

lee_duncan — Fri, 28 Feb 2025 21:56:00 +0000

Yes, sadly, that's the same response I have gotten in the past. An architecture document that, to my knowledge, nobody has followed. That document has a lot of issues that I'm pretty sure the systemd folks "don't understand".

I have a daemon that handles error connections, that needs to be running for the root disc to work correctly. And it does great at that. But it's somewhat complicated.

So to be able to use it the "systemd way", I would need to redesign it so that it can (1) run two at a time (not currently possible), and create a protocol for the initrd version to migrate its state to the post-pivot daemon, and not miss a beat if the root disc connection has an issue (or, even worse, have to daemons trying to fix the problem).

Or perhaps I'm supposed to keep my daemon from being killed? But then it's stuck in pre-pivot initrd root forever, and can't see any local filesystems. So calling it "solved for a decade" is off by about 10 years in my opinion.

Perhaps there's a working implementation of this "root storage daemon" policy? I'd love to see it if so, since none is reference in the link you supplied. Perhaps I can learn something.

"All major Linux distributions..."

rahulsundaram — Thu, 27 Feb 2025 23:18:19 +0000

> Not all of them. Slackware (the oldest continuously-maintained Linux distribution, which arguably fits the criteria for a "major" distro) does not use it, never has, and has no plan of including it in the foreseeable future.

As with many who started off with Linux early on, I too was a Slackware user at some point but I haven't seen it be considered a major distro in a really long time. It has a special place as one of the oldest Linux distros still actively maintained with a small but passionate Linux user base and over the years it has become a pretty niche distro. YMMV.

"All major Linux distributions..."

sombragris — Thu, 27 Feb 2025 22:36:23 +0000

All major Linux distributions use systemd

Not all of them. Slackware (the oldest continuously-maintained Linux distribution, which arguably fits the criteria for a "major" distro) does not use it, never has, and has no plan of including it in the foreseeable future.

The article was very informative and educational. Thanks for the reporting.

Still don't handle remote boot correctly

bluca — Wed, 26 Feb 2025 00:28:11 +0000

> We get connected in initrd, thus establishing the root disc. But then when it's time to switch root, the daemon is killed.

Then the daemon is not implemented correctly. This problem has been solved for a decade at least. See:

https://systemd.io/ROOT_STORAGE_DAEMONS/

Still don't handle remote boot correctly

lee_duncan — Tue, 25 Feb 2025 23:49:35 +0000

No. The problem is that we have a root connection, managed by the kernel, and a daemon that handles error conditions, retries, configuration, etc.

We get connected in initrd, thus establishing the root disc. But then when it's time to switch root, the daemon is killed. A new daemon is started in user space. In the time between the pre-pivot daemon stopping and the post-pivot daemon starting, the connection cannot handle any errors, such as network hickups, target hickups. This also means that the post-pivot daemon needs to rediscover everything about the existing connection.

I would like the ability to have the daemon continue during post-pivot. Evidently, this is possible, the daemon can't see the post-pivot filesystem, so it can't read config files, create or read database entries, etc. That means it can't really be interacted with. So that's not a solution.

BTW, the daemon ran uninterrupted, from boot through multi-user, pre-systemd, for historical reference.

I believe the systemd solution to this is to redesign our system so the daemon isn't needed during pivot, which also will not happen.

I'm not aware of any remote-boot solution that works well with systemd, but I'm not sure how nvme handles this.

Still don't handle remote boot correctly

bluca — Tue, 25 Feb 2025 22:36:25 +0000

That's not something that systemd can implement, it's up to whoever builds the initrd.

Still don't handle remote boot correctly

lee_duncan — Tue, 25 Feb 2025 22:12:27 +0000

I work on iSCSI, and remote iSCSI volumes used for booting have always been problematic on Linux.

Systemd had a chance to fix that, but they didn't. There's still no way to start an iSCSI session in initrd then pivot and have the full root understand that. But since it's not a preferred use case, it doesn't matter.

systemd's Image Obsession Misses the Forest for the Files

alexl — Mon, 24 Feb 2025 12:57:05 +0000

>No, that is not possible, not even theoretically, because the chain of trust MUST go up into the kernel for this to work at all. composefs mounts do not do this, by design, trust is verified in userspace. It's not a matter of implementing it, it needs to be redesigned, and mutability and integrity are essentially at opposite ends of the spectrum. Pick one or the other according to the use case, but you can't have both.

I don't see exactly how this would be impossible. I mean, obviously it would require some design and implementation work in the kernel to do the kernel-side signature validation, but nothing fundamentally impossible. The main blocker is that I don't currently consider this a critical feature.

systemd - no rhyme or reason

Klaasjan — Mon, 24 Feb 2025 06:58:14 +0000

Thanks, that is helpful.

systemd - no rhyme or reason

pabs — Sun, 23 Feb 2025 04:09:04 +0000

I believe these are the public freely licensed sources for the RedHat documentation. They do contain a lot of trademarks and other things that would not be correct on Debian though.

https://gitlab.com/redhat/centos-stream/docs/enterprise-d...

Second request

dmv — Sat, 22 Feb 2025 17:54:05 +0000

Seems like you picked the wrong day to stop sniffing glue. :)

Lazy linking -> Lazy loading

pabs — Sat, 22 Feb 2025 05:57:22 +0000

The Solaris libc had optional dynamic linking IIRC, that never got adopted by glibc though.

systemd's Image Obsession Misses the Forest for the Files

bluca — Fri, 21 Feb 2025 16:16:06 +0000

> Theoretically there is nothing that prohibits adding kernel support for IPE to e.g only allow executing binaries from a signed composefs mount, although you are right that this doesn’t currently exist.

No, that is not possible, not even theoretically, because the chain of trust MUST go up into the kernel for this to work at all. composefs mounts do not do this, by design, trust is verified in userspace. It's not a matter of implementing it, it needs to be redesigned, and mutability and integrity are essentially at opposite ends of the spectrum. Pick one or the other according to the use case, but you can't have both.

> However, I think IPE is only useful for setups that are extremely locked down.

Yes, for sure, code integrity is for cases where security requirements are stringent, and not for a generic laptop or so.

> For example, the second you have some kind of interpreter (like bash or python) available you can run essentially arbitrary code anyway.

This is being worked on, the brand new AT_EXECVE_CHECK option in 6.14 is exactly intended to allow interpreters to plug those use cases. It's absolutely true that interpreters are not covered right now, but (fingers crossed) there should be something usable for at least a couple of interesting interpreters this year if all goes according to plans.

systemd's Image Obsession Misses the Forest for the Files

alexl — Fri, 21 Feb 2025 15:17:41 +0000

Theoretically there is nothing that prohibits adding kernel support for IPE to e.g only allow executing binaries from a signed composefs mount, although you are right that this doesn’t currently exist.

However, I think IPE is only useful for setups that are extremely locked down. For example, the second you have some kind of interpreter (like bash or python) available you can run essentially arbitrary code anyway. For any kind of more generic system that can run code more dynamically it will not be applicable. For example, you could never use it on a personal laptop or a generic server.

systemd - no rhyme or reason

donald.buczek — Fri, 21 Feb 2025 07:38:50 +0000

Yes, you are right. The documentation is good for our needs, but not necessarily for others. Since we run our own in-house distribution, which is not only very different from other distributions but also from the typical target system that the systemd developers primarily have in mind, we have to integrate everything ourselves anyway and need to understand the details to do so. We don't need high-level user guides. Before making changes that affect systemd configuration, I often re-read the relevant systemd man pages from front to back. My positive experience is that the documentation for systemd is almost always sufficient. My experience with many other products is that you end up analyzing the sources after a very short time because the documentation is unclear, incomplete, contradictory or wrong.

Occasionally reading Lennard's “PID Eins” blog is more of a leisure activity, but it sometimes gives you ideas about whether one or the other idea or a new systemd feature could be useful for us as well.

systemd - no rhyme or reason

Cyberax — Fri, 21 Feb 2025 07:08:41 +0000

Most of the Lennart's blog posts linked on systemd.io predate journalctl. So they miss a lot of functionality that is useful for people (e.g. checking logs only from the current invocation of the service).

FWIW, I have a personal wrapper around systemd (`scl`). It started as an alias to `systemctl` because I didn't want to keep breaking my fingers typing it all the time. Then I added some obviously missing functionality like starting a service and viewing its logs in one command (like `docker compose up` does, for example). I think a lot of hostility to systemd could have been avoided, if something like this existed.

systemd - no rhyme or reason

zdzichu — Fri, 21 Feb 2025 06:56:11 +0000

Like https://0pointer.de/blog/projects/journalctl.html ?

systemd - no rhyme or reason

Cyberax — Fri, 21 Feb 2025 00:23:50 +0000

Yes, it's pretty old, but I love it as an example. I had its A0-sized printout on a wall at one point around 15 years ago. And it has been updated a bunch of times, of course. The updated version that I linked is still pretty accurate.

And speaking of old, systemd.io links to the series of Lennart's blog posts from 2010-2012. They're also still mostly accurate, fwiw, but do miss stuff like practical uses of journalctl.

systemd - no rhyme or reason

bluca — Thu, 20 Feb 2025 23:54:39 +0000

"This article is based on the 2.6.20 kernel" and that's THE BEST example you could find of what in your head constitutes good documentation? Nearly 20 years out of date? lol, lmao

systemd - no rhyme or reason

Klaasjan — Thu, 20 Feb 2025 23:36:05 +0000

Understood, thanks.
It would be nice if RedHat’s documentation was available under a free enough license so that it could be hosted on Debian.org as well.
Cheers

systemd - no rhyme or reason

Cyberax — Thu, 20 Feb 2025 23:33:20 +0000

Sorry. systemd.io is not any less messy.

To give an example of a good doc: https://upload.wikimedia.org/wikipedia/commons/3/37/Netfi... from https://wiki.linuxfoundation.org/networking/kernel_flow A very clear but detailed overview.

I actually looked, and I can't find anything similar for systemd. I've been following its development for years, but I can't outright tell how symlinks, overrides, and other machinery work together.

systemd - no rhyme or reason

bluca — Thu, 20 Feb 2025 23:18:34 +0000

systemd works the same everywhere - it's one of its main selling points vs what came before. 99.99% of what you find on RH's documentation will apply to Debian or any other distribution running systemd too.

systemd - no rhyme or reason

Klaasjan — Thu, 20 Feb 2025 23:14:50 +0000

Aha!
Now, what if I run systemd on a Debian system?
(Honest question, since indeed I do)

systemd's Image Obsession Misses the Forest for the Files

bluca — Thu, 20 Feb 2025 16:03:23 +0000

> This is completely wrong. Composefs checks at initrd that the fs-verity digest of the composefs image is the expected value, similar to how you would verify the root dm-verity digest of the rootfs block device.

No, that is completely wrong. With signed dm-verity + IPE the enforcement is done by the kernel on every single binary or library being executed, at any time, at runtime, forever. That's the point of code integrity.

You cannot do that with composefs, because the security model is different and fully trusts userspace. So a userspace process that has escalated privileges can simply replace the composefs mounts with anything of their choosing, and run whatever they want. A userspace attacker that has escalated privileges on a dm-verity system cannot do that, as they would need a signed volume and they do not have access to a private key trusted by the kernel keyring, so taking control of userspace is not enough, you also need to take control of the kernel, which is much harder.

systemd's Image Obsession Misses the Forest for the Files

alexl — Thu, 20 Feb 2025 15:37:48 +0000

>> I don't see a difference between having the dm-verity signature checked by the kernel, and having the composefs erofs file checked from user space in the initramfs (assuming the FS is LUKS protected), it's just different pieces of code in a UKI no ?

>The difference is huge: in the former case integrity is enforced at _all times_ by the kernel, at runtime, so it's resilient against online attacks. In the latter case, integrity is only checked _once_ during boot, and never again. Again, the use cases are different: in the first case security checks are done always, in the latter case they are there for offline protection of data at rest. The second model is strictly weaker. Which might be fine, mind you - as always it depends on the threat model.
But if your threat model does include online attacks by malicious privileged processes (and for our case in Azure it very much does), the first model can mitigate it, the second one cannot.

This is completely wrong. Composefs checks at initrd that the fs-verity digest of the composefs image is the expected value, similar to how you would verify the root dm-verity digest of the rootfs block device.

After that, each file access is validated by fs-verity at runtime. This includes reads of the EROFS image that has the metadata, as well as the backing files. And the expected fs-verity digests for the backing files are recorded in the EROFS image, and validated each time a backing file is opened.

The only difference between dm-verity and composefs is that dm-verity validates at the block level, and composefs verifies at the file level. This means that an attacker could modify the filesystem at the block level to "attack" the kernel filesystem parser. However, this difference is very small in practice. Only in a system using dm-verity for a read-only root, and *no* other filesystems does it make a difference. The moment you mount a filesystem for e.g. /var, then an attacker could just as well attack that. Any "solution" to that, such as dm-crypt would also apply to the composefs usecase.

Lack of understanding of fundamentals even after 14 years

walex — Thu, 20 Feb 2025 11:39:10 +0000

«leave the personal insults»

I will strive to never call again Poettering a "a very intelligent person" again on LWN. That is the only comment I made as to his person.

systemd's Image Obsession Misses the Forest for the Files

Rigrig — Thu, 20 Feb 2025 10:40:19 +0000

If this is imposed on users (e.g. their phones), then we have a problem

I run shell (and other) scripts on my phone using Termux, and W^X is indeed a problem: (apparently also for BOINC)

"Recent" Android API versions enforce W^X, so Termux is either
- F-Droid: built against an older version. This generates warnings, is not accepted into the Play Store, and might eventually stop working if/when Android breaks BC for that version.
- Play Store: using trickery to override exec(). This has various drawbacks though.
Downloading+running executable code seems to be against Play Store policies, so it might get randomly removed at any time anyway.

Rust and dynamic linking

farnz — Thu, 20 Feb 2025 10:23:27 +0000

There's three ways to deal with dynamic linking in Rust:

As an implementation detail of a coherent whole. In this case, you don't care that libmyapp.so has an unstable ABI, because it's built with the same compiler as myapp-bin1 and myapp-bin2, so an unstable ABI is A-OK with you.
Put a psABI shim around the Rust functions, and have a hand-maintained ABI, the way glibc does things in C land. This restricts the ABI you can use to the psABI (no generics, for example), but means that you have complete control and can make your ABI stable.
Help out with the experimental crABI work, which aims to produce an external ABI that's more capable than current psABIs, then put a crABI shim around the Rust functions, and maintain a stable ABI that way.

Which one is right for you is a decision that depends on the project goals.

NFS and systemd automount units

farnz — Thu, 20 Feb 2025 10:20:05 +0000

Out of interest, why can't you use an automount unit for your NFS mount? This has the properties you would want for a network FS, of remounting automatically if the NFS server is missing, and of not blocking anything until the first access to a file on the mount point.

DLL hell, version 2

bluca — Thu, 20 Feb 2025 09:58:25 +0000

Sure but that is all programmatically discoverable. What is needed is encoded both at the ELF level, with the dlopen note, and at the package level, with the recommends/suggests or equivalent for RPM. So the bug reporting tool(s) should query for such information when assembling the report.
You can't have optional dependencies if they are not optional...

Images are required for verified boot

bluca — Thu, 20 Feb 2025 09:57:05 +0000

...yes? That's the point of a userspace reboot?

systemd - no rhyme or reason

bluca — Thu, 20 Feb 2025 09:56:21 +0000

That sort of documentation is provided in systemd.io rather than in manpages, and for even higher level stuff there are excellent admin/user guides provided by RedHat

systemd - no rhyme or reason

Wol — Thu, 20 Feb 2025 09:34:00 +0000

The problem is (a) what does the reader want the docs for, and (b) why did the writer write the docs.

Most developer documentation is great for reminding you of what you already know. It is absolutely hopeless at teaching you how to use the system. That's why beginners make the same mistakes over and over again - from their point of view Double Dutch would probably make more sense.

I've had exactly that with systemd - I don't know where to start looking, so I can't find anything, and what I do find I can't make sense of. That is where so much stuff (not just computing) is severely lacking nowadays. We've just bought a new modern tv, which thinks that the main purpose of the user manual is to tell you where to plug the leads in. Of course, we don't have a clue how to use it, everything is trial and error, and it's so damn complicated that we can't find anything! We just want a tv we can turn on and watch!!!

And of course, most documentation FOR beginners is written BY beginners, so it's full of beginner grade errors :-(

Cheers,
Wol

systemd - no rhyme or reason

mathstuf — Thu, 20 Feb 2025 09:32:53 +0000

I feel they are similar to the CMake docs: detailed in the what. Not all that clear on the how or why. I *can* piece together how to set up a "reload service X on a timer or when $path changes", but an example would go much farther.

DLL hell, version 2

mathstuf — Thu, 20 Feb 2025 09:31:08 +0000

There is one downside to the behavior changing depending on whether a library happens to be found or not: debugging problems that only occur in the presence (or absence) of the library without indication of the runtime state is a royal pain. "Why is machine X compressing and Y not? They both have the required libraries…oh wait they were installed this boot on Y and missing when it started up." kind of stuff.