Do not use non-core systemd [LWN.net]

Do not use non-core systemd

Posted Sep 23, 2025 23:55 UTC (Tue) by WolfWings (subscriber, #56790) [Link] (25 responses)

The networking is one of the best parts of systemd IMHO though? (Along with their bootloader for UEFI systems to replace grub.)

It's usually the first thing I do when setting up a new machine is switching to networkd and switching whatever wireless SSID management it wanted to use to IWD instead.

DNS yeah, it's waaay too complicated and makes DNS feel like I'm setting up sendmail again for some reason, but the networking being wildcard-template based makes is super simple for the vast majority of configurations, even bonded links where you're bonding up an entire PCIe card or the same slots across multiple separate PCIe cards.

Do not use non-core systemd

Posted Sep 24, 2025 12:50 UTC (Wed) by linuxrocks123 (subscriber, #34648) [Link] (8 responses)

Why would someone need their bootloader? You can compile the Linux kernel to a UEFI application and boot it directly. Also, ELILO still works.

Do not use non-core systemd

Posted Sep 24, 2025 17:26 UTC (Wed) by jem (subscriber, #24231) [Link] (1 responses)

Sytemd-boot is not a bootloader, it is a boot manager which displays a menu with which you can choose between different configuration entries. A configuration entry specifies which kernel to boot and its associated kernel parameters. Incidentally, the kernel must be compiled to a UEFI application.

Do not use non-core systemd

Posted Sep 26, 2025 3:44 UTC (Fri) by ATLief (subscriber, #166135) [Link]

While Systemd-boot is not *entirely* a boot-loader, it does contain a UEFI stub in addition to the UEFI boot menu. UKIs are basically that stub with the kernel and initramfs appended to it, and they can function as standalone UEFI executables without the boot menu.

Do not use non-core systemd

Posted Sep 25, 2025 4:42 UTC (Thu) by cclifforgg (subscriber, #88919) [Link]

I have a few Dell systems where the firmware cannot correctly pass the kernel a command line, but systemd-boot arranges for it to work correctly somehow. Alternatively I believe that you can put the commandline into the EFI stub kernel, but then you cannot edit it.

(E)LILO was discontinued a decade ago and removed from Debian.

Posted Sep 25, 2025 9:12 UTC (Thu) by WolfWings (subscriber, #56790) [Link] (3 responses)

LILO and ELILO haven't been updated or supported in over a decade at this point, and LILO wasn't included since Debian Bullseye with ELILO removed back in 2014 before Jessie was released. Your counter-option really isn't.

If you enjoy manually compiling everything there's always Gentoo, but if you're using Debian you want to just install binary packages and get on with your day.

The Linux kernel most distros publish is already a UEFI binary since a LOT of bootloaders want that for signing reasons, etc, and updating it is a lot easier with fallbacks using a bootloader with a menu interface and relying on distro signing keys than having to do all that yourself.

(E)LILO was discontinued a decade ago and removed from Debian.

Posted Sep 26, 2025 2:33 UTC (Fri) by linuxrocks123 (subscriber, #34648) [Link] (2 responses)

I didn't say Debian packaged ELILO. This sub-topic is about SystemD components in general, not about Debian in particular. I said ELILO still works, because I made a dual-boot system with Windows about three weeks ago, let the Slackware-Current installer install ELILO, and it worked.

I also didn't say ELILO was updated or supported. But, since you mentioned it, I'll just say that I generally don't whether software is updated or supported, and I generally do care if it works.

Regarding boot menus, I've found the UEFI boot menu available with F12 to be sufficient, so I don't see a need for a separate program to provide that feature. Given all of the over-engineered nonsense built into UEFI, I'm pleased that the resulting mega-kludge can at least display a better boot menu than its predecessor.

But I'm still looking forward to CSMWrap being developed to completion.

(E)LILO was discontinued a decade ago and removed from Debian.

Posted Sep 26, 2025 2:55 UTC (Fri) by intelfx (subscriber, #130118) [Link] (1 responses)

> I didn't say Debian packaged ELILO. This sub-topic is about SystemD components in general, not about Debian in particular.

This comments section as a whole is about "Debian in particular", though. The commenter you were replying to is also, presumably, a user of Debian in particular.

So when you post a comment on an article (about Debian) replying to another commenter (a user of Debian) saying "Why would someone need [systemd's] bootloader? <...> ELILO still works" and then claim that it does not matter that ELILO is not, in fact, available in Deban, then you are simply being obtuse. HTHs.

(E)LILO was discontinued a decade ago and removed from Debian.

Posted Sep 26, 2025 7:47 UTC (Fri) by linuxrocks123 (subscriber, #34648) [Link]

Ah, didn't realize WolfWings was both the replier and topic originator. His original comment didn't specify Debian, but I guess he is personally mostly interested in SystemD non-core components on Debian.

In that case, I expect compiling ELILO on Debian would be straightforward, and I also expect the old .deb file from whatever the last release to include ELILO was would probably still work. However, I don't use Debian and so have not tested that.

Do not use non-core systemd

Posted Sep 26, 2025 4:01 UTC (Fri) by ATLief (subscriber, #166135) [Link]

Systemd-boot lets you easily create UKIs, and the Linux kernel alone has a bunch of annoying limitations.

Do not use non-core systemd

Posted Sep 24, 2025 18:22 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

I like the _idea_ of networkd. Heck, I like the ideas behind most of the systemd modules.

It's the implementation that is problematic. There are weird little "features" like missing routing metrics. So if you have a 10Gb wired network and a 2.4GHz wireless connection, networkd will assign both of them the same metric. With no way to fix it, because it's not exposed in the config.

I also had problems with DHCP and VPN interactions where systemd's "interface online" probes misfired.

Network Manager at least gives preference to wired connections.

Do not use non-core systemd

Posted Sep 24, 2025 19:45 UTC (Wed) by TomH (subscriber, #56149) [Link]

You know you can use RouteMetric in an Address section and Metric in a Route section to set metrics right... There are similar things for routes received via IPv6 RA and so on.

About the only route you can't set a metric for I think is ones added by an Address or Gateway directive in a Network section but just use an Address or Route section instead if you need to customise the metric.

Do not use non-core systemd

Posted Sep 25, 2025 8:53 UTC (Thu) by WolfWings (subscriber, #56790) [Link] (1 responses)

You're trying to fix what is really a failover issue with routing metrics if the ethernet and wifi share overall connectivity on the back-end. That requires a bond, and it's a standard thing to do on laptops.

Setup an active-backup bond between the two links with PrimaryReselectPolicy=always and PrimarySlave=true on the ethernet device.

Do not use non-core systemd

Posted Sep 25, 2025 19:57 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

No, it's not. My Ethernet network has a larger MTU (8000) than WiFi, so bonds don't work properly.

Do not use non-core systemd

Posted Sep 25, 2025 10:10 UTC (Thu) by intelfx (subscriber, #130118) [Link] (1 responses)

> There are weird little "features" like missing routing metrics.

Huh? What am I missing?

https://www.freedesktop.org/software/systemd/man/latest/s...
https://www.freedesktop.org/software/systemd/man/latest/s...

Do not use non-core systemd

Posted Sep 25, 2025 19:55 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

This:

> Added in version 246.

I guess I can try it again...

Do not use non-core systemd

Posted Sep 25, 2025 2:55 UTC (Thu) by sionescu (subscriber, #59410) [Link] (9 responses)

systemd-resolved doesn't work beyond simple cases: split DNS doesn't work well, mDNS is broken, dns-over-https is not implemented, etc...

Do not use non-core systemd

Posted Sep 25, 2025 5:59 UTC (Thu) by intelfx (subscriber, #130118) [Link] (8 responses)

> split DNS doesn't work well

Split DNS is basically the entire reason systemd-resolved exists. And it does in fact work very well — in my experience, this is the *first* time split DNS on Linux actually worked well, and I use it daily.

> mDNS is broken

Ditto, excepting the "first time" part (but I still use it daily).

> dns-over-https is not implemented

DNS-over-TLS is.

Do not use non-core systemd

Posted Sep 25, 2025 14:09 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (7 responses)

> Split DNS is basically the entire reason systemd-resolved exists. And it does in fact work very well — in my experience, this is the *first* time split DNS on Linux actually worked well, and I use it daily.

I have to reconfigure resolved (via `/etc/nsswitch.conf`) because it causes network namespace leakage. If I have a network namespace jailed to a VPN, systemd-resolved needs to live *in* that namespace to reliably do DNS for it (but then it cannot service non-VPN requests…). But because it leaks through `nss` plugins, I have to at least remove the `[!UNAVAIL=return]` behavior. So instead, I remove that and put it behind `dns` to avoid the latency involved when it fails to service VPN-destined requests.

Is there some configuration magic around to handle this that you know of?

Do not use non-core systemd

Posted Sep 25, 2025 23:53 UTC (Thu) by ebiederm (subscriber, #35028) [Link] (3 responses)

pair bind mounts of the config file with the network namespace?

ip netns allows for that. I don't know about systemd-networkd

Do not use non-core systemd

Posted Sep 26, 2025 3:15 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (2 responses)

Yes, `ip netns exec` is involved in using it. It knows how to deal with `resolv.conf` and mounts that in a mount namespace for the process, but not `nsswitch.conf` AFAIK.

Which reminds me that I need to also fix `resolv.conf` so that things are happy too. If it is a symlink, the `ip netns exec` mount namespace is broken when the target `resolv.conf` is rewritten with a "write then rename into place"; `cat newresolv.conf > /etc/resolv.conf` works though.

Do not use non-core systemd

Posted Sep 28, 2025 1:49 UTC (Sun) by ebiederm (subscriber, #35028) [Link] (1 responses)

It should be just a matter of putting a new nsswitch.conf in the right place.

If my memory serves the code just walks through the configuration directory and bind mounts every thing in it, into /etc.

Do not use non-core systemd

Posted Sep 28, 2025 2:55 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

Indeed it does. Thanks; I can at least use the non-symlink version then (as the symlink (target?) rewriting breaks the bind mount of the configuration file).

Do not use non-core systemd

Posted Sep 26, 2025 0:19 UTC (Fri) by intelfx (subscriber, #130118) [Link] (2 responses)

> I have to reconfigure resolved (via `/etc/nsswitch.conf`) because it causes network namespace leakage. If I have a network namespace jailed to a VPN <...>
>
> Is there some configuration magic around to handle this that you know of?

I don't think systemd-resolved is network namespace aware, and I'm not familiar with any tricks in that area. But that's a hugely niche use-case, and not what I was defending. Normal "split DNS" is when you have multiple network interfaces with disjoint sets of routes in a single network namespace.

Do not use non-core systemd

Posted Sep 26, 2025 3:17 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

Ah. I think of network namespacing tricks as a way to get split DNS without actually allowing a process to route over both at once. But perhaps I'm just doing something "more" than "just" split DNS then. In any case, I've found a block tower setup that works to my satisfaction; I'm more looking for ways to put more mortar around the base rather than "don't look at it sideways too hard" stability.

Do not use non-core systemd

Posted Sep 26, 2025 8:38 UTC (Fri) by nim-nim (subscriber, #34454) [Link]

The usual use case for split dns is a backend network with private resources, and a hardened front end network for remote access. This kind of setup to work pretty much requires dual routing (and one could route remote connexions via the backend link but that would defeat the purpose of isolating untrusted traffic on a heavily audited frontend network).

Do not use non-core systemd

Posted Sep 24, 2025 9:44 UTC (Wed) by nim-nim (subscriber, #34454) [Link] (8 responses)

The unfortunate situation is that networkd has the best configuration grammar/syntax of the lot, with clear well-documented configuration files, but networkmanager is the most complete and supported system (as for the rest their only benefit is to push the need to learn something new in the future).

So, choose your poison, good easy to maintain config files, or solid networking system with less fortunate config files

Good easy to maintain config files are really attractive in a headless server environment, people want to provision reference config files via ansible & co instead of toiling at the cli like in the bad old Cisco-centric 2000’s networking days. NetworkManager’s emphasis on nmcli is really unfortunate, a NetworkManager with a networkd config backend would make a killing.

Do not use non-core systemd

Posted Sep 24, 2025 10:09 UTC (Wed) by paulj (subscriber, #341) [Link] (6 responses)

NetworkManager supports, or used to support, the RedHat ifcfg ini-style-ish config files.

https://www.networkmanager.dev/docs/api/latest/nm-setting...

Do not use non-core systemd

Posted Sep 24, 2025 10:56 UTC (Wed) by nim-nim (subscriber, #34454) [Link] (5 responses)

All those legacy config systems are completely outmatched by networkd. Whatever faults networkd maintainers have their config system blows the alternatives out of the water.

Do not use non-core systemd

Posted Sep 24, 2025 13:20 UTC (Wed) by joib (subscriber, #8541) [Link] (4 responses)

NetworkManager supports .ini style configuration files under /{etc,run}/NetworkManager/system-connections that you can push with ansible or other configuration management tools.

The support for old-school Redhat ifcfg- files has AFAIU been removed.

Do not use non-core systemd

Posted Sep 24, 2025 13:22 UTC (Wed) by joib (subscriber, #8541) [Link] (3 responses)

And indeed, the link the parent provided even contains a link to the manual for the newer style configuration files at https://www.networkmanager.dev/docs/api/latest/nm-setting...

Per that document, the keyfiles should support the full featureset of NetworkManager.

Do not use non-core systemd

Posted Sep 24, 2025 14:11 UTC (Wed) by nim-nim (subscriber, #34454) [Link] (2 responses)

The new format is a bit better but still not up to par of what you get with networkd.

Instead of all uppercase ifcfg variables you get all lowercase variables (with is arguably nicer but short of the tasteful capitalization networkd-side that improves legibility).

The keyfiles are structured but the doc is still unstructured registry-style.

Some properties use - as separator, others _

Networkmanager still writes optional properties with default values instead of limiting itself to things where a specific value has been set (the meaningful part of the configuration).

It very unfortunately names all wired connexions “Wired connexion X” with X some non deterministic value (à la Windows) instead of using something reliable such as the interface name (when in fact each “Wired connexion X” is locked to a specific interface). Even worse the naming is not in fact “Wired connexion X” but a localized string that will wreak havoc with automation (reminds me of all the awful XDG variables that most everyone ignores to hardcode simple consistent .config/.cache/etc everywhere)

And so on, I won’t even get into the advanced wildcard tricks networkd configuration inherits from systemd

Nothing is completely awful but it falls very short of the syntaxic care and maturity exhibited networkd-side.

Do not use non-core systemd

Posted Sep 24, 2025 16:25 UTC (Wed) by anselm (subscriber, #2796) [Link] (1 responses)

Nothing is completely awful but it falls very short of the syntaxic care and maturity exhibited networkd-side.

This, plus configuration files work the same across all of systemd. That level of consistency alone is worth the price of admission, compared to the traditional setup where every single component had its own unique configuration file format.

Do not use non-core systemd

Posted Sep 25, 2025 11:12 UTC (Thu) by kpfleming (subscriber, #23250) [Link]

Yes, the systemd-networkd configuration system is fantastic, and as you say the fact that it uses the same configuration mechanisms as the rest of the systemd means you get drop-in support for free, and in my mind that's worth the price of admission. The ability for a tool like Ansible to manage an underlying network interface with VLANs on top of it in separate (but additive) configuration files is really powerful.

Do not use non-core systemd

Posted Sep 26, 2025 16:04 UTC (Fri) by marcH (subscriber, #57642) [Link]

> So, choose your poison, good easy to maintain config files, or solid networking system with less fortunate config files

For software I choose the one which is "as simple as possible, but not simpler".

That means systemd-networkd in simple, static network configurations and NetworkManager on laptops with wifi, VPNs and what not.

Do not use non-core systemd

Posted Sep 25, 2025 22:15 UTC (Thu) by intgr (subscriber, #39733) [Link] (7 responses)

> The good parts include [...] home directory management.

I'm not sure, that part might be abandonware. You would think that if it was maintained, not corrupting user data world be high on the list of priorities.

A year ago I had an incident where during a regular system shutdown, the systemd-homed resize operation took too long and got killed due to timeout. Result was a corrupted partition table inside the user home image and I could no longer access my data.

OK, bugs happen, but what baffled me was that to this day, no systemd developers have acknowledged or reacted the bug report, despite reports from other users also being affected.

Thankfully after a few hours of struggling, I managed to mount the image and could recover my data. But to this day, I still hold my breath every time I restart my system that this bug doesn't hit again.

Do not use non-core systemd

Posted Sep 25, 2025 23:39 UTC (Thu) by smurf (subscriber, #17840) [Link] (6 responses)

I'd assume that the partition table of any image (or hard disk) is essentially read-only, in the sense that nothing writes to these areas, and thus cannot be corrupted by something that merely hangs, no matter what you do to the process controlling it.

Thus, whichever freaky bug you encountered is not reproducible. So what should the maintainers actually do, other than say "yeah it's a strange case of data corruption but if we don't see a pattern, much less a reproducer …" and leave it open?

Do not use non-core systemd

Posted Sep 25, 2025 23:57 UTC (Thu) by koverstreet (✭ supporter ✭, #4296) [Link]

Bugs without reproducers are a normal fact of life, you can't just give up any time you see one.

To start, you look at all the information you have available; in this case that'd mean looking at the corrupted partition table - hexdump if necessary - to see if you can spot any patterns or deduce anything about what happened.

Sometimes that can tell you all you need to know; I once saw a guy deduce just from a hexdump of some memory corruption that it was an errant memmove that shot past the address it was supposed to stop at - he spotted that everything had been shifted by 8 bytes - and then from that, found the exact line of code that caused it (an open coded memmove).

If that fails to identify the bug, you take what you do know and look for places where your code is weak and could be hardened; you look for ways to make entire classes of bugs impossible. This is on-disk data structures we're talking about; we don't have practical techniques for making entire codebases in systems languages bug free, but we absolutely can identify ways to limit damage and make sure that on disk data structures aren't corrupted.

Do not use non-core systemd

Posted Sep 26, 2025 0:15 UTC (Fri) by intgr (subscriber, #39733) [Link] (4 responses)

> I'd assume that the partition table of any image (or hard disk) is essentially read-only

Incorrect assumption. Systemd-homed regularly resizes images/partitions to rebalance the disk space available between multiple users.

Particularly, btrfs can only be shrunk when unmounted, but obviously needs to be done when encryption keys are still available, so it frequently happens during system shut down.

> something that merely hangs,

It's not a "hang". The resizing can take a few minutes even on fast SSD if lots of data needs to be relocated.

Do not use non-core systemd

Posted Sep 26, 2025 0:44 UTC (Fri) by intelfx (subscriber, #130118) [Link] (1 responses)

> Particularly, btrfs can only be shrunk when unmounted

Without commenting on the rest of the analysis, that's not true.

Do not use non-core systemd

Posted Sep 26, 2025 7:08 UTC (Fri) by intgr (subscriber, #39733) [Link]

Oh, I must be misremembering. But regardless, that's when homed frequently does shrinking.

Do not use non-core systemd

Posted Sep 27, 2025 10:55 UTC (Sat) by smurf (subscriber, #17840) [Link] (1 responses)

> Incorrect assumption. Systemd-homed regularly resizes images/partitions to rebalance the disk space available between multiple users.

Ah. Thanks for the correction.

One might assume that it was an SSD that was powered down during writing. In this case all bets are off. Don't Do That.

Thus there needs to be an "work in progress, do NOT shut down" dialog/message/whatever. (Assuming that systemd doesn't time out the resize.)

Do not use non-core systemd

Posted Sep 28, 2025 1:18 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

That sounds like a job for `systemd-inhibit(1)`. See https://systemd.io/INHIBITOR_LOCKS/

Do not use non-core systemd

Posted Sep 26, 2025 17:17 UTC (Fri) by kreijack (guest, #43513) [Link] (5 responses)

> The trick with systemd is to know which parts of it are good and skip the bad parts. It's really an umbrella project with lots of different modules that (try to) work with each other.

I agree. Is a too big project with a too wide scope. Am not saying that it there are "bad" parts; the problem is that all the parts are coupled, and this is source of scalability problem:
- if you have an issue with a component solved in a newer version you need to update all the components (even the one that works flawless)
- all the components can't rely on a defined interface but can access to any interface that they want. This prevent to replace a component with a more specialized component. For example I suspect that the binary format of the journal is inefficient, but it is complex to replace only this part because journald is integrated in systemd.
- I can understand that all the parts that can manage "events" should be integrated in systemd. But why (e.g.) sd-boot and homed should be integrated in the main project ?

I think that dividing systemd in different sub-projects separated by a clean/clear interface, would speed up the development, and open the cooperation with different projects that may provide more specialized tool for vertical case.

Do not use non-core systemd

Posted Sep 26, 2025 19:33 UTC (Fri) by NAR (subscriber, #1313) [Link] (3 responses)

I think that dividing systemd in different sub-projects separated by a clean/clear interface, would speed up the development

Not necessarily. Currently if they want to change such an interface, they can do it "in house", update both (all) sides at the same time (maybe even in the same commit? - I'm not familiar with the systemd codebase). If there are (more or less) independent (sub)projects, then it will take inter-project coordination which is always slower. And possibly keeping around technical debt for backwards compatibility.

Do not use non-core systemd

Posted Sep 28, 2025 23:35 UTC (Sun) by Wol (subscriber, #4433) [Link] (1 responses)

> Not necessarily. Currently if they want to change such an interface, they can do it "in house", update both (all) sides at the same time (maybe even in the same commit?

But is this "as simple as possible, but no simpler"? If it isn't then you have a problem full stop.

If you have to do it "in house", you need to ask the question "why". And if you don't have an answer, you need to fix it.

Cheers,
Wol

Do not use non-core systemd

Posted Sep 29, 2025 1:41 UTC (Mon) by anselm (subscriber, #2796) [Link]

If you have to do it "in house", you need to ask the question "why".

Presumably because it's easier and more convenient to do it that way.

Having said that, the systemd developers do document a number of their internal interfaces, so it should be possible, at least in principle, to come up with alternative implementations of some of its components (but again, we would want to ask the question “why”, especially when improving systemd itself is also an option). Some people may argue that the specific interfaces that they are interested in are not (yet?) documented, but the reasons for that are probably best discussed with the systemd developers themselves.

Do not use non-core systemd

Posted Oct 2, 2025 16:57 UTC (Thu) by kreijack (guest, #43513) [Link]

Changing an interface is a good example: changing an interface is a very expensive activity. Far more expensive than any other change. But in a "big" project where there the interface are not very well defined, everything may be an interface. Thus every time any part of the code is touched, the developer must check that this will not impact the rest of the project.

Having small module simplify (==less time) the code comprehension and the check of the change impact. Because the code to inspect is smaller.

If you need to issue an urgent change (a bug, a security fix), it is enough to audit only the involved portion of the code.

NB: despite what I wrote above, I think that systemd code has a very quality with an extensive use of modern technique to simplify the development (e.g. the __cleanup__ macro).

Do not use non-core systemd

Posted Sep 27, 2025 18:11 UTC (Sat) by mbunkus (subscriber, #87248) [Link]

> I think that dividing systemd in different sub-projects separated by a clean/clear interface, would speed up the development

I disagree. Splitting in such a way would require more effort (for coordinating releases, for implementing/testing/documenting/maintaining the interfaces you're talking about etc). If the same developers are interested in working on the various split components that're currently working on the umbrella project, the speed of development of new features will go down.

Your statement assumes that after splitting there will be a net inflow of developers providing more work, just to offset the aforementioned new overhead, never mind speed up development. "Net inflow" because there might also be developers who prefer to work on a big umbrella as it is now & will reduce their work as a result.

I would really be interested in actual studies of development speeds in more monolithic vs split-up projects (and where the difference is due to project size and not due to factors such as the monolithic project having too many a**holes on its team), though I doubt it due to how incredibly difficult it would be to have comparable data. Is there even anecdotal evidence from past/ongoing situations? Really curious.