LWN: Comments on "An unstable Debian stable update"

Hehe

jengelh — Tue, 07 Oct 2025 10:40:25 +0000

They do use systemd. But not Debian ;-)

Do not use non-core systemd

kreijack — Thu, 02 Oct 2025 16:57:49 +0000

Changing an interface is a good example: changing an interface is a very expensive activity. Far more expensive than any other change. But in a "big" project where there the interface are not very well defined, everything may be an interface. Thus every time any part of the code is touched, the developer must check that this will not impact the rest of the project.

Having small module simplify (==less time) the code comprehension and the check of the change impact. Because the code to inspect is smaller.

If you need to issue an urgent change (a bug, a security fix), it is enough to audit only the involved portion of the code.

NB: despite what I wrote above, I think that systemd code has a very quality with an extensive use of modern technique to simplify the development (e.g. the __cleanup__ macro).

Stability

wpeckham — Thu, 02 Oct 2025 15:31:11 +0000

I recall a time when Debian pushing updates to stable that disabled networking FOR ANYONE using supported configurations would not have been imaginable.
That was before SystemD was adopted.

Since then Debian Stable has been seen as LESS stable increasingly with time.

This may be more perception than reality, or not, but events like this one do NOT help!

The important question is not who to blame, or if users should or should not avoid DOCUMENTED AND SUPPORTED CONFIGURATIONS, it is "How do we avoid this kind of event in the future?".

Because we absolutely should be focused on that!

Some comments are just… beyond comment

jubal — Mon, 29 Sep 2025 16:09:51 +0000

the actual problem is that boccassi is unpleasant to deal with in most public communication settings because of his communication style (abrasive, dismissive, frequently insulting). this is not the first time this behaviour created or increased friction, and made a simple, fixable issue into a flamefest. and it doesn't matter if he's privately the loveliest of the nice people around; the image he projects publicly will overshadow that, and adds to the bad reputation of systemd maintainers' social skills.

Do not use non-core systemd

anselm — Mon, 29 Sep 2025 01:41:35 +0000

If you have to do it "in house", you need to ask the question "why".

Presumably because it's easier and more convenient to do it that way.

Having said that, the systemd developers do document a number of their internal interfaces, so it should be possible, at least in principle, to come up with alternative implementations of some of its components (but again, we would want to ask the question “why”, especially when improving systemd itself is also an option). Some people may argue that the specific interfaces that they are interested in are not (yet?) documented, but the reasons for that are probably best discussed with the systemd developers themselves.

Do not use non-core systemd

Wol — Sun, 28 Sep 2025 23:35:42 +0000

> Not necessarily. Currently if they want to change such an interface, they can do it "in house", update both (all) sides at the same time (maybe even in the same commit?

But is this "as simple as possible, but no simpler"? If it isn't then you have a problem full stop.

If you have to do it "in house", you need to ask the question "why". And if you don't have an answer, you need to fix it.

Cheers,
Wol

Some comments are just… beyond comment

smurf — Sun, 28 Sep 2025 12:37:55 +0000

> But you, like a lot of people, are focusing on rudeness as the issue

Two sides of the same coin, effectively.

The point is that you need to get rid of the whole coin, i.e. "fix" both sides (for whatever value of "fix"), if you want to get a handle on the problem.

Do not use non-core systemd

mathstuf — Sun, 28 Sep 2025 02:55:41 +0000

Indeed it does. Thanks; I can at least use the non-symlink version then (as the symlink (target?) rewriting breaks the bind mount of the configuration file).

Do not use non-core systemd

ebiederm — Sun, 28 Sep 2025 01:49:56 +0000

It should be just a matter of putting a new nsswitch.conf in the right place.

If my memory serves the code just walks through the configuration directory and bind mounts every thing in it, into /etc.

Do not use non-core systemd

mathstuf — Sun, 28 Sep 2025 01:18:05 +0000

That sounds like a job for `systemd-inhibit(1)`. See https://systemd.io/INHIBITOR_LOCKS/

Some comments are just… beyond comment

marcH — Sat, 27 Sep 2025 19:56:49 +0000

> But you, like a lot of people, are focusing on rudeness as the issue,...

Yes, because if you care about any success metric, then communication is the most important thing by far. Building something exceptional _alone_ never goes anywhere. I mean it can be fun and beautiful but that's all. If it's really exceptional _and_ re-usable, then it's sometimes rescued by other people who can work together. And then the "inventor" complains he was misunderstood. He was. Often it stays unused on the shelf. Everyone can admire it but that's all.

No maintainer is perfect, very far from it. But any widely distributed component is direct evidence that the maintainer has at least the most basic communication skills. If not, then either the maintainer or the component gets forked or replaced sooner or later. Simple as that. The same cannot be said about entitled users shouting on the Internet, there's just no evidence on their side.

> Do you think people at Google would maintain their calm demeanor if you had to keep everything running while dealing with that?

They would stay professional long enough for the problematic person to be demoted or transferred to another place better suited for them. If that does not happen or takes too long, then professional people try to find a another job in a proper workplace.

Too many ways to configure networking

koverstreet — Sat, 27 Sep 2025 18:11:58 +0000

Yeah, it's gotten really bad. I used to be able to block them with a 10 line script that scraped the apache log, sorted IPs by number of accesses, and iptables blocked the ones that were over some silly number.

Now, they're all coming from different IPs - it's only a few accesses per IP - and the load was so bad the git fetches for the CI workers were unable to run. It ground my test infrastructure to a halt!

They're a bloody menace.

Although I'd also really like to know why git is so ridiculously memory hungry.

Do not use non-core systemd

mbunkus — Sat, 27 Sep 2025 18:11:22 +0000

> I think that dividing systemd in different sub-projects separated by a clean/clear interface, would speed up the development

I disagree. Splitting in such a way would require more effort (for coordinating releases, for implementing/testing/documenting/maintaining the interfaces you're talking about etc). If the same developers are interested in working on the various split components that're currently working on the umbrella project, the speed of development of new features will go down.

Your statement assumes that after splitting there will be a net inflow of developers providing more work, just to offset the aforementioned new overhead, never mind speed up development. "Net inflow" because there might also be developers who prefer to work on a big umbrella as it is now & will reduce their work as a result.

I would really be interested in actual studies of development speeds in more monolithic vs split-up projects (and where the difference is due to project size and not due to factors such as the monolithic project having too many a**holes on its team), though I doubt it due to how incredibly difficult it would be to have comparable data. Is there even anecdotal evidence from past/ongoing situations? Really curious.

Too many ways to configure networking

smurf — Sat, 27 Sep 2025 11:18:21 +0000

I took that opportunity to look at my own cgit traffic. Yikes. Anubis downloaded and installed, nginx config adapted. Took ten minutes. "git clone" still works, no config modification required.

I do wonder how long it'll take these *censored* to notice that each request now results in an identical 12-kByte result (1.2 on the wire, it's compressed) … in the meantime I bet we're going to get a veritable ton of random search hits (and AI chat replies) for basically anything that smells like Anubis' challenge page.

What was that curse again … "may you live in interesting times".

NB my log shows a marked increase in the number of AI bot requests since yesterday, which is not at all surprising: Anubis replies a whole lot faster than cgit.

Do not use non-core systemd

smurf — Sat, 27 Sep 2025 10:55:34 +0000

> Incorrect assumption. Systemd-homed regularly resizes images/partitions to rebalance the disk space available between multiple users.

Ah. Thanks for the correction.

One might assume that it was an SSD that was powered down during writing. In this case all bets are off. Don't Do That.

Thus there needs to be an "work in progress, do NOT shut down" dialog/message/whatever. (Assuming that systemd doesn't time out the resize.)

Do not use non-core systemd

NAR — Fri, 26 Sep 2025 19:33:26 +0000

I think that dividing systemd in different sub-projects separated by a clean/clear interface, would speed up the development

Not necessarily. Currently if they want to change such an interface, they can do it "in house", update both (all) sides at the same time (maybe even in the same commit? - I'm not familiar with the systemd codebase). If there are (more or less) independent (sub)projects, then it will take inter-project coordination which is always slower. And possibly keeping around technical debt for backwards compatibility.

Do not use non-core systemd

kreijack — Fri, 26 Sep 2025 17:17:57 +0000

> The trick with systemd is to know which parts of it are good and skip the bad parts. It's really an umbrella project with lots of different modules that (try to) work with each other.

I agree. Is a too big project with a too wide scope. Am not saying that it there are "bad" parts; the problem is that all the parts are coupled, and this is source of scalability problem:
- if you have an issue with a component solved in a newer version you need to update all the components (even the one that works flawless)
- all the components can't rely on a defined interface but can access to any interface that they want. This prevent to replace a component with a more specialized component. For example I suspect that the binary format of the journal is inefficient, but it is complex to replace only this part because journald is integrated in systemd.
- I can understand that all the parts that can manage "events" should be integrated in systemd. But why (e.g.) sd-boot and homed should be integrated in the main project ?

I think that dividing systemd in different sub-projects separated by a clean/clear interface, would speed up the development, and open the cooperation with different projects that may provide more specialized tool for vertical case.

Do not use non-core systemd

marcH — Fri, 26 Sep 2025 16:04:55 +0000

> So, choose your poison, good easy to maintain config files, or solid networking system with less fortunate config files

For software I choose the one which is "as simple as possible, but not simpler".

That means systemd-networkd in simple, static network configurations and NetworkManager on laptops with wifi, VPNs and what not.

Do not use non-core systemd

nim-nim — Fri, 26 Sep 2025 08:38:05 +0000

The usual use case for split dns is a backend network with private resources, and a hardened front end network for remote access. This kind of setup to work pretty much requires dual routing (and one could route remote connexions via the backend link but that would defeat the purpose of isolating untrusted traffic on a heavily audited frontend network).

(E)LILO was discontinued a decade ago and removed from Debian.

linuxrocks123 — Fri, 26 Sep 2025 07:47:19 +0000

Ah, didn't realize WolfWings was both the replier and topic originator. His original comment didn't specify Debian, but I guess he is personally mostly interested in SystemD non-core components on Debian.

In that case, I expect compiling ELILO on Debian would be straightforward, and I also expect the old .deb file from whatever the last release to include ELILO was would probably still work. However, I don't use Debian and so have not tested that.

Do not use non-core systemd

intgr — Fri, 26 Sep 2025 07:08:39 +0000

Oh, I must be misremembering. But regardless, that's when homed frequently does shrinking.

Do not use non-core systemd

ATLief — Fri, 26 Sep 2025 04:01:42 +0000

Systemd-boot lets you easily create UKIs, and the Linux kernel alone has a bunch of annoying limitations.

Do not use non-core systemd

ATLief — Fri, 26 Sep 2025 03:44:23 +0000

While Systemd-boot is not *entirely* a boot-loader, it does contain a UEFI stub in addition to the UEFI boot menu. UKIs are basically that stub with the kernel and initramfs appended to it, and they can function as standalone UEFI executables without the boot menu.

Hehe

ATLief — Fri, 26 Sep 2025 03:37:25 +0000

Presumably Debian would continue to support the version series that was included in Stable for the remainder of its support period. That being said, bigger projects almost always provide plenty of warning, and many of them even maintain older version series themselves.

Do not use non-core systemd

mathstuf — Fri, 26 Sep 2025 03:17:23 +0000

Ah. I think of network namespacing tricks as a way to get split DNS without actually allowing a process to route over both at once. But perhaps I'm just doing something "more" than "just" split DNS then. In any case, I've found a block tower setup that works to my satisfaction; I'm more looking for ways to put more mortar around the base rather than "don't look at it sideways too hard" stability.

Do not use non-core systemd

mathstuf — Fri, 26 Sep 2025 03:15:30 +0000

Yes, `ip netns exec` is involved in using it. It knows how to deal with `resolv.conf` and mounts that in a mount namespace for the process, but not `nsswitch.conf` AFAIK.

Which reminds me that I need to also fix `resolv.conf` so that things are happy too. If it is a symlink, the `ip netns exec` mount namespace is broken when the target `resolv.conf` is rewritten with a "write then rename into place"; `cat newresolv.conf > /etc/resolv.conf` works though.

(E)LILO was discontinued a decade ago and removed from Debian.

intelfx — Fri, 26 Sep 2025 02:55:43 +0000

> I didn't say Debian packaged ELILO. This sub-topic is about SystemD components in general, not about Debian in particular.

This comments section as a whole is about "Debian in particular", though. The commenter you were replying to is also, presumably, a user of Debian in particular.

So when you post a comment on an article (about Debian) replying to another commenter (a user of Debian) saying "Why would someone need [systemd's] bootloader? <...> ELILO still works" and then claim that it does not matter that ELILO is not, in fact, available in Deban, then you are simply being obtuse. HTHs.

(E)LILO was discontinued a decade ago and removed from Debian.

linuxrocks123 — Fri, 26 Sep 2025 02:33:04 +0000

I didn't say Debian packaged ELILO. This sub-topic is about SystemD components in general, not about Debian in particular. I said ELILO still works, because I made a dual-boot system with Windows about three weeks ago, let the Slackware-Current installer install ELILO, and it worked.

I also didn't say ELILO was updated or supported. But, since you mentioned it, I'll just say that I generally don't whether software is updated or supported, and I generally do care if it works.

Regarding boot menus, I've found the UEFI boot menu available with F12 to be sufficient, so I don't see a need for a separate program to provide that feature. Given all of the over-engineered nonsense built into UEFI, I'm pleased that the resulting mega-kludge can at least display a better boot menu than its predecessor.

But I'm still looking forward to CSMWrap being developed to completion.

Do not use non-core systemd

intelfx — Fri, 26 Sep 2025 00:44:13 +0000

> Particularly, btrfs can only be shrunk when unmounted

Without commenting on the rest of the analysis, that's not true.

Do not use non-core systemd

intelfx — Fri, 26 Sep 2025 00:19:35 +0000

> I have to reconfigure resolved (via `/etc/nsswitch.conf`) because it causes network namespace leakage. If I have a network namespace jailed to a VPN <...>
>
> Is there some configuration magic around to handle this that you know of?

I don't think systemd-resolved is network namespace aware, and I'm not familiar with any tricks in that area. But that's a hugely niche use-case, and not what I was defending. Normal "split DNS" is when you have multiple network interfaces with disjoint sets of routes in a single network namespace.

Do not use non-core systemd

intgr — Fri, 26 Sep 2025 00:15:02 +0000

> I'd assume that the partition table of any image (or hard disk) is essentially read-only

Incorrect assumption. Systemd-homed regularly resizes images/partitions to rebalance the disk space available between multiple users.

Particularly, btrfs can only be shrunk when unmounted, but obviously needs to be done when encryption keys are still available, so it frequently happens during system shut down.

> something that merely hangs,

It's not a "hang". The resizing can take a few minutes even on fast SSD if lots of data needs to be relocated.

Do not use non-core systemd

koverstreet — Thu, 25 Sep 2025 23:57:58 +0000

Bugs without reproducers are a normal fact of life, you can't just give up any time you see one.

To start, you look at all the information you have available; in this case that'd mean looking at the corrupted partition table - hexdump if necessary - to see if you can spot any patterns or deduce anything about what happened.

Sometimes that can tell you all you need to know; I once saw a guy deduce just from a hexdump of some memory corruption that it was an errant memmove that shot past the address it was supposed to stop at - he spotted that everything had been shifted by 8 bytes - and then from that, found the exact line of code that caused it (an open coded memmove).

If that fails to identify the bug, you take what you do know and look for places where your code is weak and could be hardened; you look for ways to make entire classes of bugs impossible. This is on-disk data structures we're talking about; we don't have practical techniques for making entire codebases in systems languages bug free, but we absolutely can identify ways to limit damage and make sure that on disk data structures aren't corrupted.

Do not use non-core systemd

ebiederm — Thu, 25 Sep 2025 23:53:13 +0000

pair bind mounts of the config file with the network namespace?

ip netns allows for that. I don't know about systemd-networkd

Too many ways to configure networking

koverstreet — Thu, 25 Sep 2025 23:50:03 +0000

yeah I just had to flip off cgit, thanks to - you guessed it - too much AI crawler traffic

so until we get anubis going, there's a github mirror: https://github.com/koverstreet/ktest/

Do not use non-core systemd

smurf — Thu, 25 Sep 2025 23:39:41 +0000

I'd assume that the partition table of any image (or hard disk) is essentially read-only, in the sense that nothing writes to these areas, and thus cannot be corrupted by something that merely hangs, no matter what you do to the process controlling it.

Thus, whichever freaky bug you encountered is not reproducible. So what should the maintainers actually do, other than say "yeah it's a strange case of data corruption but if we don't see a pattern, much less a reproducer …" and leave it open?

Too many ways to configure networking

smurf — Thu, 25 Sep 2025 23:32:00 +0000

This link throws a 404.

Do not use non-core systemd

intgr — Thu, 25 Sep 2025 22:15:19 +0000

> The good parts include [...] home directory management.

I'm not sure, that part might be abandonware. You would think that if it was maintained, not corrupting user data world be high on the list of priorities.

A year ago I had an incident where during a regular system shutdown, the systemd-homed resize operation took too long and got killed due to timeout. Result was a corrupted partition table inside the user home image and I could no longer access my data.

OK, bugs happen, but what baffled me was that to this day, no systemd developers have acknowledged or reacted the bug report, despite reports from other users also being affected.

Thankfully after a few hours of struggling, I managed to mount the image and could recover my data. But to this day, I still hold my breath every time I restart my system that this bug doesn't hit again.

Some comments are just… beyond comment

madscientist — Thu, 25 Sep 2025 21:14:11 +0000

The quoted comment was not responding in that aggressive way to the existence of the bug. It was responding to the handling of the bug report: in particular deciding the issue had an importance of "minor" with all that implies procedurally. This is not really comparable to your situation, unless Google SREs are in the habit of dismissing colleagues' outages as minor and not important enough to fast-track a fix for.

I try to follow the old internet protocol advice "be strict in what you send and lenient in what you receive", when dealing with other people. You never know what is happening in someone's life. The question is not whether this particular comment is abrasive. The question is, is this a pattern of behavior or is it an aberrant, in the moment reaction? If it's a one-off, give grace and look the other way. If it's a pattern of behavior then something should be said. If it keeps happening after that, then something needs to be done.

But that goes for the maintainer, too, not just the commenter. I think it's disingenuous to concentrate only on one side of the conversation. It's idealistic, not realistic, to say "it doesn't matter what one side does, there's no excuse for the other side to behave in anything less than a fully professional manner".

(In)competence

smurf — Thu, 25 Sep 2025 20:56:01 +0000

Competence manifests in more than the ability to code.

When you're as dismissive and abrasive as bluca to people who happen to hold different opinions, sorry but I call that incompetence. Plain and simple. Not in coding but in relating to people, and in understanding their concerns.

Also, let's be brutally plain here: regardless of *any* nonsense in your config files, it is *never* OK to segfault, and by extension it's also never OK to dismiss a segfault-inducing bug as unimportant.

Not if you're a core system component.

Some comments are just… beyond comment

koverstreet — Thu, 25 Sep 2025 20:23:42 +0000

I used to work at Google too, and the difference is that it's a closed environment where you have to demonstrate a high degree of competence to be there, and there's very much a culture (especially among SREs) of post mortems, not assigning blame, etc. - and critically, there's a chain of command so that if you screw up at your job and aren't learning or taking responsibility for your mistakes, your coworkers have recourse and won't just be stuck with you. And you control your _entire_ stack, you can call up or walk to the desk of whatever component you rely that isn't working.

It's wonderful for the people who get to work there.

But you, like a lot of people, are focusing on rudeness as the issue, when a lot of us are going "uh, machines going down on update, intransigent maintainer with a pattern of not acknowledging or fixing his own mistakes, blame shifting, and forcing other people to cover - yeah, I'd be pissed too".

Do you think people at Google would maintain their calm demeanor if you had to keep everything running while dealing with that?

I think you might have cause and effect a bit backwards :)