LWN: Comments on "Systemd discusses its kernel-version needs"

Upgrade till it's supported, then... ?

HelloWorld — Sun, 01 May 2022 13:31:58 +0000

> But the maintenance burden of those three lines is immense!
You can ridicule it all you want, but it's actually true. It's not that those three lines are complicated, it's that if you’re going to support old kernels, you also need to do the associated testing/QA work to make sure it keeps working, which is a huge pain. This becomes especially relevant when you try to significantly refactor parts of the code.

Besides, kronat’s point is largely hypothetical. Changes that break the kernel’s ability to run on old hardware just don’t happen all that often. Unless of course you insist on running proprietary drivers, in which case the problem isn’t with systemd but the hardware vendor.

anti-systemd is a broad church

farnz — Thu, 14 Apr 2022 12:11:08 +0000

The other two components I see are that Poettering's "style" of software development is to consider the system as a whole, not just his package's piece of it and that Poettering is happy to do the non-coding work involved in keeping the contributor pool healthy.

The first one leads to issues like early PulseAudio releases, where PulseAudio refused to work around bugs in ALSA drivers; instead, Poettering insisted that the drivers should be fixed to correctly implement the ALSA API. Other users of the ALSA API were happy to work around bugs in drivers, and some drivers had simply implemented the API well enough to make common applications of the era work, but had bugs in implementations of things like snd_pcm_query_chmaps, snd_pcm_rewind and the mixer controls that would result in issues for a caller that expected the API contract to be met. Because PulseAudio was the only significant ALSA client at the time (JACK was the other, but users of JACK tended to own sound hardware with good drivers) that cared about the quality of your ALSA driver implementation, Poettering got an unfairly bad reputation, not helped by Ubuntu forcing people over to PulseAudio at a point where the PulseAudio developers (including Poettering) felt it wasn't ready for widespread adoption because they were still sorting through the driver bugs that needed to be fixed.

It also leads to people not liking systemd because it used features of the Linux kernel that had previously been ignored by most users. Poettering took (and takes) the view that if a feature is not meant to be used, then it should not be in the kernel, which has upset certain people who consider features systemd depends upon to be of sufficiently poor quality that they should be removed.

The second point, about looking after the contributor pool, means that Poettering puts up with flak aimed at him when it's in fact someone else's learning curve that's causing the errors; so usrmerge (which was mostly Kay Sievers and Harald Hoyer) gets blamed on Poettering because Poettering wrote it up clearly. When Kay Sievers made a bad decision in udev, Poettering got the flak because he explained the background to that decision, and why, given the context, it wasn't an obviously bad decision to make until the consequences became clear.

Bisecting with old kernels

johannbg — Thu, 14 Apr 2022 10:13:31 +0000

> In my former life as a Linux instructor teaching advanced system administration I had to deal with various people who had picked up the popular notion at the time that the traditional 1980s setup was the bee's knees and systemd the spawn of the devil, but in the end they all admitted, however grudgingly, that systemd is really not a bad idea at all. It takes a little getting used to for people who have been brought up on the traditional approach, but it's worth it, and in my experience people who are entirely new to Linux system administration have a much easier time learning systemd than the traditional setup.

It was apparent early on in systemd's development that getting people to adapt to the new concept would be one of the more difficult challenges to overcome, which is why I was a hard advocated for the 'rip the Band-Aid off' approach at that time ( like no backwards compatibility with the legacy service foo commands ) which in turn would have forced people to let go of the past and adapt the new concept and the commands that came with it. ( if you understood the concept you would understand the approach and also understand everything else that followed )

More cruder, more effective, less painful ( as in having people scream into the void once rather than prolong those screams for over a decade like it has ) approach.

Bisecting with old kernels

pebolle — Wed, 13 Apr 2022 19:56:03 +0000

> Things That Mere Mortals Cannot Fathom

My pet theory is that things like Linux, slowly but certainly, getting easier to run on PC's, virtualization and cloud computing making running servers easy too, and - the big one - mobile phones making countless, almost magical things really easy for almost everyone made clear to many system administrators that their expertise wouldn't be needed for much longer.

It's hard to prove, but I do think the vocal systemd opponents were just shouting as hard as they could to make that future, where their expertise wasn't needed that much, please disappear.

anti-systemd is a broad church

atnot — Wed, 13 Apr 2022 18:21:19 +0000

...and that would, I think your right, certainly have put us in a more productive and fruitful environment for criticizing systemd than what we have today.

anti-systemd is a broad church

atnot — Wed, 13 Apr 2022 17:53:48 +0000

The reason I didn't bring up greybeards is because I don't think they actually played a very big role in what happened with systemd. I'm sure the greybeards grumbled plenty — that's what they do — but rarely does that result in more than obnoxious email theads, nevermind people getting death threats. What made systemd different is that a much broader audience took up the cause. I mean, if one has the misfortune of going to a website like reddit even today, one will still find no shortage of people who installed linux a year ago exhalting the great evils of systemd. Among these were a subcrowd for whom this was not merely about pid1, but the future of FOSS, the west, nay humanity itself (I am not joking), which would eventually remain as the core of the most vocal opposition.

That is not to say that the greybeards hold no responsibility for what happened. But I personally believe that if nobody but them had been stoking the flames, the whole controversy would have probably died down within five years at most.

anti-systemd is a broad church

karath — Wed, 13 Apr 2022 14:56:28 +0000

The above analysis feels mostly true. One sub-community that it appears not to cover are the greybeards* that started with Unix/BSD/Linux in the 70s/80s/90s that had very hard-won experience and expertise that was overturned by the simplicity that systemd introduced. While systemd could not make Linux 'sing' like they could, all of the major distros adopted it and they felt threatened with irrelevance. On the other hand, many others with similar deep expertise did make the jump to systemd, but that's not something that anyone really rants about.

What this means is that few are looking into systemd to find real technical issues. For example, would it make sense to spin out parts of systemd into separate repositories? My feeling is that this is self-censorship - if I conceptually support systemd and then I publicly criticise/question it, then could I be tagged as one of those rabid anti-systemders?

* I am an actual greybeard that started using and programming on Unix v6 in 1980 - but I went in another direction and really only got back to hobby usage of Linux in the past 10 years.

Bisecting with old kernels

anselm — Wed, 13 Apr 2022 13:47:46 +0000

Symbolically rejecting systemd is table stakes for being accepted in these communities, and this is where I think one finds the true reason for the continued ferver.

There's also the issue that some people who have grown up with the traditional system and are heavily invested in having learnt its weird ins and outs hate to see it replaced by something that is more straightforward and unified (let alone technically superior), because that undermines their status as the venerable “greybeards” who Know Things That Mere Mortals Cannot Fathom. People used to defer to their wisdom, and of course we can't have young upstarts like that Poettering taking over and changing everything just because they can. Who do these runts think they are? There ought to be a rite of passage. (Also there must be a conspiracy involved because it is obviously impossible that people would actually prefer something like systemd for its technical merits; it is clear that some nefarious means must have somehow been used to brainwash all the big Linux distributions into adopting it against the collective resistance of the “greybeards”.)

Right now we're seeing this once more with the “/ vs. /usr” debate, where the main counterargument apart from “Resist Poettering forever!” is basically “That's how we always did it and we don't want it to change”, with a side order of irrelevant mumbo-jumbo like “/ is faster than /usr”, even though the real reason (PDP-11 hard disks in the 1970s were too small) is very well known, other Unix vendors have already done the change, and today there are obvious technical advantages to merging the two. This is systemd all over again.

Bisecting with old kernels

atnot — Wed, 13 Apr 2022 10:55:28 +0000

I think that, while the initial resistance to systemd was understandable, at this point it is much more fitting to view resistance to it through a social lens than a technical one. The need for, if not systemd, a service manager much like it has been recognized universally enough that at this point, few people even think about it and the main arguments made against it are vague and selective appeals to nontangible things like tradition. This can be surprising when one considers the ferver with which these points are usually argued, as well as the curious lack of people working on serious alternatives to systemd.

A look at the social side can be very enlightening there. While on a technical level, systemd criticism has failed, on a social level it has been immensely succesful. The anti-systemd movement fourished on 4chan in the early 2010s, initially comprising mostly of reactionary and homophobic sentiments towards poettering, who was there seen as the figurehead of the dawn of a more inclusive foss community. This later mutated into being the core of the more moderate anti-systemd movement we know today, as well as a few auxillary ones like the letters in support of stallman. Symbolically rejecting systemd is table stakes for being accepted in these communities, and this is where I think one finds the true reason for the continued ferver.

The lack of compelling technical arguments or alternatives is because ultimately, what systemd symbolizes has become much more important than any of it's technical merits. This has only increased with perceived symbolic losses like code of conducts and discussions of inclusivity. And while of course, I am still painting with a very broad brush here as of 2022, I think those strokes will become more and more accurate as the movement continues to radicalize and shed less extreme members as it has over the last decade.

Bisecting with old kernels

anselm — Wed, 13 Apr 2022 09:04:30 +0000

The phrase is "do the ends justify the means?". And FWIW, I don't see systemd as having done any of that (as someone who doesn't run a DE but instead puts pieces together).

At the risk of flogging a dead horse that has left the stable a long time ago, here are a few things that systemd gets right which the 1980s-vintage hodgepodge that seems to be so near and dear to some people doesn't:

Consistent configuration scheme across the whole setup (in the traditional approach, literally every service has its own configuration file format)
Clean separation between distribution-provided configuration and local changes, for enhanced maintainability (not addressed by the traditional setup)
Consistent way of starting subprocesses, with enhanced security options, resource limits, etc. available everywhere (in the traditional approach there are multiple ways of starting services, e.g, through /etc/inittab, /etc/init.d, /etc/inetd.conf, /etc/crontab (and friends), …, all different and with different options)
Consistent way of stopping subprocesses (via cgroups; in the traditional setup there is no telling which processes belong to which service, and unless the service itself cooperates it is very hard to clean up)
Detection and re-start of services that have crashed (a problem that the traditional setup mostly ignores unless you deploy non-standard extra tooling)
Reasonable dependency management between services (System-V init requires you to maintain your own topological order of services)
Management of services on behalf of individual users (not addressed at all by the traditional setup)
Convenient access to service status (only rudimentary support in the traditional setup)
Reasonable documentation
…

This is just the tip of the iceberg, and all of these make eminent sense even on systems that don't run a DE. So as far as I'm concerned the answer to the question “is that worth the trouble” is a resounding “absolutely”. You can of course run a Linux system without availing yourself of all these advantages (we've been doing that for decades after all) but, in 2022, why would you?

In my former life as a Linux instructor teaching advanced system administration I had to deal with various people who had picked up the popular notion at the time that the traditional 1980s setup was the bee's knees and systemd the spawn of the devil, but in the end they all admitted, however grudgingly, that systemd is really not a bad idea at all. It takes a little getting used to for people who have been brought up on the traditional approach, but it's worth it, and in my experience people who are entirely new to Linux system administration have a much easier time learning systemd than the traditional setup.

Systemd discusses its kernel-version needs

mirabilos — Wed, 13 Apr 2022 00:58:10 +0000

Just don’t use systemd for these very valid applications.

Or, you know, well, don’t use it, at all. ;-)

Bisecting with old kernels

mathstuf — Tue, 12 Apr 2022 16:47:02 +0000

> “does the purpose justify the actions?”

The phrase is "do the ends justify the means?". And FWIW, I don't see systemd as having done any of that (as someone who doesn't run a DE but instead puts pieces together).

Bisecting with old kernels

andy_shev — Tue, 12 Apr 2022 16:41:39 +0000

Unfortunately systemd did to Unix so many damages that the philosophical question “does the purpose justify the actions?” (it’s my rewording since I don’t know by heart its English variation) is quite actual. It works more or less now, but it may take ages to restore status quo.

Systemd discusses its kernel-version needs

nix — Mon, 11 Apr 2022 22:23:25 +0000

Even when you have a well-supported one... GPUs are complex and change fast and the fallout of associated bugs seems to be almost continuous. I hit one every few weeks, breaking something or other.

The most recent... I upgraded from 5.16.14 to 5.16.17 on the 26th March. All 3D-using Steam games abruptly stopped rendering anything. I thought it just possible that might be Mesa-related, since I upgraded to staging/22.0 Mesa after restarting X the last time but long before rebooting. I know this means that if it breaks I get to keep the shattered bits, but I'm doing it anyway because Mesa 22.x finally did something to fix the game Big Pharma such that it produces readable pixmaps rather than chopped-up messes. (This particular game is really picky and has rendered only black screens or garbage for so many years on so many different platforms that I'd given up hope that it would ever work anywhere again, so I was very happy to try it and find it working at last.)

I upgraded again yesterday, to 5.16.19, without touching Mesa, and everything is working again! I speculated at the time of the reboot that this may be unanticipated fallout of the recent DMA security mess, but I just looked and amdgpu uses none of the affected calls, and none of the amdgpu code changed in suspicious-looking ways between the kernel releases in question.

Is this actually even a regression or not? I have no idea... maybe I was lucky, and it'll go wrong again on the next boot, or the next X restart, or the next Steam restart. I have no idea. I don't even know how to begin finding out.

I have several more bugs like this still live, with different games that don't work, most of which have never worked well, some of which have never worked at all. I should report some of them... not sure where to, though. Mesa's bugzilla? The drm layer's? These are all non-free and most haven't been updated in years, so quite possibly the game is just doing something evil and wrong and unfixable.

(I don't play games much. All this breakage means it's too much like work.)

Systemd discusses its kernel-version needs

bfields — Mon, 11 Apr 2022 19:22:08 +0000

In one scenario, you have a crappy ARM vendor. But then, crappy ARM vendors tend to just make a single code drop (if they drop the code at all) and you get a stale userspace as well. I don't see many attempting to update that.

Isn't that the kind of thing that community distros for phones or wireless routers sometimes try to do? Proprietary drivers and out-of-tree patches might leave them stuck with the original, but they might still have some hope of replacing userspace.

But I may be wrong, I'm not familiar enough with those projects to give specific examples.

Bisecting with old kernels

rahulsundaram — Mon, 11 Apr 2022 16:58:39 +0000

>Al, somehow I sense that you just called me a "cow"? Can you be please be more explicit who specifically you are trying to insult as a cow here?

I don't think that is it. Refer to http://www.catb.org/jargon/html/C/cow-orker.html. There is an insulting reference to "braindamage" which is pretty problematic for people actually suffering from that but atleast in this case, it is directed towards a program, not a person.

Bisecting with old kernels

mpr22 — Mon, 11 Apr 2022 16:48:59 +0000

"cow-orker" is a traditionally adopted deliberate misspelling of co-worker used in certain circles.

Bisecting with old kernels

mezcalero — Mon, 11 Apr 2022 11:48:14 +0000

Al, somehow I sense that you just called me a "cow"? Can you be please be more explicit who specifically you are trying to insult as a cow here?

Is this the new, friendlier Linux kernel community I hear so much about? Because you are certainly cementing the kernel community's questionable reputation with comments like this.

Lennart

Systemd discusses its kernel-version needs

mrugiero — Thu, 07 Apr 2022 14:24:04 +0000

I still don't find the arguments for "maybe systemd will be running on ancient kernels" compelling.

In one scenario, you have a crappy ARM vendor. But then, crappy ARM vendors tend to just make a single code drop (if they drop the code at all) and you get a stale userspace as well. I don't see many attempting to update that. So, even in the unlikely case this vendor's image ships systemd, it's likely to be frozen as well.

In the other, you have some LTS distro that is expected to remain stable. Due or undue, systemd is famed as unstable. I can't see any distro maintainer being dumb enough to pretend to keep an old kernel for stability but then following the latest releases of pid 1, specifically in a version with so many features (and thus surface for stability). I'd dare say the same applies to other systemd services, but at the very least pid 1 I wouldn't expect to be updated.

Systemd discusses its kernel-version needs

zuki — Wed, 06 Apr 2022 10:14:06 +0000

To wrap this up for people who might be reading this article in the future:

In the end in systemd upstream we took the route of recommending newer kernel versions, but not removing any compat code, at least for now. We'll emit a warning when kernels <4.15 are used with systemd-251: https://github.com/systemd/systemd/commit/277f05872f8d5c0...

Systemd discusses its kernel-version needs

dtardon — Wed, 06 Apr 2022 06:27:59 +0000

> Take systemd as an example [1] directly partaking in maintaining that illusion

Except that it is not, as you'd have discovered yourself if you had checked... The overwhelming majority of backports in systemd-stable goes to the latest two releases, i.e., those that are officially supported by systemd upstream. As for the remaining minority, I don't think anything has ever been backported to a release older than $current-3.

> A resource being spent in maintaining that stable branch could easily be freed up by simply stating they will only ever support the current and next release of the upstreams they are dependent on

No, nothing would be freed up. First, systemd-stable already primarily supports the latest two releases, so nothing would change. Second, it is a collaborative effort of distribution packagers. If it haven't existed, every distro packager would have to do backports separately, thus more effort would be spent overall...

(That said, I think there wouldn't be such a need for systemd-stable if systemd managed to do releases with greater frequency, say, once in 4-6 weeks. But then, preparing a release also requires some effort...)

Systemd discusses its kernel-version needs

riking — Wed, 06 Apr 2022 04:26:38 +0000

This is specifically talking about a "conformance test suite" that you could run on a mystery meat kernel and get back a "is systemd supported?" decision.

Systemd discusses its kernel-version needs

dtardon — Tue, 05 Apr 2022 07:11:04 +0000

> High number of unmerged PR's is a clear indicator that a project lacks resources since PR have value higher than issues ( PR's more often than not save time ).

It's not. It only indicates that the project has got a lot of contributors. It would only be a clear indicator that the project lacks resources if the number were increasing over time.

Using (again) systemd as an example, there are 174 open PRs as I write this. When I substract PRs labeled "dont-merge" (22), "reviewed/needs-rework" (68), "ci-fails/needs-rework" (17), "needs-rebase" (28), or "needs-discussion" (35), I get 41 PRs. Let's also consider that systemd is now in stabilisation phase for release of v251, which means that only important PRs are being merged; others, even if approved, are postponed to be merged after the release.

Looking from the other side, the Insights page tells me that 181 PRs have been merged during the past month.

Side note: the reaction time of systemd upstream is amazing compared to other upstreams I've interacted with...

Systemd discusses its kernel-version needs

alison — Sat, 02 Apr 2022 01:14:46 +0000

Looks like /proc/acpi has a tortured history. 7d7ee958867a in 2013 says

commit 7d7ee958867ad3c9c829a36c56e870879e83391f
Author: Lan Tianyu
Date: Fri Oct 11 09:54:10 2013 +0800

ACPI: Remove CONFIG_ACPI_PROCFS_POWER and cm_sbsc.c

There is no user of cm_sbs.c and CONFIG_ACPI_PROCFS_POWER. So remove
them. Prepare for removing /proc/acpi

...That patch was reverted in 2014:

commit e2a7c3d7812369daae56f069eab2e8f3e548d231
Author: Lan Tianyu
Date: Sun May 4 11:07:24 2014 +0800

ACPI: Revert "ACPI: Remove CONFIG_ACPI_PROCFS_POWER and cm_sbsc.c"

The commit 1e2d9cd and 7d7ee95 remove ACPI Proc Battery
directory and breaks some old userspace tools. This patch
is to revert 7d7ee95.

...Then in 2018 the documentation was updated, but /proc/acpi continued onward:

commit 7e46b32bd58768b4823e767dc14d88ab59c6902e
Author: Randy Dunlap
Date: Fri Mar 2 16:55:54 2018 -0800

ACPI / Kconfig: Update ACPI_PROCFS_POWER help text

Fix grammar and punctuation (end sentences with a period) in the
Kconfig help text for ACPI_PROCFS_POWER.

I was looking at this since it appears to be going away (again,
some day) and I have a working script that uses this info to tell me
battery usage. I can update the script to use /sys/class/power_supply
(in theory) but the contents (with units) should be documented in
Documentation/ABI/ before /proc/acpi/battery/ is removed (IMO).</p>

..."Some day" turned out finally to be 5.10. The kernel does sometime "break userspace," but not usually in any haste.

Systemd discusses its kernel-version needs

developer122 — Fri, 01 Apr 2022 21:52:13 +0000

The required support rarely makes it's way into mainline. This stuff is largely out of the sight of kernel devs.

Upgrade till it's supported, then... ?

jafd — Fri, 01 Apr 2022 20:26:08 +0000

But the maintenance burden of those three lines is immense!

Systemd discusses its kernel-version needs

jafd — Fri, 01 Apr 2022 20:24:37 +0000

Imagine all the fun if you have two AMD GPUs in your system and one is for VFIO purposes. 5.17 breaks it because it allocates its memory for simplefb no matter what you tell it. I also had a corruption where the passed through display ended up showing, in UEFI environment, garbled parts of the display manager running on the other GPU.

I would understand (but not forgive) it if the GPUs were identical, but they were not.

Bisecting with old kernels

viro — Fri, 01 Apr 2022 14:55:17 +0000

That's an obvious reason to stay away from systemd. Among the first steps in my debugging workflow: check if I can reproduce it on setup where that thing is not installed. If I can - great; if not - see if it's an effect of systemd braindamage wrt mount subtree sharing setup; if not that either - curse certain cow-orkers (again and again and...) and resign myself to massive headache when trying to do bisections. If it's something more or less recently introduced - lucky me, if it's one of the times when the root of breakage goes back to 2.6.something - well... it happens.

Upgrade till it's supported, then... ?

kronat — Fri, 01 Apr 2022 12:59:01 +0000

Once upon a time, the only way to give old hardware a new life was to install GNU/Linux on top of it. I have an ASUS EEE 900 mini-laptop, and I would expect that installing ArchLinux on top of it, would be feasible (if I only had the time to do it...).

Now, what if in these years, something was added to the kernel sources making me unable to use the latest kernel? I would have to grab some old installer of Debian, let's say 4.0 that was there in 2008, and never touch anything on it, and hoping that everything will be usable (are most websites still working with, let's say, Firefox 3.0?). I consider this case as a regression: I would expect that most of the userspace would continue working with an older kernel (in the end, what I would need to install on it? Basic stuff -- like systemd (unfortunately), a browser, maybe a light DE like XFCE...).

In this light, having a look at the PR linked in one of the first posts is like sand in my eyes. A simple code that checks for a functionality (which doesn't seem to be bug-prone or a source of possible problems) is just removed, and the minimum supported version is bumped.

What are my options against this programmed obsolescence? Just accept it, and stick with proprietary vendors which may decide that every 2-3 years I need to buy a new laptop? I am without words.

Most of the time, the quality one receives is directly proportional to the money one is paying.

Systemd discusses its kernel-version needs

matthias — Fri, 01 Apr 2022 08:36:35 +0000

> The stability of the userspace API means that *most* applications will still work on it though, ...

Actually, the stability of userspace API works exactly the other way round. Old userspace is still supported on new kernels. The kernel cannot make a promise that new userspace will work on old kernels. If new userspace still runs on old kernels then this is purely because of compatibility layers in userspace, e.g., libc falling back to older APIs if newer ones are not available.

And of course it really helps that most userspace code does not really care about most of the kernel interfaces. Much of the interaction with the kernel is passed through system libraries, which thankfully have compatibility layers to support old kernels. And many kernel interfaces are only relevant to the basic plumbing layer like systemd, udev, etc., that is to a very small set of programs.

Systemd discusses its kernel-version needs

johannbg — Fri, 01 Apr 2022 07:32:12 +0000

> That is why long term support and backports are important and also why people pay companies good money to worry about this stuff for them.

No what is important is for the software to work with each other with the latest release so you dont have to backport in the first place.

I suspect that you have never had to go through the actual support channel for these LTS offering service companies have you?

Or have to rely on their "developers" or rely on their incident response team after an breach.

If you did you would have realized you are just literally paying for moral support through a phone or an issue tracker.

Your company would be better off spending all that support/license fee and what not just hiring a psychiatrist, psychologist, or therapist for the relevant managers and IT department to talk to.

Systemd discusses its kernel-version needs

donald.buczek — Fri, 01 Apr 2022 05:41:50 +0000

> > The kernel-upgrade resistance is something that the stable maintainers (and others) have been struggling with for a long time

> It happened to me quite recently that they released a kernel with a buggy intel driver so my machine took about 20 minutes to reach the login screen and was so slow to be completely impossible to use.

We've hit so many regressions updating to newer (stable!) releases, that a kernel update is considered a risk and generally avoided. Kernel developers focus on improvements instead of quality and stability. Bug fixes are interwoven with refactoring and new ideas, so that even in stable releases old bugs are just replaced with new bugs. Some subsystems have just a single active developer and there are not enough independent reviews. The kernel is moving to fast for the developers to sustain or improve quality.

Bugs from mainline are backported into stable releases or are just inherited when a new stable branch is forked. "Stable" doesn't help you much, because sooner or later you need to leave your stable branch anyway and the longer you wait with that, the further you leap and the more problems you have all at once.

Systemd discusses its kernel-version needs

wtarreau — Fri, 01 Apr 2022 04:31:11 +0000

But in fact it does happen because some old interfaces become deprecated, and others are bogus and eventually get fixed. For example I've been training my fingers to read the battery state on my laptop by doing "cat /proc/acpi/battery/BAT0/state" for more than a decade, and recently after upgrading to 5.10 I finally lost it, and now have to read several files in /sys.

Similarly some entries in /sys used to be misdesigned, with multiple values in the same entry, which is contrary to the /sys design rules. Some of them were finally fixed and split into multiple entries.

Some quickly written scripts that rely on these entries may occasionally break.

The bonding configuration interface changed over time to only use /sys. In the past you would pass arguments to modprobe then use ifenslave to configure the interface (but that wouldn't scale to multiple interfaces). Similarly code written for this eventually had to be changed.

There are definitely situations that cause visible breakage. It's not necessarily a regression but often the replacement for something wrong that cannot be supported anymore and that very limited stuff used to rely on. And these changes do require extra validation on upgrades that cannot make them 100% transparent. That's why LTS kernels are much appreciated for deployment in remote devices at customers'. They limit the risk of surprise on a supposedly harmless upgrade.

Systemd discusses its kernel-version needs

Ranguvar — Fri, 01 Apr 2022 00:53:15 +0000

Yep, 5.17.1 seems to break video output on boot.

I was able to login blind, but will add an LTS kernel to my boot options for future use.

Systemd discusses its kernel-version needs

Matt_G — Fri, 01 Apr 2022 00:51:57 +0000

5 Years is in my opinion a very short lifespan for operating system software. At the industrial plant I work at for some production critical systems there is very limited downtime available for updates patching etc. Which is why long term support is an important thing and also why my org pays vendors for commercial support contracts and that sort of thing.

It is not my job to worry about these things but if it was I'd be incredibly angry if I heard back from vendor that 5 years into deployed life the issue we experienced can't be fixed without updating to a newer kernel, but specialized hardware or software (like a commercial database or something else) won't support the newer kernel.

That is why long term support and backports are important and also why people pay companies good money to worry about this stuff for them.

Some of our systems and equipment have 25year + lifecycles. I don't work in IT side so I don't know all the specific details I do know there is plant equipment installed in the early 90's being controlled by Unix software/code written during that era. This software was highly likely to be designed to be run on commercial hardware which probably isn't supported by modern kernels.

Systemd discusses its kernel-version needs

johannbg — Thu, 31 Mar 2022 23:09:25 +0000

> That sounds totally untrue.

Feel free to do your own research, own math yourself and your own resource measurements, but by my experience ( my own and work ) a single issue, phone call or disruption from a coworker, the cost of distraction to the developer is always atleast one hour on average.

With meetings these numbers go up significantly, which is why you never schedule meetings with developers unless it's strictly necessary and those meetings will have to be held earlies in the morning ( prior to them starting work ) or at the end of the workday ( after they finished their work ).

> There would be a serious resourcing problem if the issues don't even get triaged
That an issue has been labelled ( you can call that label what you want, triage, RFE whatever ) just means two things a) the issue has been looked at ( or labelled automatically like we do in our projects ) and b) the bug will be looked at again ( otherwise it would not still remain open ).

So an hour of an individuals time has been spent on the issue ( if manually looked at ) and atleast an hour will be spent on it again until it enters the closed state in it's lifecycle.

High number of unmerged PR's is a clear indicator that a project lacks resources since PR have value higher than issues ( PR's more often than not save time ).

Obviously I'm just pointing out the calculation for the upstream maintainers as in I'm not bringing into the equation the time spent by the reporters, documentation writers qa's etc. ( which often get's ignored,overlooked since most people seem to be fixated strictly on the developers ) since when you looking at projects resources you look at the available pool of contributors time. ( not just the developers but everyone partaking in the project ).

Systemd discusses its kernel-version needs

excors — Thu, 31 Mar 2022 22:11:08 +0000

> a project that has to prioritize is a project that is starved of resources

That sounds totally untrue. A healthy well-resourced project will have lots of users, who inevitably submit many low-quality bug reports where it will take a large amount of effort to investigate a very rare or non-existent bug, and will also have an endless stream of ideas for new features and improvements with varying levels of effort and value. It's always important to prioritise, because resources are always finite and there will be lots of issues that are never worth fixing. A project that *doesn't* have to prioritise is one that has few users and the developers have run out of ideas, which would not be a good sign.

Also the effort/value cutoff point will vary over time, as the list of outstanding issues and available resources vary, or as you accumulate more reports of a particular bug or feature request, so it's not practical to immediately WONTFIX everything below a certain threshold - it's better to leave them open in the issue tracker because there's a chance they might become worthwhile later. As long as the issue tracker has a decent UI, it's fine to have thousands of open issues.

(There would be a serious resourcing problem if the issues don't even get triaged and nobody can tell whether they ought to be high priority or not. But systemd doesn't appear to have that problem - nearly all the recent still-open issues have comments and labels that indicate a developer has looked at them, and a large majority of the ones labeled "regression" or "bug" are closed, and half of the open issues are RFEs, so the first impression is that they're doing okay.)

Systemd discusses its kernel-version needs

rahulsundaram — Thu, 31 Mar 2022 20:45:40 +0000

> Upstreams either directly ( through their employer ) or indirectly are also partaking in maintaining the stability theatre industry illusion that offers only long term moral support through phone or issue tracker at best

This is incorrect. There are many cases for example where bugs are identified and present in upstream releases, fixed upstream and backported to these branches, hotfixes supplied directly in response to customer issues, security issued identified and reported, documentation written in response to questions etc.

> even the kernel community ]2] that maintains multiple stable branches and multiple long term supported kernels which make literally no sense ( there is no support in (F)OSS project only ever best effort ).

There certainly is support in FOSS projects, just not commercial support which is only one of many support possibilities and since providing bug and security fixes is a form of support, LTS makes sense within that context.

Systemd discusses its kernel-version needs

johannbg — Thu, 31 Mar 2022 20:29:56 +0000

Upstreams either directly ( through their employer ) or indirectly are also partaking in maintaining the stability theatre industry illusion that offers only long term moral support through phone or issue tracker at best.

Take systemd as an example [1] directly partaking in maintaining that illusion or even the kernel community ]2] that maintains multiple stable branches and multiple long term supported kernels which make literally no sense ( there is no support in (F)OSS project only ever best effort ).

Not only are upstreams that partake in that illusion shooting themselves in the foot by adding that maintainership to the overal cost of development for their project but they are also preventing themselves from progressing since they are getting distracted having to maintain their "stable" and or longterm branches.

Using systemd again as an example there does not even exist resources in the project to maintain that stable branch when the project itself has 177 PR's open and 1.6k open issues to go through on it's main branch. ( it takes at least 1700 hours to go through all that and a project that has to prioritize is a project that is starved of resources ).

A resource being spent in maintaining that stable branch could easily be freed up by simply stating they will only ever support the current and next release of the upstreams they are dependent on, any older release than that yeah sure it might work in conjunction with older kernels and what not but you are on your own if you decide to mix and match various releases of various components that systemd uses or is depentent upon because the project is *always* moving forward along with the other upstream it uses or is depentent upon.

1. https://github.com/systemd/systemd-stable
2. https://kernel.org/

Systemd discusses its kernel-version needs

Ranguvar — Thu, 31 Mar 2022 19:51:33 +0000

Huh. I have a WX 5100 but haven't tried 5.17.x yet.

I use a second GPU but it's an NVIDIA card I use vfio-pci with.

I'll report back if any trouble tonight.