Systemd discusses its kernel-version needs [LWN.net]

By Jake Edge
March 30, 2022

A query regarding the possibility of dropping support for older kernels in systemd led to some discussion on the systemd-devel mailing list recently. As might be guessed, exactly which kernel would be the minimum supported, what kernel features systemd is using, and when those kernel features became available, were all part of that conversation. A component like systemd that is closely tied to the kernel, and the interfaces different versions provide, has a number of different factors to consider when making a decision of this sort.

Zbigniew Jędrzejewski-Szmek started things off by asking if changing the minimum required kernel version for systemd to 4.4 would cause problems for anyone. He said that if it did, "please substantiate why you are running new systemd with such old kernels". Currently, systemd minimally requires Linux 3.15 or later, as noted in its README file.

Which version?

But Greg Kroah-Hartman was quick to point out that the stable-kernel team recently stopped supporting 4.4, so the 4.9 kernel is the oldest version that is still getting stable updates. He suggested using that as the baseline instead. Jędrzejewski-Szmek noted that the Civil Infrastructure Platform (CIP) will continue to use 4.4, but that Debian 9.x ("stretch") uses 4.9 and has its end of life in June 2022. "Of course we'd like to move to 4.19, but we don't want to disrupt distros that use older kernels. Is ≤4.19 really unused?"

Michael Biebl filled in more details for the Debian picture. Newer systemd versions are being backported to Debian 11.x ("bullseye"), which uses the 5.10 kernel, but not to earlier Debian releases. He did note that downstream Debian derivatives (such as Raspbian) might have other requirements, as might those who build their own kernels. Neal Gompa said that there are active efforts to bring newer systemd releases to CentOS Stream 8, which is based on 4.18 plus a bunch of kernel-feature backports. He would like to see that continue to be supported.

Luca Boccassi said that he would like to see the minimum version only go as far as 4.4 since "what's on kernel.org is not really that important, as real usage is downstream from there anyway". The important thing is to identify what's needed for systemd core compatibility—all of that is available in 4.4, he said. He noted that support for version 2 of control groups (cgroupv2) came in 4.1; being able to count on cgroupv2 support would be helpful. Control groups are a resource-management mechanism that is used extensively by systemd.

Not surprisingly, Kroah-Hartman had a rather different take on that; "anyone still using 4.4.y today has an insecure and unsupported system". He does not want to encourage anyone to use that version. While CIP is supporting 4.4 until 2027, it is using that kernel "in a very limited set of use cases and configurations". He is somewhat skeptical that CIP will be able to keep that kernel "alive and secure"; for anyone else that is still using 4.4, "send them to me please".

Boccassi disagreed with that view; kernel-version choices are often out of the hands of the user because they have to use what their distribution or vendor provides. If the kernel version for systemd changes, he expects that the project will hear from "some poor soul stuck on 3.x because of $crappy_arm_vendor with no way to move on from there". Even so, moving to 4.4 has some advantages:

Jumping forward from 3.13 to 4.4 as the baseline, allowing to take cgroupsv2 for granted, seems like a good starting point to me. There's very obvious and public evidence of that being used in the wild. We can start to drop a bunch of backward-compat cruft, wait and see who complains, and if nobody does we can re-evaluate again in a couple of years.

But Kroah-Hartman thought it is unlikely that people stuck on 3.x kernels are updating to the latest systemd version; "if they are, they need to get $crappy_arm_vendor to do the work for them as they PAID for that support and work already".

Ulrich Windl said there can be value in sticking with the same kernel over time, though, because "new features intended to improve things sometimes actually make things worse". But in choosing which kernels to support for systemd, that is not really the issue, Kroah-Hartman said:

Do you want to run a kernel with known security problems, or one with "unknown potential problems." The latter is always the case, so please don't pick the known-insecure one, that's just foolish.

Boccassi was not impressed with that argument, noting that "'security problems' are a dime a dozen, as they say" and complaining that the stable kernels often have regressions of various sorts. He made a rather surprising claim about the nature of those regressions, however:

Upgrading major kernel version is like rolling a dice, you never know what kind of extremely expensive and time consuming rabbit hole you'll be dragged into because the kernel plays fast and loose with its userspace interfaces, and each and every time there's a chance one might end up having to do major reworks to deal with it.

Kroah-Hartman seemed surprised to hear that, since the kernel developers take user-space compatibility seriously. As readers of the LWN kernel page know, Linus Torvalds will quickly step in and remind kernel developers, sometimes rather harshly, that the kernel does not break user space. There are often complaints about regressions in the stable tree, but it does not typically involve breaking user-space interfaces. Kroah-Hartman said that he was interested in hearing any such reports and that the proper place to report them was the kernel's regressions mailing list.

That was something of a side conversation to the main point of the discussion, but the resistance to upgrading to new kernel versions is real; it is somewhat unclear why that resistance would not also extend to systemd versions, however. The kernel-upgrade resistance is something that the stable maintainers (and others) have been struggling with for a long time. Those who plan to stay on older releases, or not upgrade series that are still supported, are playing a dangerous game. Kroah-Hartman clearly sees the choice of the minimum supported kernel version as a means to help spread that message further, though it is unclear how successful it would be.

Control group evolution

Back to systemd itself, Lennart Poettering said that there had been an evolution in cgroupv2 support over time, so narrowing down exactly what systemd needs or wants in that support is needed:

Note that "cgroupv2 exists" and "cgroupv2 works well" are two distinct things. Initially too few controllers supported cgroupv2 for cgroupv2 to be actually useful.
What I am trying to say is that it would actually help us a lot if we'd not just be able to take [cgroupv2] for granted but to take a reasonably complete cgroupv2 for granted.

Boccassi agreed that information would be useful, especially which cgroupv2 controllers are needed and when they were added. As Jędrzejewski-Szmek pointed out, the README does list some features that systemd can use and what kernel is needed for them:

Linux kernel >= 4.2 for unified cgroup hierarchy support
Linux kernel >= 4.10 for cgroup-bpf egress and ingress hooks
Linux kernel >= 4.15 for cgroup-bpf device hook
Linux kernel >= 4.17 for cgroup-bpf socket address hooks

In this light, 4.19 is better than 4.4 or 4.9 ;)

Given that, Kroah-Hartman was in favor of choosing 4.19: "I strongly doubt that any distro that is using older kernels would ever be willing to update systemd." Windl added a data point, saying that the most recent service pack of SUSE Enterprise Linux 12 (SLES12 SP5) is using 4.12, but he agreed that it was not likely it would be moving to newer systemd versions.

The list of kernel versions in the README is mostly concerned with various BPF-related features, Boccassi said. But systemd should still run without support for BPF; "There's plenty of use cases that disable it entirely (in the sense, things shouldn't fall apart if we run on a non-bpf kernel and there are no bpf options configured in any unit). I have one of them." BPF-related features should remain optional in systemd forever, he said.

Poettering took a look through the systemd man pages to get a sense for what kernel features are being used and when they were added. He noted that cgroupv2 support was not really viable for systemd until 4.15, since before that it lacked the CPU controller, which is used to partition the CPU use between different groups. Beyond that:

some other interesting milestones:

kcmp → 3.5
renameat2 on all relevant file systems → 4.0
pids controller in cgroupv1 → 4.3
pids controller in cgroupv2 → 4.5
cgroup namespaces → 4.6
statx → 4.11
pidfd → 5.3
This is just some quick search through man pages. There might be a lot of other stuff that would make sense for us to be able to rely on.

Windl wondered if some sort of conformance test suite for systemd made sense; "If the test suite succeeds, systemd might work; if it doesn't, manual steps are needed." Windl guessed that if manual steps were required, it might deter most people from going any further. But Poettering thought that was not the right path forward: "One goal here is to reduce our [maintenance] burden, not increase it. Another is to communicate clearly what we support and what we don't. Any such test suite collides with both these goals."

The conversation wound down, seemingly without any final conclusions. While support for 4.4 might send the wrong message, at least from the stable-kernel maintainers' point of view, it may make sense if the CIP project has plans to upgrade its systemd along the way. Jędrzejewski-Szmek opened an issue on the project's GitLab site to try to find out. It does seem reasonable to at least conjecture that distributions (and others) that are unwilling to upgrade their kernel might also be leery of upgrading a major component like systemd.

Since 4.4 lacks the cgroupv2 support that is desirable, 4.9 or even 4.19 seem like the next logical possibilities, though 4.9 reaches end of life in January 2023. In addition, 4.9 is lacking in the cgroupv2 department as well. The 4.19 kernel will be supported for nearly two years longer than 4.9, until December 2024, and has most of the desired features. It may well be where the project decides to land.

In the end, the systemd project will need to find a good balance between the features it needs from the kernel and the kernel versions that its users are running—if they still plan to upgrade to newer systemd versions anyway. Most distributions will likely shy away from kernels that have fallen outside of the stable-kernel support windows, though, so that helps bound the search space to a certain extent. Settling on a choice, and being able to remove a bunch of compatibility code for earlier kernels, will presumably result in a nice code cleanup as well.

to post comments

Systemd discusses its kernel-version needs

Posted Mar 30, 2022 22:05 UTC (Wed) by bluca (subscriber, #118303) [Link] (2 responses)

We did get started from somewhere, a baby step but better than nothing:

https://github.com/systemd/systemd/releases/tag/v251-rc1

"The minimum kernel version required has been bumped from 3.13 to 3.15,
and CLOCK_BOOTTIME is now assumed to always exist."

https://github.com/systemd/systemd/pull/22885

This looks like a great approach to me, as it clearly shows the cruft that we can get rid of. If nobody complains, rinse and repeat. I like it more than picking an arbitrary number from kernel.org.

Systemd discusses its kernel-version needs

Posted Mar 30, 2022 22:41 UTC (Wed) by dbnichol (subscriber, #39622) [Link]

I think that's the right approach, too. It's not systemd's job to be a proxy for the kernel's support policy. It's systemd's job to figure out what kernel features and interfaces it requires and communicate that to users. Then the balance is in how much compat code systemd wants to carry vs what systems it cuts off by raising the minimum kernel version. The kernel's support policy plays into the second part, but that's not a decision that the systemd developers are a part of.

Systemd discusses its kernel-version needs

Posted Mar 30, 2022 23:56 UTC (Wed) by flussence (guest, #85566) [Link]

They have a pretty good idea of the backcompat legacy support baggage they're carrying, and what kernel versions those things map to, right?

So document that top-level in a pretty table ordered by kernel release date, add a EOL roadmap chart explaining when people can expect each one to go away, and start pruning them one per release. They could even pre-write all the commits up front, add the diffstat in the roadmap to hammer home the point, and simply cherry-pick them into mainline when the deadline approaches - and nobody would be able to complain of surprises since it's all right there.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 3:50 UTC (Thu) by developer122 (guest, #152928) [Link] (5 responses)

Kernel devs may be loath to accept it, but people install brand-spanking-new userspaces on top of truly prehistoric kernels all. the. time. A chip may at best have a 2.6 vendor kernel, but the chip will be cheap/well known and the kernel will boot. That 2.6 setup is the only known working configuration. Either mainline was never ported to the chip (or doesn't work anymore), or you're working with binary-only modules that make key bits of hardware work, or some similar sordid arrangement.

The stability of the userspace API means that *most* applications will still work on it though, so whoever is supporting it (a product engineer pre-launch or someone managing a company's fleet of hardware in the wild) will dutifully patch together a workable userspace, one required binary at a time. Take the old kernel and modules, build most of a debian root, add a brand new version of smbd, an ancient version of busybox, a semi-recent version of perl, and so on until all your application needs are met. This is how 2.6 devices continue to enter the wild and continue to survive.

If you want to see a public version of this, look no further than the wifi router hacking scene out on the fringes of openWRT. Things like the Transcend wifi-enabled SD card that turned out to be a miniature linux box. Ancient kernel and kernel modules to run wifi and the custom SD interface, but someone built a brand new debian userspace for it and ran firefox over X11. A few people now use it "in production" as a means to get podcasts into dumb CD players and one guy built it a "motherboard."

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 7:24 UTC (Thu) by johannbg (guest, #65743) [Link] (2 responses)

Little to do with loathing to accept it but everything to do with having to or be expected to support it, which costs time of someone's lifespan.

Here's the thing people are like water, they always take the shortest path which in this case is pushing all the work that is required to make that work on the hands of upstreams as opposed to they themselves having to do the ( hard ) work, of making that work themselves.

Those that have realized this and value other peoples time are the once that are making the PR's to upstreams, putting in the required leg work themselves not the ones that are making the decisions downstream.

Upstream should just band together and put a stop to this vendor/downstream release madness that has been going on for decades, which can easily be done by simply not supporting it.

Form a union of upstream of sorts.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 9:43 UTC (Thu) by taladar (subscriber, #68407) [Link]

It is not always upstream. Sometimes upstream are also the unreasonable ones and distro maintainers have to patch their mess of old vendored dependencies to work with up-to-date maintained versions without security holes.

In general I agree though. Don't allow other people to push the maintenance burden for their choices onto you without your consent.

Systemd discusses its kernel-version needs

Posted Apr 1, 2022 21:52 UTC (Fri) by developer122 (guest, #152928) [Link]

The required support rarely makes it's way into mainline. This stuff is largely out of the sight of kernel devs.

Systemd discusses its kernel-version needs

Posted Apr 1, 2022 8:36 UTC (Fri) by matthias (subscriber, #94967) [Link]

> The stability of the userspace API means that *most* applications will still work on it though, ...

Actually, the stability of userspace API works exactly the other way round. Old userspace is still supported on new kernels. The kernel cannot make a promise that new userspace will work on old kernels. If new userspace still runs on old kernels then this is purely because of compatibility layers in userspace, e.g., libc falling back to older APIs if newer ones are not available.

And of course it really helps that most userspace code does not really care about most of the kernel interfaces. Much of the interaction with the kernel is passed through system libraries, which thankfully have compatibility layers to support old kernels. And many kernel interfaces are only relevant to the basic plumbing layer like systemd, udev, etc., that is to a very small set of programs.

Systemd discusses its kernel-version needs

Posted Apr 13, 2022 0:58 UTC (Wed) by mirabilos (subscriber, #84359) [Link]

Just don’t use systemd for these very valid applications.

Or, you know, well, don’t use it, at all. ;-)

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 3:55 UTC (Thu) by developer122 (guest, #152928) [Link] (3 responses)

> Windl wondered if some sort of conformance test suite for systemd made sense; "If the test suite succeeds, systemd might work; if it doesn't, manual steps are needed." Windl guessed that if manual steps were required, it might deter most people from going any further. But Poettering thought that was not the right path forward: "One goal here is to reduce our [maintenance] burden, not increase it. Another is to communicate clearly what we support and what we don't. Any such test suite collides with both these goals."

This sounds absurd. Has systemd limped along all this time without any testing suite? no continuous integration? nothing? Untested software will *only* raise your maintenance burden as you're forced to go back and fix bugs that slip through all the way to users. It's also itself a clear demonstration of what configurations are and are not supported. If you run the test suite on a 2.6 kernel and it fails because a key kernel feature is needed for functionality exercised by one of the tests, well that says it all now doesn't it?

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 5:58 UTC (Thu) by bookworm (subscriber, #114190) [Link]

Did you even bother to check?
I mean, it would've been quicker to have a look at the code than throwing your rant at the input form.

https://github.com/systemd/systemd/tree/main/src/test

https://github.com/systemd/systemd/tree/main/test

And there's no way that the systemd devs can build a full matrix of every kernel + config combination out there, not even the kernel has that.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 8:12 UTC (Thu) by bluca (subscriber, #118303) [Link]

https://coveralls.io/github/systemd/systemd

Feel free to send a PR to improve coverage

Systemd discusses its kernel-version needs

Posted Apr 6, 2022 4:26 UTC (Wed) by riking (guest, #95706) [Link]

This is specifically talking about a "conformance test suite" that you could run on a mystery meat kernel and get back a "is systemd supported?" decision.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 6:52 UTC (Thu) by johannbg (guest, #65743) [Link] (9 responses)

Components that make up the core/baseOS layer should only support mainline, stable and the latest long term supported linux kernel as well as the latest development and stable branches of each other, to keep the cost of development down.

Any vendor/distribution/users that is under the illusion that evolution can be frozen in time simply by slapping the word LTS on it or making some internal/community decision so they can either make profit or have somekind of a warm and fuzzy feeling of thinking things are actually working or are secure that way, should feel the weight of that by having to carry all that maintenance burden themselves, downstream.

Maybe that way they can snap back into the reality the world lives in as opposed to expect that upstream partakes in their illusion, their poor cocktail recipe of mixing and matching different upstream releases of the component that makes up the core/baseOS layer kool-aid, components which they then implement out of spec/standards and literal, virtual litter like trash all over the filesystem, just to make themselves feel "different" and call that [insert distro name] from one another, and upstream should maintain that warm and fuzzy feeling they get from doing that, for free, forever...

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 9:49 UTC (Thu) by taladar (subscriber, #68407) [Link] (8 responses)

It seems to be larger problem in society that many companies (and governments,...) like to pretend that the world is not moving around them, that the things that worked 5 or 10 years ago will continue working indefinitely and will be just as competitive as they were 5 or 10 years ago.

There is a stability theatre industry that feeds these delusions by selling them "long term support" and "backports", neglecting to mention that there is absolutely no possibility of having as much knowledge about the software as the original upstream developer team for every one of thousands of packages in a distribution and that an old version with a backported fix can also have new bugs and also needs testing, which often does not happen to nearly the same extent as it does with the latest upstream stable release or at least a reasonably up-to-date stable version of a Linux distro.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 20:29 UTC (Thu) by johannbg (guest, #65743) [Link] (5 responses)

Upstreams either directly ( through their employer ) or indirectly are also partaking in maintaining the stability theatre industry illusion that offers only long term moral support through phone or issue tracker at best.

Take systemd as an example [1] directly partaking in maintaining that illusion or even the kernel community ]2] that maintains multiple stable branches and multiple long term supported kernels which make literally no sense ( there is no support in (F)OSS project only ever best effort ).

Not only are upstreams that partake in that illusion shooting themselves in the foot by adding that maintainership to the overal cost of development for their project but they are also preventing themselves from progressing since they are getting distracted having to maintain their "stable" and or longterm branches.

Using systemd again as an example there does not even exist resources in the project to maintain that stable branch when the project itself has 177 PR's open and 1.6k open issues to go through on it's main branch. ( it takes at least 1700 hours to go through all that and a project that has to prioritize is a project that is starved of resources ).

A resource being spent in maintaining that stable branch could easily be freed up by simply stating they will only ever support the current and next release of the upstreams they are dependent on, any older release than that yeah sure it might work in conjunction with older kernels and what not but you are on your own if you decide to mix and match various releases of various components that systemd uses or is depentent upon because the project is *always* moving forward along with the other upstream it uses or is depentent upon.

1. https://github.com/systemd/systemd-stable
2. https://kernel.org/

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 20:45 UTC (Thu) by rahulsundaram (subscriber, #21946) [Link]

> Upstreams either directly ( through their employer ) or indirectly are also partaking in maintaining the stability theatre industry illusion that offers only long term moral support through phone or issue tracker at best

This is incorrect. There are many cases for example where bugs are identified and present in upstream releases, fixed upstream and backported to these branches, hotfixes supplied directly in response to customer issues, security issued identified and reported, documentation written in response to questions etc.

> even the kernel community ]2] that maintains multiple stable branches and multiple long term supported kernels which make literally no sense ( there is no support in (F)OSS project only ever best effort ).

There certainly is support in FOSS projects, just not commercial support which is only one of many support possibilities and since providing bug and security fixes is a form of support, LTS makes sense within that context.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 22:11 UTC (Thu) by excors (subscriber, #95769) [Link] (2 responses)

> a project that has to prioritize is a project that is starved of resources

That sounds totally untrue. A healthy well-resourced project will have lots of users, who inevitably submit many low-quality bug reports where it will take a large amount of effort to investigate a very rare or non-existent bug, and will also have an endless stream of ideas for new features and improvements with varying levels of effort and value. It's always important to prioritise, because resources are always finite and there will be lots of issues that are never worth fixing. A project that *doesn't* have to prioritise is one that has few users and the developers have run out of ideas, which would not be a good sign.

Also the effort/value cutoff point will vary over time, as the list of outstanding issues and available resources vary, or as you accumulate more reports of a particular bug or feature request, so it's not practical to immediately WONTFIX everything below a certain threshold - it's better to leave them open in the issue tracker because there's a chance they might become worthwhile later. As long as the issue tracker has a decent UI, it's fine to have thousands of open issues.

(There would be a serious resourcing problem if the issues don't even get triaged and nobody can tell whether they ought to be high priority or not. But systemd doesn't appear to have that problem - nearly all the recent still-open issues have comments and labels that indicate a developer has looked at them, and a large majority of the ones labeled "regression" or "bug" are closed, and half of the open issues are RFEs, so the first impression is that they're doing okay.)

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 23:09 UTC (Thu) by johannbg (guest, #65743) [Link] (1 responses)

> That sounds totally untrue.

Feel free to do your own research, own math yourself and your own resource measurements, but by my experience ( my own and work ) a single issue, phone call or disruption from a coworker, the cost of distraction to the developer is always atleast one hour on average.

With meetings these numbers go up significantly, which is why you never schedule meetings with developers unless it's strictly necessary and those meetings will have to be held earlies in the morning ( prior to them starting work ) or at the end of the workday ( after they finished their work ).

> There would be a serious resourcing problem if the issues don't even get triaged
That an issue has been labelled ( you can call that label what you want, triage, RFE whatever ) just means two things a) the issue has been looked at ( or labelled automatically like we do in our projects ) and b) the bug will be looked at again ( otherwise it would not still remain open ).

So an hour of an individuals time has been spent on the issue ( if manually looked at ) and atleast an hour will be spent on it again until it enters the closed state in it's lifecycle.

High number of unmerged PR's is a clear indicator that a project lacks resources since PR have value higher than issues ( PR's more often than not save time ).

Obviously I'm just pointing out the calculation for the upstream maintainers as in I'm not bringing into the equation the time spent by the reporters, documentation writers qa's etc. ( which often get's ignored,overlooked since most people seem to be fixated strictly on the developers ) since when you looking at projects resources you look at the available pool of contributors time. ( not just the developers but everyone partaking in the project ).

Systemd discusses its kernel-version needs

Posted Apr 5, 2022 7:11 UTC (Tue) by dtardon (subscriber, #53317) [Link]

> High number of unmerged PR's is a clear indicator that a project lacks resources since PR have value higher than issues ( PR's more often than not save time ).

It's not. It only indicates that the project has got a lot of contributors. It would only be a clear indicator that the project lacks resources if the number were increasing over time.

Using (again) systemd as an example, there are 174 open PRs as I write this. When I substract PRs labeled "dont-merge" (22), "reviewed/needs-rework" (68), "ci-fails/needs-rework" (17), "needs-rebase" (28), or "needs-discussion" (35), I get 41 PRs. Let's also consider that systemd is now in stabilisation phase for release of v251, which means that only important PRs are being merged; others, even if approved, are postponed to be merged after the release.

Looking from the other side, the Insights page tells me that 181 PRs have been merged during the past month.

Side note: the reaction time of systemd upstream is amazing compared to other upstreams I've interacted with...

Systemd discusses its kernel-version needs

Posted Apr 6, 2022 6:27 UTC (Wed) by dtardon (subscriber, #53317) [Link]

> Take systemd as an example [1] directly partaking in maintaining that illusion

Except that it is not, as you'd have discovered yourself if you had checked... The overwhelming majority of backports in systemd-stable goes to the latest two releases, i.e., those that are officially supported by systemd upstream. As for the remaining minority, I don't think anything has ever been backported to a release older than $current-3.

> A resource being spent in maintaining that stable branch could easily be freed up by simply stating they will only ever support the current and next release of the upstreams they are dependent on

No, nothing would be freed up. First, systemd-stable already primarily supports the latest two releases, so nothing would change. Second, it is a collaborative effort of distribution packagers. If it haven't existed, every distro packager would have to do backports separately, thus more effort would be spent overall...

(That said, I think there wouldn't be such a need for systemd-stable if systemd managed to do releases with greater frequency, say, once in 4-6 weeks. But then, preparing a release also requires some effort...)

Systemd discusses its kernel-version needs

Posted Apr 1, 2022 0:51 UTC (Fri) by Matt_G (subscriber, #112824) [Link] (1 responses)

5 Years is in my opinion a very short lifespan for operating system software. At the industrial plant I work at for some production critical systems there is very limited downtime available for updates patching etc. Which is why long term support is an important thing and also why my org pays vendors for commercial support contracts and that sort of thing.

It is not my job to worry about these things but if it was I'd be incredibly angry if I heard back from vendor that 5 years into deployed life the issue we experienced can't be fixed without updating to a newer kernel, but specialized hardware or software (like a commercial database or something else) won't support the newer kernel.

That is why long term support and backports are important and also why people pay companies good money to worry about this stuff for them.

Some of our systems and equipment have 25year + lifecycles. I don't work in IT side so I don't know all the specific details I do know there is plant equipment installed in the early 90's being controlled by Unix software/code written during that era. This software was highly likely to be designed to be run on commercial hardware which probably isn't supported by modern kernels.

Systemd discusses its kernel-version needs

Posted Apr 1, 2022 7:32 UTC (Fri) by johannbg (guest, #65743) [Link]

> That is why long term support and backports are important and also why people pay companies good money to worry about this stuff for them.

No what is important is for the software to work with each other with the latest release so you dont have to backport in the first place.

I suspect that you have never had to go through the actual support channel for these LTS offering service companies have you?

Or have to rely on their "developers" or rely on their incident response team after an breach.

If you did you would have realized you are just literally paying for moral support through a phone or an issue tracker.

Your company would be better off spending all that support/license fee and what not just hiring a psychiatrist, psychologist, or therapist for the relevant managers and IT department to talk to.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 7:14 UTC (Thu) by LtWorf (subscriber, #124958) [Link] (4 responses)

> Kroah-Hartman seemed surprised to hear that, since the kernel developers take user-space compatibility seriously. As readers of the LWN kernel page know, Linus Torvalds will quickly step in and remind kernel developers, sometimes rather harshly, that the kernel does not break user space.

A large asterisk is required there. They mean this for kernel API, but have no problem in continuously moving around stuff in /proc and /sys making configuration scripts require constant maintenance.

> The kernel-upgrade resistance is something that the stable maintainers (and others) have been struggling with for a long time

It happened to me quite recently that they released a kernel with a buggy intel driver so my machine took about 20 minutes to reach the login screen and was so slow to be completely impossible to use.

Had to revert to an older kernel.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 13:58 UTC (Thu) by corbet (editor, #1) [Link] (2 responses)

/proc and /sys are definitely considered part of the user-space ABI. Have you reported changes that caused you problems? That really isn't meant to happen.

Systemd discusses its kernel-version needs

Posted Apr 1, 2022 4:31 UTC (Fri) by wtarreau (subscriber, #51152) [Link] (1 responses)

But in fact it does happen because some old interfaces become deprecated, and others are bogus and eventually get fixed. For example I've been training my fingers to read the battery state on my laptop by doing "cat /proc/acpi/battery/BAT0/state" for more than a decade, and recently after upgrading to 5.10 I finally lost it, and now have to read several files in /sys.

Similarly some entries in /sys used to be misdesigned, with multiple values in the same entry, which is contrary to the /sys design rules. Some of them were finally fixed and split into multiple entries.

Some quickly written scripts that rely on these entries may occasionally break.

The bonding configuration interface changed over time to only use /sys. In the past you would pass arguments to modprobe then use ifenslave to configure the interface (but that wouldn't scale to multiple interfaces). Similarly code written for this eventually had to be changed.

There are definitely situations that cause visible breakage. It's not necessarily a regression but often the replacement for something wrong that cannot be supported anymore and that very limited stuff used to rely on. And these changes do require extra validation on upgrades that cannot make them 100% transparent. That's why LTS kernels are much appreciated for deployment in remote devices at customers'. They limit the risk of surprise on a supposedly harmless upgrade.

Systemd discusses its kernel-version needs

Posted Apr 2, 2022 1:14 UTC (Sat) by alison (subscriber, #63752) [Link]

Looks like /proc/acpi has a tortured history. 7d7ee958867a in 2013 says

commit 7d7ee958867ad3c9c829a36c56e870879e83391f
Author: Lan Tianyu
Date: Fri Oct 11 09:54:10 2013 +0800

ACPI: Remove CONFIG_ACPI_PROCFS_POWER and cm_sbsc.c

There is no user of cm_sbs.c and CONFIG_ACPI_PROCFS_POWER. So remove
them. Prepare for removing /proc/acpi

...That patch was reverted in 2014:

commit e2a7c3d7812369daae56f069eab2e8f3e548d231
Author: Lan Tianyu
Date: Sun May 4 11:07:24 2014 +0800

ACPI: Revert "ACPI: Remove CONFIG_ACPI_PROCFS_POWER and cm_sbsc.c"

The commit 1e2d9cd and 7d7ee95 remove ACPI Proc Battery
directory and breaks some old userspace tools. This patch
is to revert 7d7ee95.

...Then in 2018 the documentation was updated, but /proc/acpi continued onward:

commit 7e46b32bd58768b4823e767dc14d88ab59c6902e
Author: Randy Dunlap
Date: Fri Mar 2 16:55:54 2018 -0800

ACPI / Kconfig: Update ACPI_PROCFS_POWER help text

Fix grammar and punctuation (end sentences with a period) in the
Kconfig help text for ACPI_PROCFS_POWER.

I was looking at this since it appears to be going away (again,
some day) and I have a working script that uses this info to tell me
battery usage. I can update the script to use /sys/class/power_supply
(in theory) but the contents (with units) should be documented in
Documentation/ABI/ before /proc/acpi/battery/ is removed (IMO).</p>

..."Some day" turned out finally to be 5.10. The kernel does sometime "break userspace," but not usually in any haste.

Systemd discusses its kernel-version needs

Posted Apr 1, 2022 5:41 UTC (Fri) by donald.buczek (subscriber, #112892) [Link]

> > The kernel-upgrade resistance is something that the stable maintainers (and others) have been struggling with for a long time

> It happened to me quite recently that they released a kernel with a buggy intel driver so my machine took about 20 minutes to reach the login screen and was so slow to be completely impossible to use.

We've hit so many regressions updating to newer (stable!) releases, that a kernel update is considered a risk and generally avoided. Kernel developers focus on improvements instead of quality and stability. Bug fixes are interwoven with refactoring and new ideas, so that even in stable releases old bugs are just replaced with new bugs. Some subsystems have just a single active developer and there are not enough independent reviews. The kernel is moving to fast for the developers to sustain or improve quality.

Bugs from mainline are backported into stable releases or are just inherited when a new stable branch is forked. "Stable" doesn't help you much, because sooner or later you need to leave your stable branch anyway and the longer you wait with that, the further you leap and the more problems you have all at once.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 12:25 UTC (Thu) by jafd (subscriber, #129642) [Link] (4 responses)

Case in point to pinning a particular kernel. I have two AMD GPUs in my system, and it seems like kernels starting with 5.17 really, REALLY, don't like this setup. The current 5.18 tree doesn't boot at all, panicking as soon as the amdgpu module is inserted. So I'm pretty much contemplating running 5.16.x or even 5.15.x forever.

When I read about what's new in the amdgpu land, it's always churn to support the newest and shiniest GPUs. Regressions are almost never fixed. Old bugs are almost never being fixed. You have a WX5100? Be damned, you luddite! We're busy bolting on support for the latest RX and W series.

Systemd discusses its kernel-version needs

Posted Mar 31, 2022 19:51 UTC (Thu) by Ranguvar (subscriber, #56734) [Link] (2 responses)

Huh. I have a WX 5100 but haven't tried 5.17.x yet.

I use a second GPU but it's an NVIDIA card I use vfio-pci with.

I'll report back if any trouble tonight.

Systemd discusses its kernel-version needs

Posted Apr 1, 2022 0:53 UTC (Fri) by Ranguvar (subscriber, #56734) [Link] (1 responses)

Yep, 5.17.1 seems to break video output on boot.

I was able to login blind, but will add an LTS kernel to my boot options for future use.

Systemd discusses its kernel-version needs

Posted Apr 1, 2022 20:24 UTC (Fri) by jafd (subscriber, #129642) [Link]

Imagine all the fun if you have two AMD GPUs in your system and one is for VFIO purposes. 5.17 breaks it because it allocates its memory for simplefb no matter what you tell it. I also had a corruption where the passed through display ended up showing, in UEFI environment, garbled parts of the display manager running on the other GPU.

I would understand (but not forgive) it if the GPUs were identical, but they were not.

Systemd discusses its kernel-version needs

Posted Apr 11, 2022 22:23 UTC (Mon) by nix (subscriber, #2304) [Link]

Even when you have a well-supported one... GPUs are complex and change fast and the fallout of associated bugs seems to be almost continuous. I hit one every few weeks, breaking something or other.

The most recent... I upgraded from 5.16.14 to 5.16.17 on the 26th March. All 3D-using Steam games abruptly stopped rendering anything. I thought it just possible that might be Mesa-related, since I upgraded to staging/22.0 Mesa after restarting X the last time but long before rebooting. I know this means that if it breaks I get to keep the shattered bits, but I'm doing it anyway because Mesa 22.x finally did something to fix the game Big Pharma such that it produces readable pixmaps rather than chopped-up messes. (This particular game is really picky and has rendered only black screens or garbage for so many years on so many different platforms that I'd given up hope that it would ever work anywhere again, so I was very happy to try it and find it working at last.)

I upgraded again yesterday, to 5.16.19, without touching Mesa, and everything is working again! I speculated at the time of the reboot that this may be unanticipated fallout of the recent DMA security mess, but I just looked and amdgpu uses none of the affected calls, and none of the amdgpu code changed in suspicious-looking ways between the kernel releases in question.

Is this actually even a regression or not? I have no idea... maybe I was lucky, and it'll go wrong again on the next boot, or the next X restart, or the next Steam restart. I have no idea. I don't even know how to begin finding out.

I have several more bugs like this still live, with different games that don't work, most of which have never worked well, some of which have never worked at all. I should report some of them... not sure where to, though. Mesa's bugzilla? The drm layer's? These are all non-free and most haven't been updated in years, so quite possibly the game is just doing something evil and wrong and unfixable.

(I don't play games much. All this breakage means it's too much like work.)

Bisecting with old kernels

Posted Mar 31, 2022 13:54 UTC (Thu) by abatters (✭ supporter ✭, #6932) [Link] (15 responses)

An obvious use case for new userspace with old kernel is when bisecting a kernel problem, especially after a large jump in the kernel version. Bisecting is easier if the same userspace works for all the kernel versions you need to test, which has nothing to do with whether or not those kernels are still supported now. So I recommend to support what kernels you can, and drop support when they become too much of a drain on resources, not because they are no longer supported upstream.

Bisecting with old kernels

Posted Apr 1, 2022 14:55 UTC (Fri) by viro (subscriber, #7872) [Link] (14 responses)

That's an obvious reason to stay away from systemd. Among the first steps in my debugging workflow: check if I can reproduce it on setup where that thing is not installed. If I can - great; if not - see if it's an effect of systemd braindamage wrt mount subtree sharing setup; if not that either - curse certain cow-orkers (again and again and...) and resign myself to massive headache when trying to do bisections. If it's something more or less recently introduced - lucky me, if it's one of the times when the root of breakage goes back to 2.6.something - well... it happens.

Bisecting with old kernels

Posted Apr 11, 2022 11:48 UTC (Mon) by mezcalero (subscriber, #45103) [Link] (2 responses)

Al, somehow I sense that you just called me a "cow"? Can you be please be more explicit who specifically you are trying to insult as a cow here?

Is this the new, friendlier Linux kernel community I hear so much about? Because you are certainly cementing the kernel community's questionable reputation with comments like this.

Lennart

Bisecting with old kernels

Posted Apr 11, 2022 16:48 UTC (Mon) by mpr22 (subscriber, #60784) [Link]

"cow-orker" is a traditionally adopted deliberate misspelling of co-worker used in certain circles.

Bisecting with old kernels

Posted Apr 11, 2022 16:58 UTC (Mon) by rahulsundaram (subscriber, #21946) [Link]

>Al, somehow I sense that you just called me a "cow"? Can you be please be more explicit who specifically you are trying to insult as a cow here?

I don't think that is it. Refer to http://www.catb.org/jargon/html/C/cow-orker.html. There is an insulting reference to "braindamage" which is pretty problematic for people actually suffering from that but atleast in this case, it is directed towards a program, not a person.

Bisecting with old kernels

Posted Apr 12, 2022 16:41 UTC (Tue) by andy_shev (subscriber, #75870) [Link] (10 responses)

Unfortunately systemd did to Unix so many damages that the philosophical question “does the purpose justify the actions?” (it’s my rewording since I don’t know by heart its English variation) is quite actual. It works more or less now, but it may take ages to restore status quo.

Bisecting with old kernels

Posted Apr 12, 2022 16:47 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (9 responses)

> “does the purpose justify the actions?”

The phrase is "do the ends justify the means?". And FWIW, I don't see systemd as having done any of that (as someone who doesn't run a DE but instead puts pieces together).

Bisecting with old kernels

Posted Apr 13, 2022 9:04 UTC (Wed) by anselm (subscriber, #2796) [Link] (8 responses)

The phrase is "do the ends justify the means?". And FWIW, I don't see systemd as having done any of that (as someone who doesn't run a DE but instead puts pieces together).

At the risk of flogging a dead horse that has left the stable a long time ago, here are a few things that systemd gets right which the 1980s-vintage hodgepodge that seems to be so near and dear to some people doesn't:

Consistent configuration scheme across the whole setup (in the traditional approach, literally every service has its own configuration file format)
Clean separation between distribution-provided configuration and local changes, for enhanced maintainability (not addressed by the traditional setup)
Consistent way of starting subprocesses, with enhanced security options, resource limits, etc. available everywhere (in the traditional approach there are multiple ways of starting services, e.g, through /etc/inittab, /etc/init.d, /etc/inetd.conf, /etc/crontab (and friends), …, all different and with different options)
Consistent way of stopping subprocesses (via cgroups; in the traditional setup there is no telling which processes belong to which service, and unless the service itself cooperates it is very hard to clean up)
Detection and re-start of services that have crashed (a problem that the traditional setup mostly ignores unless you deploy non-standard extra tooling)
Reasonable dependency management between services (System-V init requires you to maintain your own topological order of services)
Management of services on behalf of individual users (not addressed at all by the traditional setup)
Convenient access to service status (only rudimentary support in the traditional setup)
Reasonable documentation
…

This is just the tip of the iceberg, and all of these make eminent sense even on systems that don't run a DE. So as far as I'm concerned the answer to the question “is that worth the trouble” is a resounding “absolutely”. You can of course run a Linux system without availing yourself of all these advantages (we've been doing that for decades after all) but, in 2022, why would you?

In my former life as a Linux instructor teaching advanced system administration I had to deal with various people who had picked up the popular notion at the time that the traditional 1980s setup was the bee's knees and systemd the spawn of the devil, but in the end they all admitted, however grudgingly, that systemd is really not a bad idea at all. It takes a little getting used to for people who have been brought up on the traditional approach, but it's worth it, and in my experience people who are entirely new to Linux system administration have a much easier time learning systemd than the traditional setup.

Bisecting with old kernels

Posted Apr 13, 2022 10:55 UTC (Wed) by atnot (guest, #124910) [Link] (6 responses)

I think that, while the initial resistance to systemd was understandable, at this point it is much more fitting to view resistance to it through a social lens than a technical one. The need for, if not systemd, a service manager much like it has been recognized universally enough that at this point, few people even think about it and the main arguments made against it are vague and selective appeals to nontangible things like tradition. This can be surprising when one considers the ferver with which these points are usually argued, as well as the curious lack of people working on serious alternatives to systemd.

A look at the social side can be very enlightening there. While on a technical level, systemd criticism has failed, on a social level it has been immensely succesful. The anti-systemd movement fourished on 4chan in the early 2010s, initially comprising mostly of reactionary and homophobic sentiments towards poettering, who was there seen as the figurehead of the dawn of a more inclusive foss community. This later mutated into being the core of the more moderate anti-systemd movement we know today, as well as a few auxillary ones like the letters in support of stallman. Symbolically rejecting systemd is table stakes for being accepted in these communities, and this is where I think one finds the true reason for the continued ferver.

The lack of compelling technical arguments or alternatives is because ultimately, what systemd symbolizes has become much more important than any of it's technical merits. This has only increased with perceived symbolic losses like code of conducts and discussions of inclusivity. And while of course, I am still painting with a very broad brush here as of 2022, I think those strokes will become more and more accurate as the movement continues to radicalize and shed less extreme members as it has over the last decade.

Bisecting with old kernels

Posted Apr 13, 2022 13:47 UTC (Wed) by anselm (subscriber, #2796) [Link] (1 responses)

Symbolically rejecting systemd is table stakes for being accepted in these communities, and this is where I think one finds the true reason for the continued ferver.

There's also the issue that some people who have grown up with the traditional system and are heavily invested in having learnt its weird ins and outs hate to see it replaced by something that is more straightforward and unified (let alone technically superior), because that undermines their status as the venerable “greybeards” who Know Things That Mere Mortals Cannot Fathom. People used to defer to their wisdom, and of course we can't have young upstarts like that Poettering taking over and changing everything just because they can. Who do these runts think they are? There ought to be a rite of passage. (Also there must be a conspiracy involved because it is obviously impossible that people would actually prefer something like systemd for its technical merits; it is clear that some nefarious means must have somehow been used to brainwash all the big Linux distributions into adopting it against the collective resistance of the “greybeards”.)

Right now we're seeing this once more with the “/ vs. /usr” debate, where the main counterargument apart from “Resist Poettering forever!” is basically “That's how we always did it and we don't want it to change”, with a side order of irrelevant mumbo-jumbo like “/ is faster than /usr”, even though the real reason (PDP-11 hard disks in the 1970s were too small) is very well known, other Unix vendors have already done the change, and today there are obvious technical advantages to merging the two. This is systemd all over again.

Bisecting with old kernels

Posted Apr 13, 2022 19:56 UTC (Wed) by pebolle (guest, #35204) [Link]

> Things That Mere Mortals Cannot Fathom

My pet theory is that things like Linux, slowly but certainly, getting easier to run on PC's, virtualization and cloud computing making running servers easy too, and - the big one - mobile phones making countless, almost magical things really easy for almost everyone made clear to many system administrators that their expertise wouldn't be needed for much longer.

It's hard to prove, but I do think the vocal systemd opponents were just shouting as hard as they could to make that future, where their expertise wasn't needed that much, please disappear.

anti-systemd is a broad church

Posted Apr 13, 2022 14:56 UTC (Wed) by karath (subscriber, #19025) [Link] (3 responses)

The above analysis feels mostly true. One sub-community that it appears not to cover are the greybeards* that started with Unix/BSD/Linux in the 70s/80s/90s that had very hard-won experience and expertise that was overturned by the simplicity that systemd introduced. While systemd could not make Linux 'sing' like they could, all of the major distros adopted it and they felt threatened with irrelevance. On the other hand, many others with similar deep expertise did make the jump to systemd, but that's not something that anyone really rants about.

What this means is that few are looking into systemd to find real technical issues. For example, would it make sense to spin out parts of systemd into separate repositories? My feeling is that this is self-censorship - if I conceptually support systemd and then I publicly criticise/question it, then could I be tagged as one of those rabid anti-systemders?

* I am an actual greybeard that started using and programming on Unix v6 in 1980 - but I went in another direction and really only got back to hobby usage of Linux in the past 10 years.

anti-systemd is a broad church

Posted Apr 13, 2022 17:53 UTC (Wed) by atnot (guest, #124910) [Link] (2 responses)

The reason I didn't bring up greybeards is because I don't think they actually played a very big role in what happened with systemd. I'm sure the greybeards grumbled plenty — that's what they do — but rarely does that result in more than obnoxious email theads, nevermind people getting death threats. What made systemd different is that a much broader audience took up the cause. I mean, if one has the misfortune of going to a website like reddit even today, one will still find no shortage of people who installed linux a year ago exhalting the great evils of systemd. Among these were a subcrowd for whom this was not merely about pid1, but the future of FOSS, the west, nay humanity itself (I am not joking), which would eventually remain as the core of the most vocal opposition.

That is not to say that the greybeards hold no responsibility for what happened. But I personally believe that if nobody but them had been stoking the flames, the whole controversy would have probably died down within five years at most.

anti-systemd is a broad church

Posted Apr 13, 2022 18:21 UTC (Wed) by atnot (guest, #124910) [Link]

...and that would, I think your right, certainly have put us in a more productive and fruitful environment for criticizing systemd than what we have today.

anti-systemd is a broad church

Posted Apr 14, 2022 12:11 UTC (Thu) by farnz (subscriber, #17727) [Link]

The other two components I see are that Poettering's "style" of software development is to consider the system as a whole, not just his package's piece of it and that Poettering is happy to do the non-coding work involved in keeping the contributor pool healthy.

The first one leads to issues like early PulseAudio releases, where PulseAudio refused to work around bugs in ALSA drivers; instead, Poettering insisted that the drivers should be fixed to correctly implement the ALSA API. Other users of the ALSA API were happy to work around bugs in drivers, and some drivers had simply implemented the API well enough to make common applications of the era work, but had bugs in implementations of things like snd_pcm_query_chmaps, snd_pcm_rewind and the mixer controls that would result in issues for a caller that expected the API contract to be met. Because PulseAudio was the only significant ALSA client at the time (JACK was the other, but users of JACK tended to own sound hardware with good drivers) that cared about the quality of your ALSA driver implementation, Poettering got an unfairly bad reputation, not helped by Ubuntu forcing people over to PulseAudio at a point where the PulseAudio developers (including Poettering) felt it wasn't ready for widespread adoption because they were still sorting through the driver bugs that needed to be fixed.

It also leads to people not liking systemd because it used features of the Linux kernel that had previously been ignored by most users. Poettering took (and takes) the view that if a feature is not meant to be used, then it should not be in the kernel, which has upset certain people who consider features systemd depends upon to be of sufficiently poor quality that they should be removed.

The second point, about looking after the contributor pool, means that Poettering puts up with flak aimed at him when it's in fact someone else's learning curve that's causing the errors; so usrmerge (which was mostly Kay Sievers and Harald Hoyer) gets blamed on Poettering because Poettering wrote it up clearly. When Kay Sievers made a bad decision in udev, Poettering got the flak because he explained the background to that decision, and why, given the context, it wasn't an obviously bad decision to make until the consequences became clear.

Bisecting with old kernels

Posted Apr 14, 2022 10:13 UTC (Thu) by johannbg (guest, #65743) [Link]

> In my former life as a Linux instructor teaching advanced system administration I had to deal with various people who had picked up the popular notion at the time that the traditional 1980s setup was the bee's knees and systemd the spawn of the devil, but in the end they all admitted, however grudgingly, that systemd is really not a bad idea at all. It takes a little getting used to for people who have been brought up on the traditional approach, but it's worth it, and in my experience people who are entirely new to Linux system administration have a much easier time learning systemd than the traditional setup.

It was apparent early on in systemd's development that getting people to adapt to the new concept would be one of the more difficult challenges to overcome, which is why I was a hard advocated for the 'rip the Band-Aid off' approach at that time ( like no backwards compatibility with the legacy service foo commands ) which in turn would have forced people to let go of the past and adapt the new concept and the commands that came with it. ( if you understood the concept you would understand the approach and also understand everything else that followed )

More cruder, more effective, less painful ( as in having people scream into the void once rather than prolong those screams for over a decade like it has ) approach.

Upgrade till it's supported, then... ?

Posted Apr 1, 2022 12:59 UTC (Fri) by kronat (subscriber, #117266) [Link] (2 responses)

Once upon a time, the only way to give old hardware a new life was to install GNU/Linux on top of it. I have an ASUS EEE 900 mini-laptop, and I would expect that installing ArchLinux on top of it, would be feasible (if I only had the time to do it...).

Now, what if in these years, something was added to the kernel sources making me unable to use the latest kernel? I would have to grab some old installer of Debian, let's say 4.0 that was there in 2008, and never touch anything on it, and hoping that everything will be usable (are most websites still working with, let's say, Firefox 3.0?). I consider this case as a regression: I would expect that most of the userspace would continue working with an older kernel (in the end, what I would need to install on it? Basic stuff -- like systemd (unfortunately), a browser, maybe a light DE like XFCE...).

In this light, having a look at the PR linked in one of the first posts is like sand in my eyes. A simple code that checks for a functionality (which doesn't seem to be bug-prone or a source of possible problems) is just removed, and the minimum supported version is bumped.

What are my options against this programmed obsolescence? Just accept it, and stick with proprietary vendors which may decide that every 2-3 years I need to buy a new laptop? I am without words.

Most of the time, the quality one receives is directly proportional to the money one is paying.

Upgrade till it's supported, then... ?

Posted Apr 1, 2022 20:26 UTC (Fri) by jafd (subscriber, #129642) [Link] (1 responses)

But the maintenance burden of those three lines is immense!

Upgrade till it's supported, then... ?

Posted May 1, 2022 13:31 UTC (Sun) by HelloWorld (guest, #56129) [Link]

> But the maintenance burden of those three lines is immense!
You can ridicule it all you want, but it's actually true. It's not that those three lines are complicated, it's that if you’re going to support old kernels, you also need to do the associated testing/QA work to make sure it keeps working, which is a huge pain. This becomes especially relevant when you try to significantly refactor parts of the code.

Besides, kronat’s point is largely hypothetical. Changes that break the kernel’s ability to run on old hardware just don’t happen all that often. Unless of course you insist on running proprietary drivers, in which case the problem isn’t with systemd but the hardware vendor.

Systemd discusses its kernel-version needs

Posted Apr 6, 2022 10:14 UTC (Wed) by zuki (subscriber, #41808) [Link]

To wrap this up for people who might be reading this article in the future:

In the end in systemd upstream we took the route of recommending newer kernel versions, but not removing any compat code, at least for now. We'll emit a warning when kernels <4.15 are used with systemd-251: https://github.com/systemd/systemd/commit/277f05872f8d5c0...

Systemd discusses its kernel-version needs

Posted Apr 7, 2022 14:24 UTC (Thu) by mrugiero (guest, #153040) [Link] (1 responses)

I still don't find the arguments for "maybe systemd will be running on ancient kernels" compelling.

In one scenario, you have a crappy ARM vendor. But then, crappy ARM vendors tend to just make a single code drop (if they drop the code at all) and you get a stale userspace as well. I don't see many attempting to update that. So, even in the unlikely case this vendor's image ships systemd, it's likely to be frozen as well.

In the other, you have some LTS distro that is expected to remain stable. Due or undue, systemd is famed as unstable. I can't see any distro maintainer being dumb enough to pretend to keep an old kernel for stability but then following the latest releases of pid 1, specifically in a version with so many features (and thus surface for stability). I'd dare say the same applies to other systemd services, but at the very least pid 1 I wouldn't expect to be updated.

Systemd discusses its kernel-version needs

Posted Apr 11, 2022 19:22 UTC (Mon) by bfields (subscriber, #19510) [Link]

In one scenario, you have a crappy ARM vendor. But then, crappy ARM vendors tend to just make a single code drop (if they drop the code at all) and you get a stale userspace as well. I don't see many attempting to update that.

Isn't that the kind of thing that community distros for phones or wireless routers sometimes try to do? Proprietary drivers and out-of-tree patches might leave them stuck with the original, but they might still have some hope of replacing userspace.

But I may be wrong, I'm not familiar enough with those projects to give specific examples.