Systemd discusses its kernel-version needs
A query regarding the possibility of dropping support for older kernels in systemd led to some discussion on the systemd-devel mailing list recently. As might be guessed, exactly which kernel would be the minimum supported, what kernel features systemd is using, and when those kernel features became available, were all part of that conversation. A component like systemd that is closely tied to the kernel, and the interfaces different versions provide, has a number of different factors to consider when making a decision of this sort.
Zbigniew Jędrzejewski-Szmek started things off by asking
if changing the minimum required kernel version for systemd to 4.4
would cause problems for anyone. He said that if it did, "please
substantiate why you are running new
systemd with such old kernels
". Currently, systemd minimally
requires Linux 3.15 or later, as noted in its README file.
Which version?
But Greg Kroah-Hartman was quick to
point out that the stable-kernel team
recently stopped
supporting 4.4, so the 4.9 kernel is the oldest
version that is still getting stable updates. He suggested using that as
the baseline instead. Jędrzejewski-Szmek noted that the Civil
Infrastructure Platform (CIP) will continue to use 4.4, but that
Debian 9.x ("stretch") uses 4.9 and has its end of life in
June 2022. "Of course we'd like to move to 4.19, but we don't
want to disrupt distros
that use older kernels. Is ≤4.19 really unused?
"
Michael Biebl filled in more details for the Debian picture. Newer systemd versions are being backported to Debian 11.x ("bullseye"), which uses the 5.10 kernel, but not to earlier Debian releases. He did note that downstream Debian derivatives (such as Raspbian) might have other requirements, as might those who build their own kernels. Neal Gompa said that there are active efforts to bring newer systemd releases to CentOS Stream 8, which is based on 4.18 plus a bunch of kernel-feature backports. He would like to see that continue to be supported.
Luca Boccassi said that he would like to
see the minimum version only go as far as 4.4 since "what's on
kernel.org is not really that
important, as real usage is downstream from there anyway
". The
important thing is to identify what's needed for systemd core
compatibility—all of that is available in 4.4, he said. He noted that support
for version 2
of control groups (cgroupv2) came in 4.1; being
able to count on cgroupv2 support would be helpful. Control groups are a
resource-management mechanism that is used extensively by systemd.
Not surprisingly, Kroah-Hartman had a rather
different take on that; "anyone still using 4.4.y today has an
insecure and unsupported system
". He does not want to encourage
anyone to use that version. While CIP is supporting 4.4
until 2027, it is using that kernel "in a very limited set of
use cases and configurations
". He is somewhat skeptical that CIP
will be able to keep that kernel "alive and
secure
"; for anyone else that is still using 4.4, "send them to
me please
".
Boccassi disagreed with that view;
kernel-version choices are often out of the hands of the user because they
have to use what their distribution or vendor provides. If the kernel
version for systemd changes, he expects that the project will hear from
"some poor soul stuck on 3.x because of $crappy_arm_vendor with no
way
to move on from there
". Even so, moving to 4.4 has some
advantages:
Jumping forward from 3.13 to 4.4 as the baseline, allowing to take cgroupsv2 for granted, seems like a good starting point to me. There's very obvious and public evidence of that being used in the wild. We can start to drop a bunch of backward-compat cruft, wait and see who complains, and if nobody does we can re-evaluate again in a couple of years.
But Kroah-Hartman thought it is unlikely that
people stuck on 3.x kernels are updating to the latest systemd
version; "if
they are, they need to get $crappy_arm_vendor to do the work for them as
they PAID for that support and work already
".
Ulrich Windl said there can be value in
sticking with the same kernel over time, though, because "new
features intended to improve things sometimes actually make things
worse
". But in choosing which kernels to support for systemd, that
is not really the issue, Kroah-Hartman said:
Do you want to run a kernel with known security problems, or one with "unknown potential problems." The latter is always the case, so please don't pick the known-insecure one, that's just foolish.
Boccassi was not impressed with that
argument, noting that "'security problems' are a dime a dozen, as
they say
" and complaining that the stable kernels often have regressions of
various sorts. He made a rather surprising claim about the nature of those
regressions, however:
Upgrading major kernel version is like rolling a dice, you never know what kind of extremely expensive and time consuming rabbit hole you'll be dragged into because the kernel plays fast and loose with its userspace interfaces, and each and every time there's a chance one might end up having to do major reworks to deal with it.
Kroah-Hartman seemed surprised to hear that, since the kernel developers take user-space compatibility seriously. As readers of the LWN kernel page know, Linus Torvalds will quickly step in and remind kernel developers, sometimes rather harshly, that the kernel does not break user space. There are often complaints about regressions in the stable tree, but it does not typically involve breaking user-space interfaces. Kroah-Hartman said that he was interested in hearing any such reports and that the proper place to report them was the kernel's regressions mailing list.
That was something of a side conversation to the main point of the discussion, but the resistance to upgrading to new kernel versions is real; it is somewhat unclear why that resistance would not also extend to systemd versions, however. The kernel-upgrade resistance is something that the stable maintainers (and others) have been struggling with for a long time. Those who plan to stay on older releases, or not upgrade series that are still supported, are playing a dangerous game. Kroah-Hartman clearly sees the choice of the minimum supported kernel version as a means to help spread that message further, though it is unclear how successful it would be.
Control group evolution
Back to systemd itself, Lennart Poettering said that there had been an evolution in cgroupv2 support over time, so narrowing down exactly what systemd needs or wants in that support is needed:
Note that "cgroupv2 exists" and "cgroupv2 works well" are two distinct things. Initially too few controllers supported cgroupv2 for cgroupv2 to be actually useful.What I am trying to say is that it would actually help us a lot if we'd not just be able to take [cgroupv2] for granted but to take a reasonably complete cgroupv2 for granted.
Boccassi agreed that information would be useful, especially which cgroupv2 controllers are needed and when they were added. As Jędrzejewski-Szmek pointed out, the README does list some features that systemd can use and what kernel is needed for them:
Linux kernel >= 4.2 for unified cgroup hierarchy support
Linux kernel >= 4.10 for cgroup-bpf egress and ingress hooks
Linux kernel >= 4.15 for cgroup-bpf device hook
Linux kernel >= 4.17 for cgroup-bpf socket address hooks
In this light, 4.19 is better than 4.4 or 4.9 ;)
Given that, Kroah-Hartman was in favor of
choosing 4.19: "I strongly doubt that any distro that is using older
kernels would ever be willing to update systemd.
" Windl added a data point, saying that the
most recent service pack of SUSE Enterprise Linux 12 (SLES12 SP5) is
using 4.12, but he agreed that it was not likely it would be
moving to newer systemd versions.
The list of kernel versions in the README is mostly concerned with
various BPF-related features, Boccassi said. But systemd should still run without
support for
BPF; "There's plenty of use cases
that disable it entirely (in the sense, things shouldn't fall apart if
we run on a non-bpf kernel and there are no bpf options configured in
any unit). I have one of them.
" BPF-related features should remain
optional in systemd forever, he said.
Poettering took a look through the systemd man pages to get a sense for what kernel features are being used and when they were added. He noted that cgroupv2 support was not really viable for systemd until 4.15, since before that it lacked the CPU controller, which is used to partition the CPU use between different groups. Beyond that:
some other interesting milestones:This is just some quick search through man pages. There might be a lot of other stuff that would make sense for us to be able to rely on.
- kcmp → 3.5
- renameat2 on all relevant file systems → 4.0
- pids controller in cgroupv1 → 4.3
- pids controller in cgroupv2 → 4.5
- cgroup namespaces → 4.6
- statx → 4.11
- pidfd → 5.3
Windl wondered if some sort of conformance
test suite for systemd made sense; "If the test suite succeeds, systemd
might work; if it doesn't, manual steps are needed.
" Windl guessed
that if manual steps were required, it might deter most people from going
any further. But Poettering thought that
was not the right path forward:
"One goal here is to reduce our [maintenance] burden, not increase
it. Another is to communicate clearly what we support and what we
don't. Any such test suite collides with both these goals.
"
The conversation wound down, seemingly without any final conclusions. While support for 4.4 might send the wrong message, at least from the stable-kernel maintainers' point of view, it may make sense if the CIP project has plans to upgrade its systemd along the way. Jędrzejewski-Szmek opened an issue on the project's GitLab site to try to find out. It does seem reasonable to at least conjecture that distributions (and others) that are unwilling to upgrade their kernel might also be leery of upgrading a major component like systemd.
Since 4.4 lacks the cgroupv2 support that is desirable, 4.9 or even 4.19 seem like the next logical possibilities, though 4.9 reaches end of life in January 2023. In addition, 4.9 is lacking in the cgroupv2 department as well. The 4.19 kernel will be supported for nearly two years longer than 4.9, until December 2024, and has most of the desired features. It may well be where the project decides to land.
In the end, the systemd project will need to find a good balance between the features it needs from the kernel and the kernel versions that its users are running—if they still plan to upgrade to newer systemd versions anyway. Most distributions will likely shy away from kernels that have fallen outside of the stable-kernel support windows, though, so that helps bound the search space to a certain extent. Settling on a choice, and being able to remove a bunch of compatibility code for earlier kernels, will presumably result in a nice code cleanup as well.
Posted Mar 30, 2022 22:05 UTC (Wed)
by bluca (subscriber, #118303)
[Link] (2 responses)
https://github.com/systemd/systemd/releases/tag/v251-rc1
"The minimum kernel version required has been bumped from 3.13 to 3.15,
https://github.com/systemd/systemd/pull/22885
This looks like a great approach to me, as it clearly shows the cruft that we can get rid of. If nobody complains, rinse and repeat. I like it more than picking an arbitrary number from kernel.org.
Posted Mar 30, 2022 22:41 UTC (Wed)
by dbnichol (subscriber, #39622)
[Link]
Posted Mar 30, 2022 23:56 UTC (Wed)
by flussence (guest, #85566)
[Link]
So document that top-level in a pretty table ordered by kernel release date, add a EOL roadmap chart explaining when people can expect each one to go away, and start pruning them one per release. They could even pre-write all the commits up front, add the diffstat in the roadmap to hammer home the point, and simply cherry-pick them into mainline when the deadline approaches - and nobody would be able to complain of surprises since it's all right there.
Posted Mar 31, 2022 3:50 UTC (Thu)
by developer122 (guest, #152928)
[Link] (5 responses)
The stability of the userspace API means that *most* applications will still work on it though, so whoever is supporting it (a product engineer pre-launch or someone managing a company's fleet of hardware in the wild) will dutifully patch together a workable userspace, one required binary at a time. Take the old kernel and modules, build most of a debian root, add a brand new version of smbd, an ancient version of busybox, a semi-recent version of perl, and so on until all your application needs are met. This is how 2.6 devices continue to enter the wild and continue to survive.
If you want to see a public version of this, look no further than the wifi router hacking scene out on the fringes of openWRT. Things like the Transcend wifi-enabled SD card that turned out to be a miniature linux box. Ancient kernel and kernel modules to run wifi and the custom SD interface, but someone built a brand new debian userspace for it and ran firefox over X11. A few people now use it "in production" as a means to get podcasts into dumb CD players and one guy built it a "motherboard."
Posted Mar 31, 2022 7:24 UTC (Thu)
by johannbg (guest, #65743)
[Link] (2 responses)
Here's the thing people are like water, they always take the shortest path which in this case is pushing all the work that is required to make that work on the hands of upstreams as opposed to they themselves having to do the ( hard ) work, of making that work themselves.
Those that have realized this and value other peoples time are the once that are making the PR's to upstreams, putting in the required leg work themselves not the ones that are making the decisions downstream.
Upstream should just band together and put a stop to this vendor/downstream release madness that has been going on for decades, which can easily be done by simply not supporting it.
Form a union of upstream of sorts.
Posted Mar 31, 2022 9:43 UTC (Thu)
by taladar (subscriber, #68407)
[Link]
In general I agree though. Don't allow other people to push the maintenance burden for their choices onto you without your consent.
Posted Apr 1, 2022 21:52 UTC (Fri)
by developer122 (guest, #152928)
[Link]
Posted Apr 1, 2022 8:36 UTC (Fri)
by matthias (subscriber, #94967)
[Link]
Actually, the stability of userspace API works exactly the other way round. Old userspace is still supported on new kernels. The kernel cannot make a promise that new userspace will work on old kernels. If new userspace still runs on old kernels then this is purely because of compatibility layers in userspace, e.g., libc falling back to older APIs if newer ones are not available.
And of course it really helps that most userspace code does not really care about most of the kernel interfaces. Much of the interaction with the kernel is passed through system libraries, which thankfully have compatibility layers to support old kernels. And many kernel interfaces are only relevant to the basic plumbing layer like systemd, udev, etc., that is to a very small set of programs.
Posted Apr 13, 2022 0:58 UTC (Wed)
by mirabilos (subscriber, #84359)
[Link]
Or, you know, well, don’t use it, at all. ;-)
Posted Mar 31, 2022 3:55 UTC (Thu)
by developer122 (guest, #152928)
[Link] (3 responses)
This sounds absurd. Has systemd limped along all this time without any testing suite? no continuous integration? nothing? Untested software will *only* raise your maintenance burden as you're forced to go back and fix bugs that slip through all the way to users. It's also itself a clear demonstration of what configurations are and are not supported. If you run the test suite on a 2.6 kernel and it fails because a key kernel feature is needed for functionality exercised by one of the tests, well that says it all now doesn't it?
Posted Mar 31, 2022 5:58 UTC (Thu)
by bookworm (subscriber, #114190)
[Link]
https://github.com/systemd/systemd/tree/main/src/test
https://github.com/systemd/systemd/tree/main/test
And there's no way that the systemd devs can build a full matrix of every kernel + config combination out there, not even the kernel has that.
Posted Mar 31, 2022 8:12 UTC (Thu)
by bluca (subscriber, #118303)
[Link]
Posted Apr 6, 2022 4:26 UTC (Wed)
by riking (guest, #95706)
[Link]
Posted Mar 31, 2022 6:52 UTC (Thu)
by johannbg (guest, #65743)
[Link] (9 responses)
Any vendor/distribution/users that is under the illusion that evolution can be frozen in time simply by slapping the word LTS on it or making some internal/community decision so they can either make profit or have somekind of a warm and fuzzy feeling of thinking things are actually working or are secure that way, should feel the weight of that by having to carry all that maintenance burden themselves, downstream.
Maybe that way they can snap back into the reality the world lives in as opposed to expect that upstream partakes in their illusion, their poor cocktail recipe of mixing and matching different upstream releases of the component that makes up the core/baseOS layer kool-aid, components which they then implement out of spec/standards and literal, virtual litter like trash all over the filesystem, just to make themselves feel "different" and call that [insert distro name] from one another, and upstream should maintain that warm and fuzzy feeling they get from doing that, for free, forever...
Posted Mar 31, 2022 9:49 UTC (Thu)
by taladar (subscriber, #68407)
[Link] (8 responses)
There is a stability theatre industry that feeds these delusions by selling them "long term support" and "backports", neglecting to mention that there is absolutely no possibility of having as much knowledge about the software as the original upstream developer team for every one of thousands of packages in a distribution and that an old version with a backported fix can also have new bugs and also needs testing, which often does not happen to nearly the same extent as it does with the latest upstream stable release or at least a reasonably up-to-date stable version of a Linux distro.
Posted Mar 31, 2022 20:29 UTC (Thu)
by johannbg (guest, #65743)
[Link] (5 responses)
Take systemd as an example [1] directly partaking in maintaining that illusion or even the kernel community ]2] that maintains multiple stable branches and multiple long term supported kernels which make literally no sense ( there is no support in (F)OSS project only ever best effort ).
Not only are upstreams that partake in that illusion shooting themselves in the foot by adding that maintainership to the overal cost of development for their project but they are also preventing themselves from progressing since they are getting distracted having to maintain their "stable" and or longterm branches.
Using systemd again as an example there does not even exist resources in the project to maintain that stable branch when the project itself has 177 PR's open and 1.6k open issues to go through on it's main branch. ( it takes at least 1700 hours to go through all that and a project that has to prioritize is a project that is starved of resources ).
A resource being spent in maintaining that stable branch could easily be freed up by simply stating they will only ever support the current and next release of the upstreams they are dependent on, any older release than that yeah sure it might work in conjunction with older kernels and what not but you are on your own if you decide to mix and match various releases of various components that systemd uses or is depentent upon because the project is *always* moving forward along with the other upstream it uses or is depentent upon.
1. https://github.com/systemd/systemd-stable
Posted Mar 31, 2022 20:45 UTC (Thu)
by rahulsundaram (subscriber, #21946)
[Link]
This is incorrect. There are many cases for example where bugs are identified and present in upstream releases, fixed upstream and backported to these branches, hotfixes supplied directly in response to customer issues, security issued identified and reported, documentation written in response to questions etc.
> even the kernel community ]2] that maintains multiple stable branches and multiple long term supported kernels which make literally no sense ( there is no support in (F)OSS project only ever best effort ).
There certainly is support in FOSS projects, just not commercial support which is only one of many support possibilities and since providing bug and security fixes is a form of support, LTS makes sense within that context.
Posted Mar 31, 2022 22:11 UTC (Thu)
by excors (subscriber, #95769)
[Link] (2 responses)
That sounds totally untrue. A healthy well-resourced project will have lots of users, who inevitably submit many low-quality bug reports where it will take a large amount of effort to investigate a very rare or non-existent bug, and will also have an endless stream of ideas for new features and improvements with varying levels of effort and value. It's always important to prioritise, because resources are always finite and there will be lots of issues that are never worth fixing. A project that *doesn't* have to prioritise is one that has few users and the developers have run out of ideas, which would not be a good sign.
Also the effort/value cutoff point will vary over time, as the list of outstanding issues and available resources vary, or as you accumulate more reports of a particular bug or feature request, so it's not practical to immediately WONTFIX everything below a certain threshold - it's better to leave them open in the issue tracker because there's a chance they might become worthwhile later. As long as the issue tracker has a decent UI, it's fine to have thousands of open issues.
(There would be a serious resourcing problem if the issues don't even get triaged and nobody can tell whether they ought to be high priority or not. But systemd doesn't appear to have that problem - nearly all the recent still-open issues have comments and labels that indicate a developer has looked at them, and a large majority of the ones labeled "regression" or "bug" are closed, and half of the open issues are RFEs, so the first impression is that they're doing okay.)
Posted Mar 31, 2022 23:09 UTC (Thu)
by johannbg (guest, #65743)
[Link] (1 responses)
Feel free to do your own research, own math yourself and your own resource measurements, but by my experience ( my own and work ) a single issue, phone call or disruption from a coworker, the cost of distraction to the developer is always atleast one hour on average.
With meetings these numbers go up significantly, which is why you never schedule meetings with developers unless it's strictly necessary and those meetings will have to be held earlies in the morning ( prior to them starting work ) or at the end of the workday ( after they finished their work ).
> There would be a serious resourcing problem if the issues don't even get triaged
So an hour of an individuals time has been spent on the issue ( if manually looked at ) and atleast an hour will be spent on it again until it enters the closed state in it's lifecycle.
High number of unmerged PR's is a clear indicator that a project lacks resources since PR have value higher than issues ( PR's more often than not save time ).
Obviously I'm just pointing out the calculation for the upstream maintainers as in I'm not bringing into the equation the time spent by the reporters, documentation writers qa's etc. ( which often get's ignored,overlooked since most people seem to be fixated strictly on the developers ) since when you looking at projects resources you look at the available pool of contributors time. ( not just the developers but everyone partaking in the project ).
Posted Apr 5, 2022 7:11 UTC (Tue)
by dtardon (subscriber, #53317)
[Link]
It's not. It only indicates that the project has got a lot of contributors. It would only be a clear indicator that the project lacks resources if the number were increasing over time.
Using (again) systemd as an example, there are 174 open PRs as I write this. When I substract PRs labeled "dont-merge" (22), "reviewed/needs-rework" (68), "ci-fails/needs-rework" (17), "needs-rebase" (28), or "needs-discussion" (35), I get 41 PRs. Let's also consider that systemd is now in stabilisation phase for release of v251, which means that only important PRs are being merged; others, even if approved, are postponed to be merged after the release.
Looking from the other side, the Insights page tells me that 181 PRs have been merged during the past month.
Side note: the reaction time of systemd upstream is amazing compared to other upstreams I've interacted with...
Posted Apr 6, 2022 6:27 UTC (Wed)
by dtardon (subscriber, #53317)
[Link]
Except that it is not, as you'd have discovered yourself if you had checked... The overwhelming majority of backports in systemd-stable goes to the latest two releases, i.e., those that are officially supported by systemd upstream. As for the remaining minority, I don't think anything has ever been backported to a release older than $current-3.
> A resource being spent in maintaining that stable branch could easily be freed up by simply stating they will only ever support the current and next release of the upstreams they are dependent on
No, nothing would be freed up. First, systemd-stable already primarily supports the latest two releases, so nothing would change. Second, it is a collaborative effort of distribution packagers. If it haven't existed, every distro packager would have to do backports separately, thus more effort would be spent overall...
(That said, I think there wouldn't be such a need for systemd-stable if systemd managed to do releases with greater frequency, say, once in 4-6 weeks. But then, preparing a release also requires some effort...)
Posted Apr 1, 2022 0:51 UTC (Fri)
by Matt_G (subscriber, #112824)
[Link] (1 responses)
It is not my job to worry about these things but if it was I'd be incredibly angry if I heard back from vendor that 5 years into deployed life the issue we experienced can't be fixed without updating to a newer kernel, but specialized hardware or software (like a commercial database or something else) won't support the newer kernel.
That is why long term support and backports are important and also why people pay companies good money to worry about this stuff for them.
Some of our systems and equipment have 25year + lifecycles. I don't work in IT side so I don't know all the specific details I do know there is plant equipment installed in the early 90's being controlled by Unix software/code written during that era. This software was highly likely to be designed to be run on commercial hardware which probably isn't supported by modern kernels.
Posted Apr 1, 2022 7:32 UTC (Fri)
by johannbg (guest, #65743)
[Link]
No what is important is for the software to work with each other with the latest release so you dont have to backport in the first place.
I suspect that you have never had to go through the actual support channel for these LTS offering service companies have you?
Or have to rely on their "developers" or rely on their incident response team after an breach.
If you did you would have realized you are just literally paying for moral support through a phone or an issue tracker.
Your company would be better off spending all that support/license fee and what not just hiring a psychiatrist, psychologist, or therapist for the relevant managers and IT department to talk to.
Posted Mar 31, 2022 7:14 UTC (Thu)
by LtWorf (subscriber, #124958)
[Link] (4 responses)
A large asterisk is required there. They mean this for kernel API, but have no problem in continuously moving around stuff in /proc and /sys making configuration scripts require constant maintenance.
> The kernel-upgrade resistance is something that the stable maintainers (and others) have been struggling with for a long time
It happened to me quite recently that they released a kernel with a buggy intel driver so my machine took about 20 minutes to reach the login screen and was so slow to be completely impossible to use.
Had to revert to an older kernel.
Posted Mar 31, 2022 13:58 UTC (Thu)
by corbet (editor, #1)
[Link] (2 responses)
Posted Apr 1, 2022 4:31 UTC (Fri)
by wtarreau (subscriber, #51152)
[Link] (1 responses)
Similarly some entries in /sys used to be misdesigned, with multiple values in the same entry, which is contrary to the /sys design rules. Some of them were finally fixed and split into multiple entries.
Some quickly written scripts that rely on these entries may occasionally break.
The bonding configuration interface changed over time to only use /sys. In the past you would pass arguments to modprobe then use ifenslave to configure the interface (but that wouldn't scale to multiple interfaces). Similarly code written for this eventually had to be changed.
There are definitely situations that cause visible breakage. It's not necessarily a regression but often the replacement for something wrong that cannot be supported anymore and that very limited stuff used to rely on. And these changes do require extra validation on upgrades that cannot make them 100% transparent. That's why LTS kernels are much appreciated for deployment in remote devices at customers'. They limit the risk of surprise on a supposedly harmless upgrade.
Posted Apr 2, 2022 1:14 UTC (Sat)
by alison (subscriber, #63752)
[Link]
commit 7d7ee958867ad3c9c829a36c56e870879e83391f
ACPI: Remove CONFIG_ACPI_PROCFS_POWER and cm_sbsc.c
...That patch was reverted in 2014:
commit e2a7c3d7812369daae56f069eab2e8f3e548d231
ACPI: Revert "ACPI: Remove CONFIG_ACPI_PROCFS_POWER and cm_sbsc.c"
...Then in 2018 the documentation was updated, but /proc/acpi continued onward:
commit 7e46b32bd58768b4823e767dc14d88ab59c6902e
ACPI / Kconfig: Update ACPI_PROCFS_POWER help text
..."Some day" turned out finally to be 5.10. The kernel does sometime "break userspace," but not usually in any haste.
Posted Apr 1, 2022 5:41 UTC (Fri)
by donald.buczek (subscriber, #112892)
[Link]
> It happened to me quite recently that they released a kernel with a buggy intel driver so my machine took about 20 minutes to reach the login screen and was so slow to be completely impossible to use.
We've hit so many regressions updating to newer (stable!) releases, that a kernel update is considered a risk and generally avoided. Kernel developers focus on improvements instead of quality and stability. Bug fixes are interwoven with refactoring and new ideas, so that even in stable releases old bugs are just replaced with new bugs. Some subsystems have just a single active developer and there are not enough independent reviews. The kernel is moving to fast for the developers to sustain or improve quality.
Bugs from mainline are backported into stable releases or are just inherited when a new stable branch is forked. "Stable" doesn't help you much, because sooner or later you need to leave your stable branch anyway and the longer you wait with that, the further you leap and the more problems you have all at once.
Posted Mar 31, 2022 12:25 UTC (Thu)
by jafd (subscriber, #129642)
[Link] (4 responses)
When I read about what's new in the amdgpu land, it's always churn to support the newest and shiniest GPUs. Regressions are almost never fixed. Old bugs are almost never being fixed. You have a WX5100? Be damned, you luddite! We're busy bolting on support for the latest RX and W series.
Posted Mar 31, 2022 19:51 UTC (Thu)
by Ranguvar (subscriber, #56734)
[Link] (2 responses)
I use a second GPU but it's an NVIDIA card I use vfio-pci with.
I'll report back if any trouble tonight.
Posted Apr 1, 2022 0:53 UTC (Fri)
by Ranguvar (subscriber, #56734)
[Link] (1 responses)
I was able to login blind, but will add an LTS kernel to my boot options for future use.
Posted Apr 1, 2022 20:24 UTC (Fri)
by jafd (subscriber, #129642)
[Link]
I would understand (but not forgive) it if the GPUs were identical, but they were not.
Posted Apr 11, 2022 22:23 UTC (Mon)
by nix (subscriber, #2304)
[Link]
The most recent... I upgraded from 5.16.14 to 5.16.17 on the 26th March. All 3D-using Steam games abruptly stopped rendering anything. I thought it just possible that might be Mesa-related, since I upgraded to staging/22.0 Mesa after restarting X the last time but long before rebooting. I know this means that if it breaks I get to keep the shattered bits, but I'm doing it anyway because Mesa 22.x finally did something to fix the game Big Pharma such that it produces readable pixmaps rather than chopped-up messes. (This particular game is really picky and has rendered only black screens or garbage for so many years on so many different platforms that I'd given up hope that it would ever work anywhere again, so I was very happy to try it and find it working at last.)
I upgraded again yesterday, to 5.16.19, without touching Mesa, and everything is working again! I speculated at the time of the reboot that this may be unanticipated fallout of the recent DMA security mess, but I just looked and amdgpu uses none of the affected calls, and none of the amdgpu code changed in suspicious-looking ways between the kernel releases in question.
Is this actually even a regression or not? I have no idea... maybe I was lucky, and it'll go wrong again on the next boot, or the next X restart, or the next Steam restart. I have no idea. I don't even know how to begin finding out.
I have several more bugs like this still live, with different games that don't work, most of which have never worked well, some of which have never worked at all. I should report some of them... not sure where to, though. Mesa's bugzilla? The drm layer's? These are all non-free and most haven't been updated in years, so quite possibly the game is just doing something evil and wrong and unfixable.
(I don't play games much. All this breakage means it's too much like work.)
Posted Mar 31, 2022 13:54 UTC (Thu)
by abatters (✭ supporter ✭, #6932)
[Link] (15 responses)
Posted Apr 1, 2022 14:55 UTC (Fri)
by viro (subscriber, #7872)
[Link] (14 responses)
Posted Apr 11, 2022 11:48 UTC (Mon)
by mezcalero (subscriber, #45103)
[Link] (2 responses)
Is this the new, friendlier Linux kernel community I hear so much about? Because you are certainly cementing the kernel community's questionable reputation with comments like this.
Lennart
Posted Apr 11, 2022 16:48 UTC (Mon)
by mpr22 (subscriber, #60784)
[Link]
Posted Apr 11, 2022 16:58 UTC (Mon)
by rahulsundaram (subscriber, #21946)
[Link]
I don't think that is it. Refer to http://www.catb.org/jargon/html/C/cow-orker.html. There is an insulting reference to "braindamage" which is pretty problematic for people actually suffering from that but atleast in this case, it is directed towards a program, not a person.
Posted Apr 12, 2022 16:41 UTC (Tue)
by andy_shev (subscriber, #75870)
[Link] (10 responses)
Posted Apr 12, 2022 16:47 UTC (Tue)
by mathstuf (subscriber, #69389)
[Link] (9 responses)
The phrase is "do the ends justify the means?". And FWIW, I don't see systemd as having done any of that (as someone who doesn't run a DE but instead puts pieces together).
Posted Apr 13, 2022 9:04 UTC (Wed)
by anselm (subscriber, #2796)
[Link] (8 responses)
At the risk of flogging a dead horse that has left the stable a long time ago, here are a few things that systemd gets right which the 1980s-vintage hodgepodge that seems to be so near and dear to some people doesn't:
This is just the tip of the iceberg, and all of these make eminent sense even on systems that don't run a DE. So as far as I'm concerned the answer to the question “is that worth the trouble” is a resounding “absolutely”. You can of course run a Linux system without availing yourself of all these advantages (we've been doing that for decades after all) but, in 2022, why would you?
In my former life as a Linux instructor teaching advanced system administration I had to deal with various people who had picked up the popular notion at the time that the traditional 1980s setup was the bee's knees and systemd the spawn of the devil, but in the end they all admitted, however grudgingly, that systemd is really not a bad idea at all. It takes a little getting used to for people who have been brought up on the traditional approach, but it's worth it, and in my experience people who are entirely new to Linux system administration have a much easier time learning systemd than the traditional setup.
Posted Apr 13, 2022 10:55 UTC (Wed)
by atnot (subscriber, #124910)
[Link] (6 responses)
A look at the social side can be very enlightening there. While on a technical level, systemd criticism has failed, on a social level it has been immensely succesful. The anti-systemd movement fourished on 4chan in the early 2010s, initially comprising mostly of reactionary and homophobic sentiments towards poettering, who was there seen as the figurehead of the dawn of a more inclusive foss community. This later mutated into being the core of the more moderate anti-systemd movement we know today, as well as a few auxillary ones like the letters in support of stallman. Symbolically rejecting systemd is table stakes for being accepted in these communities, and this is where I think one finds the true reason for the continued ferver.
The lack of compelling technical arguments or alternatives is because ultimately, what systemd symbolizes has become much more important than any of it's technical merits. This has only increased with perceived symbolic losses like code of conducts and discussions of inclusivity. And while of course, I am still painting with a very broad brush here as of 2022, I think those strokes will become more and more accurate as the movement continues to radicalize and shed less extreme members as it has over the last decade.
Posted Apr 13, 2022 13:47 UTC (Wed)
by anselm (subscriber, #2796)
[Link] (1 responses)
There's also the issue that some people who have grown up with the traditional system and are heavily invested in having learnt its weird ins and outs hate to see it replaced by something that is more straightforward and unified (let alone technically superior), because that undermines their status as the venerable “greybeards” who Know Things That Mere Mortals Cannot Fathom. People used to defer to their wisdom, and of course we can't have young upstarts like that Poettering taking over and changing everything just because they can. Who do these runts think they are? There ought to be a rite of passage. (Also there must be a conspiracy involved because it is obviously impossible that people would actually prefer something like systemd for its technical merits; it is clear that some nefarious means must have somehow been used to brainwash all the big Linux distributions into adopting it against the collective resistance of the “greybeards”.)
Right now we're seeing this once more with the “/ vs. /usr” debate, where the main counterargument apart from “Resist Poettering forever!” is basically “That's how we always did it and we don't want it to change”, with a side order of irrelevant mumbo-jumbo like “/ is faster than /usr”, even though the real reason (PDP-11 hard disks in the 1970s were too small) is very well known, other Unix vendors have already done the change, and today there are obvious technical advantages to merging the two. This is systemd all over again.
Posted Apr 13, 2022 19:56 UTC (Wed)
by pebolle (guest, #35204)
[Link]
My pet theory is that things like Linux, slowly but certainly, getting easier to run on PC's, virtualization and cloud computing making running servers easy too, and - the big one - mobile phones making countless, almost magical things really easy for almost everyone made clear to many system administrators that their expertise wouldn't be needed for much longer.
It's hard to prove, but I do think the vocal systemd opponents were just shouting as hard as they could to make that future, where their expertise wasn't needed that much, please disappear.
Posted Apr 13, 2022 14:56 UTC (Wed)
by karath (subscriber, #19025)
[Link] (3 responses)
What this means is that few are looking into systemd to find real technical issues. For example, would it make sense to spin out parts of systemd into separate repositories? My feeling is that this is self-censorship - if I conceptually support systemd and then I publicly criticise/question it, then could I be tagged as one of those rabid anti-systemders?
* I am an actual greybeard that started using and programming on Unix v6 in 1980 - but I went in another direction and really only got back to hobby usage of Linux in the past 10 years.
Posted Apr 13, 2022 17:53 UTC (Wed)
by atnot (subscriber, #124910)
[Link] (2 responses)
That is not to say that the greybeards hold no responsibility for what happened. But I personally believe that if nobody but them had been stoking the flames, the whole controversy would have probably died down within five years at most.
Posted Apr 13, 2022 18:21 UTC (Wed)
by atnot (subscriber, #124910)
[Link]
Posted Apr 14, 2022 12:11 UTC (Thu)
by farnz (subscriber, #17727)
[Link]
The other two components I see are that Poettering's "style" of software development is to consider the system as a whole, not just his package's piece of it and that Poettering is happy to do the non-coding work involved in keeping the contributor pool healthy.
The first one leads to issues like early PulseAudio releases, where PulseAudio refused to work around bugs in ALSA drivers; instead, Poettering insisted that the drivers should be fixed to correctly implement the ALSA API. Other users of the ALSA API were happy to work around bugs in drivers, and some drivers had simply implemented the API well enough to make common applications of the era work, but had bugs in implementations of things like snd_pcm_query_chmaps, snd_pcm_rewind and the mixer controls that would result in issues for a caller that expected the API contract to be met. Because PulseAudio was the only significant ALSA client at the time (JACK was the other, but users of JACK tended to own sound hardware with good drivers) that cared about the quality of your ALSA driver implementation, Poettering got an unfairly bad reputation, not helped by Ubuntu forcing people over to PulseAudio at a point where the PulseAudio developers (including Poettering) felt it wasn't ready for widespread adoption because they were still sorting through the driver bugs that needed to be fixed.
It also leads to people not liking systemd because it used features of the Linux kernel that had previously been ignored by most users. Poettering took (and takes) the view that if a feature is not meant to be used, then it should not be in the kernel, which has upset certain people who consider features systemd depends upon to be of sufficiently poor quality that they should be removed.
The second point, about looking after the contributor pool, means that Poettering puts up with flak aimed at him when it's in fact someone else's learning curve that's causing the errors; so usrmerge (which was mostly Kay Sievers and Harald Hoyer) gets blamed on Poettering because Poettering wrote it up clearly. When Kay Sievers made a bad decision in udev, Poettering got the flak because he explained the background to that decision, and why, given the context, it wasn't an obviously bad decision to make until the consequences became clear.
Posted Apr 14, 2022 10:13 UTC (Thu)
by johannbg (guest, #65743)
[Link]
It was apparent early on in systemd's development that getting people to adapt to the new concept would be one of the more difficult challenges to overcome, which is why I was a hard advocated for the 'rip the Band-Aid off' approach at that time ( like no backwards compatibility with the legacy service foo commands ) which in turn would have forced people to let go of the past and adapt the new concept and the commands that came with it. ( if you understood the concept you would understand the approach and also understand everything else that followed )
More cruder, more effective, less painful ( as in having people scream into the void once rather than prolong those screams for over a decade like it has ) approach.
Posted Apr 1, 2022 12:59 UTC (Fri)
by kronat (subscriber, #117266)
[Link] (2 responses)
Now, what if in these years, something was added to the kernel sources making me unable to use the latest kernel? I would have to grab some old installer of Debian, let's say 4.0 that was there in 2008, and never touch anything on it, and hoping that everything will be usable (are most websites still working with, let's say, Firefox 3.0?). I consider this case as a regression: I would expect that most of the userspace would continue working with an older kernel (in the end, what I would need to install on it? Basic stuff -- like systemd (unfortunately), a browser, maybe a light DE like XFCE...).
In this light, having a look at the PR linked in one of the first posts is like sand in my eyes. A simple code that checks for a functionality (which doesn't seem to be bug-prone or a source of possible problems) is just removed, and the minimum supported version is bumped.
What are my options against this programmed obsolescence? Just accept it, and stick with proprietary vendors which may decide that every 2-3 years I need to buy a new laptop? I am without words.
Most of the time, the quality one receives is directly proportional to the money one is paying.
Posted Apr 1, 2022 20:26 UTC (Fri)
by jafd (subscriber, #129642)
[Link] (1 responses)
Posted May 1, 2022 13:31 UTC (Sun)
by HelloWorld (guest, #56129)
[Link]
Besides, kronat’s point is largely hypothetical. Changes that break the kernel’s ability to run on old hardware just don’t happen all that often. Unless of course you insist on running proprietary drivers, in which case the problem isn’t with systemd but the hardware vendor.
Posted Apr 6, 2022 10:14 UTC (Wed)
by zuki (subscriber, #41808)
[Link]
In the end in systemd upstream we took the route of recommending newer kernel versions, but not removing any compat code, at least for now. We'll emit a warning when kernels <4.15 are used with systemd-251: https://github.com/systemd/systemd/commit/277f05872f8d5c0...
Posted Apr 7, 2022 14:24 UTC (Thu)
by mrugiero (guest, #153040)
[Link] (1 responses)
In one scenario, you have a crappy ARM vendor. But then, crappy ARM vendors tend to just make a single code drop (if they drop the code at all) and you get a stale userspace as well. I don't see many attempting to update that. So, even in the unlikely case this vendor's image ships systemd, it's likely to be frozen as well.
In the other, you have some LTS distro that is expected to remain stable. Due or undue, systemd is famed as unstable. I can't see any distro maintainer being dumb enough to pretend to keep an old kernel for stability but then following the latest releases of pid 1, specifically in a version with so many features (and thus surface for stability). I'd dare say the same applies to other systemd services, but at the very least pid 1 I wouldn't expect to be updated.
Posted Apr 11, 2022 19:22 UTC (Mon)
by bfields (subscriber, #19510)
[Link]
Isn't that the kind of thing that community distros for phones or wireless routers sometimes try to do? Proprietary drivers and out-of-tree patches might leave them stuck with the original, but they might still have some hope of replacing userspace.
But I may be wrong, I'm not familiar enough with those projects to give specific examples.
Systemd discusses its kernel-version needs
and CLOCK_BOOTTIME is now assumed to always exist."
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
I mean, it would've been quicker to have a look at the code than throwing your rant at the input form.
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
2. https://kernel.org/
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
That an issue has been labelled ( you can call that label what you want, triage, RFE whatever ) just means two things a) the issue has been looked at ( or labelled automatically like we do in our projects ) and b) the bug will be looked at again ( otherwise it would not still remain open ).
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
/proc and /sys are definitely considered part of the user-space ABI. Have you reported changes that caused you problems? That really isn't meant to happen.
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Author: Lan Tianyu
Date: Fri Oct 11 09:54:10 2013 +0800
There is no user of cm_sbs.c and CONFIG_ACPI_PROCFS_POWER. So remove
them. Prepare for removing /proc/acpi
Author: Lan Tianyu
Date: Sun May 4 11:07:24 2014 +0800
The commit 1e2d9cd and 7d7ee95 remove ACPI Proc Battery
directory and breaks some old userspace tools. This patch
is to revert 7d7ee95.
Author: Randy Dunlap
Date: Fri Mar 2 16:55:54 2018 -0800
Fix grammar and punctuation (end sentences with a period) in the
Kconfig help text for ACPI_PROCFS_POWER.
I was looking at this since it appears to be going away (again,
some day) and I have a working script that uses this info to tell me
battery usage. I can update the script to use /sys/class/power_supply
(in theory) but the contents (with units) should be documented in
Documentation/ABI/ before /proc/acpi/battery/ is removed (IMO).</p>
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Bisecting with old kernels
Bisecting with old kernels
Bisecting with old kernels
Bisecting with old kernels
Bisecting with old kernels
Bisecting with old kernels
Bisecting with old kernels
Bisecting with old kernels
The phrase is "do the ends justify the means?". And FWIW, I don't see systemd as having done any of that (as someone who doesn't run a DE but instead puts pieces together).
Bisecting with old kernels
Bisecting with old kernels
Symbolically rejecting systemd is table stakes for being accepted in these communities, and this is where I think one finds the true reason for the continued ferver.
Bisecting with old kernels
anti-systemd is a broad church
anti-systemd is a broad church
anti-systemd is a broad church
anti-systemd is a broad church
Bisecting with old kernels
Upgrade till it's supported, then... ?
Upgrade till it's supported, then... ?
Upgrade till it's supported, then... ?
You can ridicule it all you want, but it's actually true. It's not that those three lines are complicated, it's that if you’re going to support old kernels, you also need to do the associated testing/QA work to make sure it keeps working, which is a huge pain. This becomes especially relevant when you try to significantly refactor parts of the code.
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
Systemd discusses its kernel-version needs
In one scenario, you have a crappy ARM vendor. But then, crappy ARM vendors tend to just make a single code drop (if they drop the code at all) and you get a stale userspace as well. I don't see many attempting to update that.