Some weekend stable kernel updates

Posted Jan 20, 2024 20:31 UTC (Sat) by willy (subscriber, #9762)
In reply to: Some weekend stable kernel updates by npws
Parent article: Some weekend stable kernel updates

Sometimes there's another commit that has that as a dependency. That's been the reason for some of my patches being backported that didn't seem necessary. I haven't looked to see whether that's true for this case.

Some weekend stable kernel updates

Posted Jan 20, 2024 23:12 UTC (Sat) by pbonzini (subscriber, #60935) [Link] (11 responses)

In this case it's autosel that went a little rogue

https://www.spinics.net/lists/kernel/msg5057819.html

Some weekend stable kernel updates

Posted Jan 21, 2024 8:41 UTC (Sun) by donald.buczek (subscriber, #112892) [Link] (9 responses)

The number of vetoed autoselected patches listed at https://lore.kernel.org/stable/?q=autosel is alarming. And because of the high number of autoselected patches, I fear that most inappropriate backports might not even be adequately reviewed by domain experts.

Adding to my worry are instances of reversions of an autoselected patch from a previous release, like "Revert 'md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d'". This does not inspire confidence in the autosel system, although in this case the issue might be attributed to the complex and unstructured nature of the subsystem's parallel programming.

It might be more prudent to have stable patches be opt-in rather than opt-out. autosel could simply suggest patches to authors.

Some weekend stable kernel updates

Posted Jan 21, 2024 17:08 UTC (Sun) by willy (subscriber, #9762) [Link] (8 responses)

I mean, these are not new observations. There's a general "I don't care about stable" attitude amongst maintainers. To a certain extent that's fair; this really is the job of distribution maintainers, except that it's really hard to have domain experts on staff at each distro. And those who are on staff also have their own projects to work on.

Nobody who's currently a maintainer signed up for the extra work of backporting fixes to stable kernels. And it is a lot of work. But the fact is that users don't run top of tree, and we don't really want them to. So I think the attitude needs to change; one kernel a year with support for four years really isn't that onerous.

What I need is better tooling. If I don't do the backport as soon as Greg sends me the "failed" message, it doesn't happen. A dashboard somewhere would be helpful.

Some weekend stable kernel updates

Posted Jan 22, 2024 6:10 UTC (Mon) by rolexhamster (guest, #158445) [Link] (2 responses)

There's a general "I don't care about stable" attitude amongst maintainers. To a certain extent that's fair (...)

Then what's the point of the stable kernels if nobody takes proper care of them?

The mentioned attitude from the "maintainers" (kernel developers) is akin to throwing something over the wall (look, shiny!) and then running away from all responsibilities associated with that thing.

It's as if the kernel exists only for the sake of developing the kernel.

This brings back memories of the so-called CADT (Cascade of Attention-Deficit Teenagers) development model from GNOME 2 days.

Some weekend stable kernel updates

Posted Jan 22, 2024 8:37 UTC (Mon) by geert (subscriber, #98403) [Link]

> > There's a general "I don't care about stable" attitude amongst maintainers. To a certain extent that's fair (...)

(A)

> Then what's the point of the stable kernels if nobody takes proper care of them?

> The mentioned attitude from the "maintainers" (kernel developers) is akin to throwing something over the wall (look, shiny!) and then running away from all responsibilities associated with that thing.

(B)

A and B are two different things.
I am quite sure most maintainers and kernel developers do their best to get bugs (in the upstream kernel) fixed (in the upstream kernel) ASAP.

Some weekend stable kernel updates

Posted Jan 22, 2024 8:57 UTC (Mon) by pbonzini (subscriber, #60935) [Link]

Well, if they did not work on the develoment trees, there would be no kernel at all. Having 1-2 stable kernel branches is fine, but If somebody wants to have a long-term stable kernel, they can (surprise!) pay people to do backports.

Being a developer, a subsystem maintainer and a maintainance engineer is three different jobs. Some maintainers also work on community activities such as organizing conferences, or on tasks that are required by their employers (who are agreeing to donate at least part of their employee's time to helping volunteers and developers from other companies, at times including competitors). There's just not enough time for maintainers to do and test backports to five different long-term branches, plus the two currently-active short-term stable branches.

Some weekend stable kernel updates

Posted Jan 22, 2024 9:06 UTC (Mon) by pbonzini (subscriber, #60935) [Link]

> But the fact is that users don't run top of tree, and we don't really want them to. [...] one kernel a year with support for four years really isn't that onerous.

Or is it? Currently, almost everyone is using either 6.6 or a distro kernel. Marked-for-stable patches usually will apply to 6.6 and 6.7 just fine, and if not maintainers will usually pay attention to those. But beyond that, putting the responsibility on maintainers does nothing but cause burnout. Responsibility for a task needs to move as close as possible to whoever benefits from it being done.

So, if a patch fails to apply to a long term kernel, I just don't care. Canonical, SUSE and Oracle use LTS kernels, they can work *together*, upstream on attending to the failed-to-apply patches. They can build themselves the dashboard that you talked about, which I agree would be very useful.

Some weekend stable kernel updates

Posted Jan 22, 2024 10:37 UTC (Mon) by helmut.schmidt (subscriber, #113671) [Link] (1 responses)

> I mean, these are not new observations. There's a general "I don't care about stable" attitude amongst maintainers. To a certain extent that's fair; this really is the job of distribution maintainers, except that it's really hard to have domain experts on staff at each distro. And those who are on staff also have their own projects to work on.
> Nobody who's currently a maintainer signed up for the extra work of backporting fixes to stable kernels.

Agreed. The point I'm raising is about the autosel process determining which patches make it into stable releases. I know, that it is the general consensus that it's beneficial overall, despite the occasional mistakes, because without it many crucial patches would be missed. But every mistake feeds my doubt.

> But the fact is that users don't run top of tree, and we don't really want them to.

Why not?

Ideally, if mainline wouldn't break things, there was no need for stable kernels.

Some weekend stable kernel updates

Posted Jan 22, 2024 14:06 UTC (Mon) by cesarb (subscriber, #6266) [Link]

> because without it many crucial patches would be missed.

This is the core of the discussion: is it better to avoid missing patches at the cost of occasionally including a bad one, or is it better to avoid including bad patches at the cost of occasionally missing a good one?

> Ideally, if mainline wouldn't break things, there was no need for stable kernels.

Breaking things is hard to avoid, since mainline always has a large amount of large changes. The nice thing about stable kernels is that they (are supposed to) only apply minor patches on top of mainline which gradually fix most of the breakage caused by the large changes.

The best of both worlds is probably to follow a bit behind mainline, waiting for a few stable releases before jumping to a new mainline release (but never staying behind for too long). That is the approach used for instance by Fedora; at the moment, it's at 6.6.12 moving to 6.6.13 (according to https://admin.fedoraproject.org/updates/).

Some weekend stable kernel updates

Posted Jan 23, 2024 1:31 UTC (Tue) by npws (subscriber, #168248) [Link] (1 responses)

Well I suppose Greg signed up for it, but it seems he doesn't do any review at all. These patches are not hard to spot. If he doesn't want to do the work, he should look for someone who does a better job.

Some weekend stable kernel updates

Posted Jan 23, 2024 8:13 UTC (Tue) by Wol (subscriber, #4433) [Link]

> These patches are not hard to spot.

If you've got 24hrs, in which to do 48hrs work, these patches are hard to SEE, let alone spot ...

Cheers,
Wol

Some weekend stable kernel updates

Posted Jan 22, 2024 23:04 UTC (Mon) by sashal (✭ supporter ✭, #81842) [Link]

I found that AUTOSEL tends to pick commits that are authored by Linus almost 100% of the time. This is what happens when someone has a history of writing innocent looking commit messages for ugly bugs :)