Extensible scheduler class to be merged for 6.11

[Posted June 11, 2024 by corbet]

The extensible scheduler class ("sched_ext") framework allows the writing of CPU schedulers as a set of BPF programs. It has been somewhat controversial, and its merging into the kernel has been blocked despite a clear level of interest from users. Linus Torvalds has now let it be known that he has made a decision and, overriding the scheduler maintainer, will merge sched_ext for the 6.11 release.

I honestly see no reason to delay this any more. This whole patchset was the major (private) discussion at last year's kernel maintainer summit, and I don't find any value in having the same discussion (whether off-list or as an actual event) at the upcoming maintainer summit one year later, so to make any kind of sane progress, my current plan is to merge this for 6.11.

Great News

Posted Jun 11, 2024 23:16 UTC (Tue) by jason.rahman (subscriber, #120510) [Link] (1 responses)

Wow, a bit surprising (in a good way) to see this. The rest of the community was clearly rallying around sched_ext and demonstrating clear wins, Ubuntu in particular was very openly showing good improvements. I suppose Linus saw the writing on the wall that a lot of folks were on the cusp of adoption but as an out of tree patch set.

Overriding the MAINTAINERS

Posted Jun 13, 2024 4:38 UTC (Thu) by alison (subscriber, #63752) [Link]

The last time I recall Linus overriding a top kernel contributor was when he decided to merge Android's binder and ashmem, which had been in staging for a long time. Apparently GKH was not enthusiastic about the merge, but Linus said something like, "Millions of people are using it, so we should merge it."

Regarding use cases, SCHED_EXT author Dan Schatzberg's talk at Southern California Linux Expo was excellent, and Daroc wrote about it for LWN: https://lwn.net/Articles/966618/

The future looks bright

Posted Jun 12, 2024 5:54 UTC (Wed) by iq-0 (subscriber, #36655) [Link] (3 responses)

After all the talks about it I’m curious how merging this will make (applied) scheduler research more practical and how that’ll lead to improvements to the generic scheduler in the time to come.

The future looks bright

Posted Jun 12, 2024 6:36 UTC (Wed) by WolfWings (subscriber, #56790) [Link] (2 responses)

Honestly I expect a fairly large amount of schedulers to appear, and the major distros to end up with their own scheduler flavors available. "Desktop" vs "Server" installs will actually make a difference in that regard more likely at a minimum.

The future looks bright

Posted Jun 12, 2024 10:49 UTC (Wed) by farnz (subscriber, #17727) [Link] (1 responses)

The biggest set of experiments I expect to see are on the desktop; can you schedule based on knowledge of what the user can see and interact with, such that while overall system throughput is down, user-perceived responsiveness is up? Can you schedule such that a Steam game (which itself involves multiple processes) gets a lower maximum time per frame?

The future looks bright

Posted Jun 14, 2024 1:07 UTC (Fri) by raistlin (guest, #37586) [Link]

The Steam games part is already happening, as sched_ext is being evaluated (i.e., they have a scheduler already, with benchmarks results, etc) for the Steam Deck:

https://lore.kernel.org/lkml/20240501151312.635565-1-tj@k...

> - Valve has been working with Igalia to implement a sched_ext scheduler for
> Steam Deck. The development is still in its early stages but they are
> already happy with the results (more consistent FPS) and are planning to
> enable the scheduler on Steam Deck.
>
> https://github.com/sched-ext/scx/tree/main/scheds/rust/sc...
> https://ossna2024.sched.com/event/1aBOT/optimizing-schedu...

And:

https://blogs.igalia.com/changwoo/sched-ext-a-bpf-extensi...
https://ossna2024.sched.com/event/1aBOT/optimizing-schedu...

Scheduling

Posted Jun 12, 2024 6:31 UTC (Wed) by epa (subscriber, #39769) [Link] (1 responses)

A new property of schedulers: guaranteed sane progress.

Scheduling

Posted Jun 12, 2024 7:46 UTC (Wed) by mattburgess (subscriber, #143223) [Link]

Well played sir, well played. That gets my nomination for QOTW!

should the kernel have an engineering council?

Posted Jun 12, 2024 9:47 UTC (Wed) by danielkza (subscriber, #66161) [Link] (3 responses)

It's nice to see Linus injecting some sanity into the discussion, but it would be preferable to have a process of some sort so these kinds of deadlocks can be resolved in a consistent way.

I believe there kernel should have something analogous to distributions' Engineering Councils to settle these types of disputes. While maintainer authority is important, there must be checks to stop it from being used across reasonable subsystem barriers and/or in obstructionist ways (even if with good intentions).

should the kernel have an engineering council?

Posted Jun 12, 2024 12:21 UTC (Wed) by khim (subscriber, #9252) [Link] (1 responses)

Linus may always override any decision of any maintainer.

He uses that superpower very rarely, which is precisely how he may keep it: every time Linus does that some people become angry at him, obviously, and frequent application would make community leave Linus, but as long as he uses such superpower only once a year or even less often the majority of developers support him.

At some point kernel community would need to find a way to do that without Linus, but that that bridge would be crossed when it would be reached, there are no rush.

should the kernel have an engineering council?

Posted Aug 17, 2024 12:11 UTC (Sat) by Lennie (subscriber, #49641) [Link]

Well, at the moment if Linus somehow wasn't available long term, things would stay the same, except it would just be someone else at the top like Greg Kroah-Hartman.

Current process is appeal to Linus

Posted Jun 12, 2024 12:24 UTC (Wed) by farnz (subscriber, #17727) [Link]

It does have such a process at the moment; Linus is the final arbiter (analogous to an "Engineering Council"), and if there's a deadlock that you think should be resolved in your favour, you go to Linus to get things resolved. Linus trusts his chosen maintainers, so the process is biased in their favour, but if they're being donkeys, Linus will break the deadlock - in the extreme, by doing what he's doing with sched_ext and merging it over maintainer objection.

Ultimately, the current kernel is structured with Linus as benevolent dictator; all a Council or Committee can do is make suggestions to Linus, and see if he takes them. Of course, the same applies to maintainers; while they suggest changes to Linus via pull requests, he has the ultimate authority over what changes will and will not be accepted.