A new AUTOSEL release [LWN.net]

AUTOSEL is a tool that is used to find kernel patches that should be considered for backporting into the stable releases. Sasha Levin has announced a new and completely rewritten version of AUTOSEL for those who would like to play with it.

Unlike the previous version that relied on word statistics and older neural network techniques, AUTOSEL leverages modern large language models and embedding technology to provide significantly more accurate recommendations.

to post comments

Rebaselining?

Posted May 6, 2025 9:21 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (10 responses)

Has there been an effort to sample old patches to see if it selects:

- patches that were missed before (and what percentage of those were eventually manually picked)
- did not select patches that had been selected before (and what percentage of those had followup discussions due to issues)

I'm not saying it doesn't have "significantly more accurate recommendations", but I'd be interested in numbers.

Rebaselining?

Posted May 6, 2025 13:44 UTC (Tue) by Baughn (subscriber, #124425) [Link] (4 responses)

And to fine-tune it based on such errors? There are caveats to that—if you make any mistakes, you’ll accidentally train it lie—but given a carefully curated training set it sold be beneficial.

I would assume it has been, really, but that’s something I’d have liked to see in this article.

Rebaselining?

Posted May 6, 2025 13:50 UTC (Tue) by sashal (✭ supporter ✭, #81842) [Link] (1 responses)

There's no fine tuning involved per-se, but as you point out the results might be skewed by previous incorrect inclusions or exclusions for the stable trees.

In practice, more often then not there's no "right" answer, and many of the commits we backport land in a gray zone where you could have arguments for and against backporting, so really I wouldn't say that the LLM is trained to lie, but rather encouraged to have optionions closer to the ones of the humans that operate it.

Rebaselining?

Posted May 7, 2025 0:13 UTC (Wed) by rolexhamster (guest, #158445) [Link]

In practice, more often then not there's no "right" answer, and many of the commits we backport land in a gray zone ...

I'm sorry, but that sounds awfully like weasel wording. When pressed for quantitative measures, the above answer in effect states that quantitative measures are irrelevant.

If we can't measure progress, then how do we know this new approach is any better than the old method? Is this a case of "trust me bro, I know what I'm doing, really, pinky-promise"?

There have been documented instances of the wrong stuff being "backported" to the kernel, causing all sorts of problems. When running the new approach on these particular cases, did the wrong stuff get selected or rejected for backporting?

Rebaselining?

Posted May 6, 2025 17:32 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (1 responses)

I have no idea; I imagine LLM retraining is more difficult than some prompt tweaking. I'm just interested in an analysis of the two tools in reducing false positives and false negatives.

Rebaselining?

Posted May 7, 2025 13:03 UTC (Wed) by kleptog (subscriber, #1183) [Link]

From what I can see of the code there not a lot of retraining or baselining going on. The most interesting part is the use of embeddings to index the patches.

The idea presumably being that instead of doing matches on plain text, the LLM model is used to make an embedding that has some understanding of code and so produces better matches. The prompt is only there to explain to the LLM to compare the found "nearby patches" and decide if they're close enough.

I don't expect the prompt given to the model to really make much of a difference. Any improvements are likely to be based on improving the representation of the diff to the LLM so the embedding better represents the changes on a semantic level. Currently they just throw the commit message, the diff and list of files and other miscellaneous info in, but you could imagine a more complex setup that looks at the before/after code directly and associated email discussions to produce beter results.

Tracking false positives/false negatives is definitely an interesting idea to see where the flaws are, but it's not directly obvious what you could do with that info.

Rebaselining?

Posted May 6, 2025 13:47 UTC (Tue) by sashal (✭ supporter ✭, #81842) [Link] (4 responses)

A lot of it is my subjective opinion :)

The last AUTOSEL set I've sent out (https://lore.kernel.org/all/20250505221419.2672473-1-sash...) is basically a re-review of the v6.15 merge window using this rewrite. I've gone over that range once with the old tooling, so everything the new tool dug up is extras which we've previously missed.

Rebaselining?

Posted May 6, 2025 14:14 UTC (Tue) by Lionel_Debroux (subscriber, #30014) [Link] (3 responses)

In early January 2019, I sent gregkh 5 commit IDs for upstream commits which had been backported into grsecurity 4.4 by spender's independent backport process. Quoting myself:
"
* apparmor:
* b6b1b81b3afba (from 2014) fixes an oops (...);
* d671e890205a6 (2014) forces an update of the profile file's mtime.
* 58acf9d911c88 (2016) fixes capabilities handling related to namespaces.
I haven't checked why the two first of these commits aren't in the '2015 4.4 series, but AFAICS, these fixes are still in 4.20, though the code was more or less significantly adjusted by other changes in the meantime. Also, I haven't examined all apparmor hunks in the grsec patch against 4.4.162, just several ones which, in several seconds, looked like stable backports.

* x86 kprobes: ee6a7354a3629 (2018) prohibits probing on MOV SS and POP SS due to their odd exception delaying behaviour;
* x86 uprobes: 13ebe18c94f5b (2018) does the same as in kprobes.
"

Greg replied in November 2019 (nearly 11 months later) that all of these looked sane, and would be queued if they passed build testing.

Is the new tooling able to pinpoint these commits as potential backports to early 2019 stable kernels ? The commit message of the first one mentions "oops", so any piece of remotely decent tooling, be it based on such advanced technology as `grep`, or heavyweight newfangled A"I", would be expected to flag it for closer examination by persons with natural intelligence.

Newer grsecurity patches (last time I checked, up to 2021) can be downloaded from the open Internet under the GPLv2, and throughout the years, grsecurity patches consistently show a growing number of backports missing from each base mainline kernel version they apply to.

Rebaselining?

Posted May 6, 2025 15:26 UTC (Tue) by sashal (✭ supporter ✭, #81842) [Link] (2 responses)

That's the hope. Here's the output for the 5 commits you've listed:

https://gist.github.com/sashalevin/23490e38da1be481ac462e...

Rebaselining?

Posted May 6, 2025 22:47 UTC (Tue) by Lionel_Debroux (subscriber, #30014) [Link] (1 responses)

OK, I see the kind of output that it can provide. Sound reasonable as long as excessive false positives don't burden humans and kill the usefulness of the feature in their eyes :)
I have a hunch that most maintainers might not get _too_ grumpy if there's a single AUTOSEL patchset per mainline kernel release cycle, i.e. 5-6 patchsets per year, and it contains a reasonable proportion of useful changes amid the noise; however, an AUTOSEL patchset for every stable kernel release would be really annoying for maintainers unless it's near-perfect.

Complaints about the output of AUTOSEL seem to include the facts that it's picking up only a subset of patches which were part of a series (but that kind of information gets lost in the integration to a Git repository, right ? And frequently re-training the model on the LKML and auxiliary MLs might get costly), or that it misses followup fixes (which do not necessarily have Fixes: tags in their commit messages). No easy fixes for these, certainly.

Rebaselining?

Posted May 7, 2025 8:24 UTC (Wed) by geert (subscriber, #98403) [Link]

> it's picking up only a subset of patches which were part of a series (but that kind of information gets lost in the integration to a Git repository, right ?

If the commit has a proper Link:-tag pointing to lore[1], that information is still available, but needs a more expensive[2] reverse look-up to find the other commits in the series.

[1] Ca. 20% of the commits added between v6.14 and v6.15-rc5 lack such tags...
[2] Especially if it is a series crossing subsystems.

Backports vs. QA

Posted May 6, 2025 16:22 UTC (Tue) by koverstreet (subscriber, #4296) [Link] (2 responses)

I've been saying this privately for awhile - but we wouldn't need to invest so much effort into backports if we put a bit more effort into QA.

Automated testing - bonus points for a bit of code coverage analysis and error injection. A little really goes a long ways.

And when you have that, it makes the whole development process smoother. It's good for users and it's good for developers.

Users can stick closer to mainline if they know it's going to be stable, and more frequent, smaller updates is really what you want to be doing: delaying your updates is really a form of tech debt. When a bug does make it into your LTS release and you're only updating once every year or two - have fun bisecting that and teasing it out from all the other changes.

Life is way easier as a developer when you've good solid automated tests telling you if your code is good or not. You can move quicker and you spend a lot less time getting yanked off the thing you were working on to debug code that should've been finished.

bcachefs shows what the payoff is for good engineering practices - I'm able to write code quickly and get it out quickly because I'm not losing sleep over regressions. So far in 6.15, all the bcachefs regressions (all two of them) were found and fixed before rc1 came out.

That's not just testing though, that also comes from sinking a ton of time into assertions/integrity checking, keeping the codebase relatively clean so that good assertions are possible, and a lot of introspection and runtime debugging so that when bugs do make it out to users they're usually trivial to debug.

Backports vs. QA

Posted May 7, 2025 10:46 UTC (Wed) by atnot (guest, #124910) [Link] (1 responses)

I think the difficulty is far more with somewhat antique development practices in general, of which the lack of CI is only a part of. It would probably be good to do better testing so more people stay on mainline, but some people will for various reasons (e.g. out of tree modules) always want to stay back.

If you look at modern projects that do lots of backporting for comparison, e.g. nixos, it tends to be set up something like the following:

1. The changes are submitted, the tests run, people review the changes, approve them and so on as usual.
2. A "backport" label is added in the web interface. This can be done by anyone involved.
3. Once the label is added, or the change is merged, it triggers a bot that cherry-picks the commits and opens new pull requests against the stable branches.
4. The tests run again. The people who reviewed or submitted the original change can quickly review the backported code and deal with any failures, right after reviewing the original change.
5. The original code and backported code get merged like any other change.

I think this is a better workflow on a number of axes.

* Testing. The backported code needs to pass the same automated tests before and after being backported as a matter of course.
* Locality: There is less task switching. If you just wrote or reviewed the original code, it's much easier to review the backport too. This is much more efficient than having it done by two different people potentially weeks apart.
* Work distribution: All developers can nominate changes for backporting, everyone can review the backported changes. This reduces load on the maintainers.
* Atomicity: There is no issue of only picking up half a series. The path of least resistance is to make all of your bug fixes easily backportable because nobody else will helpfully fix them for you.
* Automation: Nobody needs to read hundreds of emails of prose telling them what to backport. You just go through the web UI and hit a button. The rigidity of the automation also sets a common baseline and encourages developers to stick to the "happy path" lest they make more work for themselves they have to clean up manually.
* Transparency: everything happens in public. There are no private email inboxes or scripts that run on people's personal machines. This helps everyone understand the process, know the status of each change, why things have or haven't been backported yet, get involved, or see when they are creating undue work for others.

I'm sure there's a few more. Point is, there are models for backporting that have a lot less friction than what the kernel is currently doing. They don't require ridiculous things like using LLMs to select patches because instead of one or two overworked people doing adversarial backporting, easy, publicly visible, automated workflows allow it to be treated as a shared responsibility instead.

Backports vs. QA

Posted May 8, 2025 0:21 UTC (Thu) by koverstreet (subscriber, #4296) [Link]

That's exactly what I imagined as my ideal process, back when bcachefs first went upstream.

Do you know if there's any good CLI tools for this, perhaps ones that NixOS is using?

If I had a CLI tool that just compared two branches, and gave me a list of patches that were marked for backporting but not yet cherry-picked into the target branch - that right there take away some of the mental juggling.

How autosel works

Posted May 6, 2025 17:59 UTC (Tue) by nickodell (subscriber, #125165) [Link]

I was curious how the author is doing this, so I took a look at the code.

First, it takes the diff of the commit.
Then, it compares that diff to other diffs by semantic similarity.
Next, he's using those similar commits to provide examples of backported/non-backported commits, and do few-shot learning. Whether a prior commit was backported is detected using git metadata. You can see the prompt here. https://git.sr.ht/~sashal/autosel/tree/master/item/src/cl...

I do wonder if might be a bias in this dataset, in that there are legitimate bugs that ought to have been backported, but never were. (Indeed, that's what autosel is meant to fix!) I'm not sure how to fix this. Perhaps it could randomly drop some percentage of negative examples to restore the positive/negative balance to the "true" backport rate?