AI review? [LWN.net]

AI review?

Posted Sep 14, 2025 22:22 UTC (Sun) by shemminger (subscriber, #5739)
Parent article: Creating a healthy kernel subsystem community

I wished AI patch review worked. It doesn't now.
You would think it would be a natural for being able to digest large volume text sources like LKML and subsystem mailing lists. But in the few times I have tried the existing tools, the results were wildly disappointing. Full of vague stuff (more error checking is needed but where) and missing common failure patterns. My guess is that generic AI in present state is unable to understand what is important. There are static checkers but those are really just better versions of lint.

to post comments

AI review?

Posted Sep 15, 2025 0:10 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

What I think would be far more useful is an LLM-as-LSP setup where, instead of conforming to a chat interface, I can have an LLM "looking over my shoulder" and adding annotations directly to my editor as I'm working. If it then feeds suggested changes as LSP code actions, I don't have to pause any editing for the LLM to crunch its numbers so that its suggestions don't conflict. Even if the suggestions are just comments like "this codepath should be tested", I can later prompt it to try and create a skeleton test case for it.

But I really don't know how to do this in a token-efficient manner when forced into a chat-shaped API. Do you send the file and some kind of patch sequences on a timer so you're not tossing the entire context at it for each edit? Pare it down to `ed` commands and send that as the patch sequence?

At least to me, it sounds far more interesting than an LLM puking out hunks of unreviewed code at me at least (as someone who cares about long-term maintenance at least).

AI review?

Posted Sep 15, 2025 8:24 UTC (Mon) by viro (subscriber, #7872) [Link] (2 responses)

Generic AI in present state is unable to understand, period. It can produce a text statistically indistinguishable from actual review of something vaguely similar to what it's been given. Which, fair enough, does pass the Turing Test - there are human beings who operate on that level; the problem is that reviews from such human beings are worse than useless.

The difference between broken and correct can be subtle and highly non-local; what's more, you need the trainers capable of doing the analysis themselves *and* of giving a usable feedback to trainees, whoever or whatever those trainees might be. It's hard to do, it takes serious time and considering the amount of training needed for AI models, employing enough of such trainers would cost way too fucking much.

AI review?

Posted Sep 15, 2025 10:02 UTC (Mon) by mb (subscriber, #50428) [Link] (1 responses)

Well, in reality though, these models do find bugs and help review or debug code.

I'm not going to argue whether an AI model "understands" anything or not, because we'd first have to define what "understanding" means.

I'm neither saying that AI models find all problems nor that their findings are always correct.
But they can find problems that are non-trivial and non-local effects. I did use AI models to do exactly that successfully multiple times.

I can give you an example of a multithreading bug that I was hunting for two weeks:
https://github.com/mbuesch/httun/commit/d801db03c8677d4eb...

In line 530 the wrong task abort handle is cloned which leads to very rarely hanging and sluggish network communication on the user level because the the task is not aborted and the other tasks still talk to the old task with outdated state.
This problem is covered up by the other layers in the system and the peer across the network having restart and retry logic.
Due to that it was not obvious at all in what part of the whole application stack the problem was.

What I did is give the source code to Gemini and describe what behavior I was seeing and what I already had discovered during my two weeks of debugging. (Basically that I suspected the problem to be in the client part and that I suspected the task's state data to be outdated.)
Gemini responded literally that there was a copy and paste problem in line 530.
My head still hurts from banging it into the wall after reading this very first sentence.

It went on to describe in an extremely detailled and correct way how that c&p problem does prevent the task from being aborted and how that would lead to old state being preserved and so on.

So, at this point I don't actually care whether Gemini "understands" my code or my explanations as long as it gives me correct results.
Would I eventually have found this bug without AI? Probably yes. Would I have been much faster and would I have less grey hair now if I would have asked Gemini earlier in the process. Totally absolutely yes!

Gemini found the problem, correctly explained it in a lengthy text and provided a correct fix for it by fixing the c&p typo (I decided to fix it differently). Therefore, AI is a tool that I like to use and it improves the quality of my code. I don't see why there would be anything wrong with that. Most of the time today's AI is unable to help me with debugging and code review. However if it only helps one in 20 times, it's absolutely worth it.

AI review?

Posted Sep 15, 2025 12:09 UTC (Mon) by iabervon (subscriber, #722) [Link]

An LLM will have a lot of success at writing a document that says the most statistically unlikely text in your code is the problem. If you've got a bug, and you've been overlooking it for a while, that document is probably actually accurate, because the issue is that you've been reading the code as if it was the predictable, correct thing, and the actual code is not that.

On the other hand, that document wouldn't be good code review for a patch that is correct, or one where the problem isn't a disagreement between the actual code and what people (or the model) expects the code to be when looking at it.