False positive identification rate

Posted Apr 1, 2026 19:17 UTC (Wed) by iabervon (subscriber, #722)
In reply to: False positive identification rate by Paf
Parent article: The role of LLMs in patch review

That's one direction of validation, but Sashiko obviously produces more comments than 50% of the later patches with fixes. If it catches half of the bugs in the code it reviews, but induces developers to introduce twice as many bugs in response to other comments, it's not making the code better. The general assumption is that if a reviewer says something about a real bug, the developer will fix it, and if the reviewer says something that's not about a real bug, the developer will do nothing, which means that false positives are a cost to the development process but safe. However, developers presumably don't really react correctly to all comments, so we should be worried about overly convincing false positives as well as true positives buried in too much noise.

to post comments

False positive identification rate

Posted Apr 2, 2026 0:14 UTC (Thu) by Paf (subscriber, #91811) [Link]

That’s fair in the abstract, but seems overblown. In my usage I have yet to see it offer “this is a bug and must be fixed” and be wrong. There’s a certain amount of “I am not sure about this?”, and those are lower quality, though often linked to questionable code readability in the source. I’m sure it does the first category of error, but it seems to be rare.

Of note, Morton thinks this tool is so good it should be made mandatory. He has some significant experience of code review processes.