Exelbierd: What's actually in a Sashiko review? [LWN.net]

Brian "bex" Exelbierd has published a blog post exploring follow-up questions raised by the recent debate about the use of the LLM-based review tool Sashiko in the memory-management subsystem. His main finding is that Sashiko reviews are bi-modal with regards to whether they contain reports about code not directly changed by the patch set — most do not, but the ones that do often have several such comments.

Hypothesis 1: Reviewers are getting told about bugs they didn't create. Sashiko's review protocol explicitly instructs the LLM to read surrounding code, not just the diff. That's good review practice — but it means the tool might flag pre-existing bugs in code the patch author merely touched, putting those problems in their inbox.

Hypothesis 2: The same pre-existing bugs surface repeatedly. If a known issue in a subsystem doesn't get fixed between review runs, every patch touching nearby code could trigger the same finding. That would create a steady drip of duplicate noise across the mailing list.

I pulled data from Sashiko's public API and tested both.

to post comments

does this undercount "complains about existing problem" ?

Posted Apr 2, 2026 15:55 UTC (Thu) by pm215 (subscriber, #98099) [Link] (2 responses)

Interesting data, but if I understand the methodology correctly it undercounts "tool complains about a problem not introduced in this patch". It looks for text like "not introduced in thus patch", "pre-existing", etc, so it relies on Sachiko itself flagging its reviews as about a problem that's already there. It won't count the cases where the tool complains about a pre-existing problem without mentioning that. As an example, this one I've linked in an earlier thread:

https://sashiko.dev/#/patchset/20260314-iio-light-vcnl403...

is entirely about preexisting problems (half of it is even commenting on code that's only in the diff context), but I think it would not have been counted by this analysis.

does this undercount "complains about existing problem" ?

Posted Apr 2, 2026 18:27 UTC (Thu) by bexelbie (subscriber, #114499) [Link] (1 responses)

The example you're providing would not count as "unrelated code" as I deliberately did not use an LLM to assess the type of finding. That could be done, but it felt like it was more "fraught with peril" if I wasn't going to go through an extensive training and verification step.

does this undercount "complains about existing problem" ?

Posted Apr 3, 2026 13:59 UTC (Fri) by pm215 (subscriber, #98099) [Link]

Mmm. Perhaps Sachiko would benefit from an interface like the online coverity one where you could mark reports as "correct", "false positive", "preexisting bug" and the like. At the moment presumably humans are looking at the reviews it produces and analysing them but there's no gathering of that data.

Unrelated bugs...

Posted Apr 9, 2026 15:17 UTC (Thu) by PaulMcKenney (✭ supporter ✭, #9624) [Link]

...still need to be fixed. That said, such bugs might not need to be fixed by the patch submitter, especially if that submitter is a newbie.