does this undercount "complains about existing problem" ? [LWN.net]

does this undercount "complains about existing problem" ?

Posted Apr 2, 2026 15:55 UTC (Thu) by pm215 (subscriber, #98099)
Parent article: Exelbierd: What's actually in a Sashiko review?

Interesting data, but if I understand the methodology correctly it undercounts "tool complains about a problem not introduced in this patch". It looks for text like "not introduced in thus patch", "pre-existing", etc, so it relies on Sachiko itself flagging its reviews as about a problem that's already there. It won't count the cases where the tool complains about a pre-existing problem without mentioning that. As an example, this one I've linked in an earlier thread:

https://sashiko.dev/#/patchset/20260314-iio-light-vcnl403...

is entirely about preexisting problems (half of it is even commenting on code that's only in the diff context), but I think it would not have been counted by this analysis.

to post comments

does this undercount "complains about existing problem" ?

Posted Apr 2, 2026 18:27 UTC (Thu) by bexelbie (subscriber, #114499) [Link] (1 responses)

The example you're providing would not count as "unrelated code" as I deliberately did not use an LLM to assess the type of finding. That could be done, but it felt like it was more "fraught with peril" if I wasn't going to go through an extensive training and verification step.

does this undercount "complains about existing problem" ?

Posted Apr 3, 2026 13:59 UTC (Fri) by pm215 (subscriber, #98099) [Link]

Mmm. Perhaps Sachiko would benefit from an interface like the online coverity one where you could mark reports as "correct", "false positive", "preexisting bug" and the like. At the moment presumably humans are looking at the reviews it produces and analysing them but there's no gathering of that data.