another positive from the blog [LWN.net]

another positive from the blog

Posted Apr 26, 2026 3:30 UTC (Sun) by aphedges (subscriber, #171718)
In reply to: another positive from the blog by aphedges
Parent article: Firefox: The zero-days are numbered

I'd also like to note that Firefox's blog post doesn't have a baseline to show improvements against. They don't compare these vulnerabilities to if Opus 4.6 were run using the same setup. I'm not saying that Mythos Preview can't find vulnerabilities, but I feel there would need to be better experimental design to make the conclusion that it's better than previous models.

to post comments

another positive from the blog

Posted Apr 26, 2026 9:52 UTC (Sun) by malmedal (subscriber, #56172) [Link] (2 responses)

> I'd also like to note that Firefox's blog post doesn't have a baseline to show improvements against

Yes it does. It says Opus 4.6 found 22 vulnerabilities that they fixed in Firefox 148 and then Mythos found a further 271 that they fixed in Firefox 150.

another positive from the blog

Posted Apr 26, 2026 23:05 UTC (Sun) by aphedges (subscriber, #171718) [Link] (1 responses)

That isn't good experimental design. The models should be run on the same base under the same conditions, and the results should be analyzed for statistical significance.

As a recently former AI researcher, I know properly designed experiments are relatively rare within the field (and are often very difficult to conduct), but it very much weakens claims that many researchers make.

another positive from the blog

Posted Apr 27, 2026 9:33 UTC (Mon) by malmedal (subscriber, #56172) [Link]

> That isn't good experimental design. The models should be run on the same base under the same conditions, and the results should be analyzed for statistical significance.

The blog post is clearly not intended as academic research. The Firefox developers are not researchers following academic rules. They are actually productive people using Mythos to improve their software. It is still a very useful and timely data-point for decision-makers evaluating Mythos.

If you want a proper academic paper you can easily write it yourself, just take the blog post as an input along with other information you can find and follow the normal rules for "Secondary research".