Self Incriminating [LWN.net]

Self Incriminating

Posted Aug 7, 2026 20:30 UTC (Fri) by kleptog (subscriber, #1183)
In reply to: Self Incriminating by rgmoore
Parent article: An LLM agent attempts to compromise a project on GitHub

> But that doesn't match what these researchers did. They didn't discuss this with the project they attacked before starting. As far as we can tell, they didn't give any kind of notice to anyone- victim, regulator, ethics committee, etc.- before getting started.

Correct. Because the test wasn't supposed to attack anyone on the internet at all. They have run these tests many times before without issue. It's a bit hard to warn any victims if you didn't plan on the model attacking anything. The task was to solve a internal cyber-challenge and it went rogue and attacked public services instead. They declared a security incident and investigated and they published the security incident and this blog is the result.

You don't need to ask an ethics committee to hack your own systems.

Interestingly, they suggest it was helped by the fact that the instructions were incorrect and they had asked the AI to attack a host that was explicitly out of scope. So it decided to get creative.

If you want to know how they intend to stop this happening in the future, read the report.

to post comments

Self Incriminating

Posted Aug 7, 2026 20:58 UTC (Fri) by mb (subscriber, #50428) [Link]

Yes. In any but AI context this would be a clear problem and a clear failure on the researcher side.
Privilege escalation in the kernel or any other normal application? A HUGE deal, even if nobody was immediately harmed.
Privilege escalation in an AI test setup? Meh. "Nobody was harmed".

>The task was to solve a internal cyber-challenge

That's good. But apparently they failed to pull the Ethernet cable to the Internet before hitting start.
This is not the first time stuff like this happened and this has to stop now.

Open Source project maintainers are already borderline overloaded. There is absolutely no excuse to attack them with AI, no matter if "accidentally" or not. Take effective precautions.