Self Incriminating [LWN.net]

Self Incriminating

Posted Aug 7, 2026 12:52 UTC (Fri) by kleptog (subscriber, #1183)
In reply to: Self Incriminating by mb
Parent article: An LLM agent attempts to compromise a project on GitHub

> >As with all criminal matters, the intent matters (of the researchers to be clear, the AI has no intent). The purpose was research

> So if I want to study bank robbery and write a paper about it, an actual bank robbery under these conditions is fine then?

Of course not, criminal prosecutions are way more nuanced than that. It's not black and white, there are all sorts of defenses that can be used, the prosecution has to weigh up the chance of success versus the public interest. Prosecutors don't have unlimited time or money.

In Civil Law countries this process is somewhat more structured, but the end result is the same.

In this case the researchers didn't even intend for their model to do all those things. Which is worthy of a report.

Hypothetically if you were a research organisation that wanted to study bank robberies and you asked a bank if you could give it a go and they said yes and it worked, yes it would be legal. You could even write a paper about it if you wanted.

> At least peoples time has been wasted

No-one has the right to not have their time wasted. Unfortunatly.

to post comments

Self Incriminating

Posted Aug 7, 2026 14:25 UTC (Fri) by mb (subscriber, #50428) [Link]

No-one has the right to not have their time wasted

Moving goal posts. I replied to

no actual harm caused

Actual harm was caused. No matter what the law says about that and no matter whether it was legal or not.

Hypothetically if you were a research organisation that wanted to study bank robberies and you asked a bank if you could give it a go and they said yes and it worked

That's exactly the point. Where the Open Source maintainers asked? No, they weren't. I get it why they didn't ask them. Their study setup would not have worked. But that's not my problem. Design a better study instead of exploiting Open Source projects.

They were attacked without knowledge and their time was wasted. I would be very upset if they had wasted my time.

There's a reason that in lots of institutions there are ethics commissions that typically have to be asked before performing a social experiment or any other experiment on living creatures. Not everything that is legal is Ok.

Self Incriminating

Posted Aug 7, 2026 15:52 UTC (Fri) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

Hypothetically if you were a research organisation that wanted to study bank robberies and you asked a bank if you could give it a go and they said yes and it worked, yes it would be legal.

But that doesn't match what these researchers did. They didn't discuss this with the project they attacked before starting. As far as we can tell, they didn't give any kind of notice to anyone- victim, regulator, ethics committee, etc.- before getting started. It's understandable why they didn't give the victim advance notice- knowing an attack was coming would put the target on guard, potentially resulting in a false negative- but that doesn't make it OK.

There's a reason the sciences have adopted a requirement to run proposed research through an ethics review before starting. Researchers can get ethical permission to perform experiments that involve deliberately concealing information from the subjects- blinded studies are common, and some psychology research requires deliberately deceiving the subjects about its purpose- but it requires extra care and will be carefully scrutinized by ethics committees. This kind of thing, where someone is involved in the research without any permission or notice, would almost never be allowed.

Self Incriminating

Posted Aug 7, 2026 20:30 UTC (Fri) by kleptog (subscriber, #1183) [Link] (1 responses)

> But that doesn't match what these researchers did. They didn't discuss this with the project they attacked before starting. As far as we can tell, they didn't give any kind of notice to anyone- victim, regulator, ethics committee, etc.- before getting started.

Correct. Because the test wasn't supposed to attack anyone on the internet at all. They have run these tests many times before without issue. It's a bit hard to warn any victims if you didn't plan on the model attacking anything. The task was to solve a internal cyber-challenge and it went rogue and attacked public services instead. They declared a security incident and investigated and they published the security incident and this blog is the result.

You don't need to ask an ethics committee to hack your own systems.

Interestingly, they suggest it was helped by the fact that the instructions were incorrect and they had asked the AI to attack a host that was explicitly out of scope. So it decided to get creative.

If you want to know how they intend to stop this happening in the future, read the report.

Self Incriminating

Posted Aug 7, 2026 20:58 UTC (Fri) by mb (subscriber, #50428) [Link]

Yes. In any but AI context this would be a clear problem and a clear failure on the researcher side.
Privilege escalation in the kernel or any other normal application? A HUGE deal, even if nobody was immediately harmed.
Privilege escalation in an AI test setup? Meh. "Nobody was harmed".

>The task was to solve a internal cyber-challenge

That's good. But apparently they failed to pull the Ethernet cable to the Internet before hitting start.
This is not the first time stuff like this happened and this has to stop now.

Open Source project maintainers are already borderline overloaded. There is absolutely no excuse to attack them with AI, no matter if "accidentally" or not. Take effective precautions.