Why? [LWN.net]

Why?

Posted Aug 5, 2026 20:48 UTC (Wed) by aimannajjar (subscriber, #184277)
Parent article: An LLM agent attempts to compromise a project on GitHub

I don't understand what the real purpose of those "experiments" is. Are they trying to manufacture some headlines? I know it's not about "measuring capability", because they could have prompted the LLM in a sandboxed environment. They could even clone git repos and provide identical private replicas of real world.

But even if it's for headlines, clearly "all publicity is good publicity" no longer holds true with all the data center opposition.

So again.. Why? I'm really baffled by American AI companies interests in this field, there's something bizarre that's motivating them to do things that are borderline psychotic.

to post comments

Why?

Posted Aug 5, 2026 21:11 UTC (Wed) by mb (subscriber, #50428) [Link] (3 responses)

Establishing a simulation environment is a lot more work than cloning a couple of git repositories. Given how many AI "enthusiasts" are actively exploiting publicly available resources, it's of no surprise to me that these people took the cheap route, too.

Why?

Posted Aug 5, 2026 22:36 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

No simulation is going to be good enough to figure out if a LLM can pull off a social engineering attack; the only way to know for sure is to put it in the real world. Of course this is like saying you can't adequately model an epidemic using a simulation, so you have to release a real disease to know for sure. It may be true, but it doesn't make the research ethical! It's thinking like this that forces scientists to run all our research past Institutional Review Boards before we can start- often before we can even ask for a grant- to make sure our research meets ethical standards.

Why?

Posted Aug 6, 2026 0:12 UTC (Thu) by aimannajjar (subscriber, #184277) [Link]

No simulation is going to be good enough to figure out if a LLM can pull off a social engineering attack; the only way to know for sure is to put it in the real world. Of course this is like saying you can't adequately model an epidemic using a simulation, so you have to release a real disease to know for sure. It may be true, but it doesn't make the research ethical!

It's not that it's unethical. It is not even a remotely interesting problem, I have never read a headline of an "incident" where researchers unleashed a phone center scammer to see if they will successfully steal a grandma's visa gift card by pretending to be an IRS agent. People deal with sloppy AI bots regularly online and no one feels impressed to be honest. Only American CEOs and the US government are fascinated by that "potential" of LLMs and are thrilled by those "incidents". It's a waste of money and time and a huge reason people loathe AI and frankly why China is winning the race.

Why?

Posted Aug 6, 2026 9:56 UTC (Thu) by NAR (subscriber, #1313) [Link]

this is like saying you can't adequately model an epidemic using a simulation, so you have to release a real disease to know for sure

Well, it kind of did happen, multiple times even in the Western World, e.g. in the UK or in the US.