Why? [LWN.net]

Why?

Posted Aug 5, 2026 22:36 UTC (Wed) by rgmoore (✭ supporter ✭, #75)
In reply to: Why? by mb
Parent article: An LLM agent attempts to compromise a project on GitHub

No simulation is going to be good enough to figure out if a LLM can pull off a social engineering attack; the only way to know for sure is to put it in the real world. Of course this is like saying you can't adequately model an epidemic using a simulation, so you have to release a real disease to know for sure. It may be true, but it doesn't make the research ethical! It's thinking like this that forces scientists to run all our research past Institutional Review Boards before we can start- often before we can even ask for a grant- to make sure our research meets ethical standards.

to post comments

Why?

Posted Aug 6, 2026 0:12 UTC (Thu) by aimannajjar (subscriber, #184277) [Link]

No simulation is going to be good enough to figure out if a LLM can pull off a social engineering attack; the only way to know for sure is to put it in the real world. Of course this is like saying you can't adequately model an epidemic using a simulation, so you have to release a real disease to know for sure. It may be true, but it doesn't make the research ethical!

It's not that it's unethical. It is not even a remotely interesting problem, I have never read a headline of an "incident" where researchers unleashed a phone center scammer to see if they will successfully steal a grandma's visa gift card by pretending to be an IRS agent. People deal with sloppy AI bots regularly online and no one feels impressed to be honest. Only American CEOs and the US government are fascinated by that "potential" of LLMs and are thrilled by those "incidents". It's a waste of money and time and a huge reason people loathe AI and frankly why China is winning the race.

Why?

Posted Aug 6, 2026 9:56 UTC (Thu) by NAR (subscriber, #1313) [Link]

this is like saying you can't adequately model an epidemic using a simulation, so you have to release a real disease to know for sure

Well, it kind of did happen, multiple times even in the Western World, e.g. in the UK or in the US.