Why?
Why?
Posted Aug 6, 2026 0:12 UTC (Thu) by aimannajjar (subscriber, #184277)In reply to: Why? by rgmoore
Parent article: An LLM agent attempts to compromise a project on GitHub
No simulation is going to be good enough to figure out if a LLM can pull off a social engineering attack; the only way to know for sure is to put it in the real world. Of course this is like saying you can't adequately model an epidemic using a simulation, so you have to release a real disease to know for sure. It may be true, but it doesn't make the research ethical!
It's not that it's unethical. It is not even a remotely interesting problem, I have never read a headline of an "incident" where researchers unleashed a phone center scammer to see if they will successfully steal a grandma's visa gift card by pretending to be an IRS agent. People deal with sloppy AI bots regularly online and no one feels impressed to be honest. Only American CEOs and the US government are fascinated by that "potential" of LLMs and are thrilled by those "incidents". It's a waste of money and time and a huge reason people loathe AI and frankly why China is winning the race.