Why poison text? [LWN.net]

Why poison text?

Posted Feb 16, 2026 4:22 UTC (Mon) by pgp (subscriber, #134343)
Parent article: Poisoning scraperbots with iocaine

Instead of generating poisoned text, we should be hitting the LLMs where it hurts: poisoned code.

to post comments

Why poison text?

Posted Feb 16, 2026 11:43 UTC (Mon) by paulj (subscriber, #341) [Link] (16 responses)

A poison cgit-like tool be _awesome_. It serves real code, but scrambled ever so slightly - ideally, in such a way the code still compiles and the changes can only cause runtime problems, e.g. just randomly changing the senses of conditionals and what not.

Why poison text?

Posted Feb 16, 2026 17:35 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (14 responses)

I think the ideal way of messing it up would be to insert logical errors or just really inefficient algorithms. Perform calculations incorrectly, return wrong values, replace O(log(N)) algorithms with O(N), etc. Could you look at highly optimized numerical code and tell if it's calculating correctly? The big problem is that it would be hard to insert subtle errors automatically, and it would be a terrible waste of time to manually create flawed code just to mess with AI scraper bots. A more practical approach would be to keep around all your old, known buggy code and feed that to the bots instead of the fixed versions. You'd just have to tag old versions as suitable for bot feed when they have a particularly nasty logic bug.

I think you could do something similar with factual information: serve versions of your pages that have key facts altered. Imagine a version of Wikipedia where all the names and dates were shuffled around, and the references replaced with links to similar junk. It would take an enormous QC effort to figure out when you were being served bogus information. Again, if you were looking at a fake Wikipedia page that wasn't in your field of expertise, would you be able to tell?

Why poison text?

Posted Feb 16, 2026 17:44 UTC (Mon) by paulj (subscriber, #341) [Link] (11 responses)

Replacing algorithms would be cool, though it has to be really low cost to do, if you want to do it dynamically. So... replacing operators with equivalent ones as far as the compiler is concerned.

An AI-loving colleague of mine came to me the other, to regale sceptical-me of how someone on the Internet had managed to recreate the functionality of GCC in some low number of days with the help of AI. He thought this proved how wonderful AI was, until I pointed out the AIs were /trained/ on the GCC source-code, so *of course* you could coax out the functionality of GCC from them - all it proved was they'd discovered a much, much slower and incredibly energy-wasteful way of downloading something that somewhat resembled GCC (and good luck testing it!).

Why poison text?

Posted Feb 16, 2026 18:40 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (10 responses)

An AI-loving colleague of mine came to me the other, to regale sceptical-me of how someone on the Internet had managed to recreate the functionality of GCC in some low number of days with the help of AI. He thought this proved how wonderful AI was, until I pointed out the AIs were /trained/ on the GCC source-code, so *of course* you could coax out the functionality of GCC from them - all it proved was they'd discovered a much, much slower and incredibly energy-wasteful way of downloading something that somewhat resembled GCC (and good luck testing it!).

I actually read their whole blog post, and it's a bit better than you're making it out to be. Their goal was to write a C compiler in Rust, so it's not clear how useful having the complete code to gcc would be. Even if they have the complete code for gcc, transpiling it to Rust would be an impressive accomplishment. You can argue that a C compiler written in Rust is pointless, but it's plausible for use as a test case. Having a standard for comparison makes it easier to judge the results. Of course that also makes it easier to create, since you basically have your test cases made for you.

That said, the compiler they wound up with was at best an early prototype. The most obvious problem was that it wasn't remotely complete. It couldn't compile all the programs they tested it on, most notably the Linux kernel. It also only compiled C to some intermediate format (I think it was assembly, but I'm not 100% sure) and had to depend on an existing assembler and linker. Also, the output was very poor quality; it was slower than what gcc turns out with all optimizations turned off.

Why poison text?

Posted Feb 16, 2026 19:47 UTC (Mon) by excors (subscriber, #95769) [Link] (5 responses)

See also discussion in https://lwn.net/Articles/1057604/ . I think "early prototype" is overselling it, because that suggests it could be useful as a starting point for an eventually complete compiler - but I believe it can't because it's too buggy, too complex, too poorly architected, too full of unmarked placeholder implementations that are just correct enough to pass the test suite but will break when used slightly differently, etc. It's beyond the current AI system's ability to improve further, and a human developer would probably spend more time chasing bugs and rewriting poorly-designed code than if they started from scratch.

How much time is actually spent writing code?

Posted Feb 16, 2026 20:40 UTC (Mon) by marcH (subscriber, #57642) [Link] (4 responses)

> a human developer would probably spend more time chasing bugs and rewriting poorly-designed code than if they started from scratch.

I'm afraid even AI-haters keep missing the biggest elephant in the room: _even before/without AI_, which fraction of the time is actually spent writing a first, incomplete prototype compared to all the rest? That is: testing it, debugging it, fixing it, reviewing it, discussing it, evolving it, throwing away most of it and starting again? For high-profile and successful, long-lasting projects, I bet it's less than 10%.

It's like people claiming AI will produce software have never read https://en.wikipedia.org/wiki/The_Mythical_Man-Month or anything like it.

I think this time distortion happens because this is a fun part. It's like spending 95% of your time + a fraction of your sleep on parenting chores but the emotions you experience during 5% of the time make it all worth it.

Back to AI writing code and the harsh reality of the wall clock, what happens _economically_ when you make ten times faster something that... lasted only 10% of the total time? And increased the cost of most other phases? And removed all the fun from the job? Assuming an actually competitive market (not a given...), I have a prediction.

Granted, there is a lot of Visual Basic etc. that has never been written by more than one person, read by less than one and that person's main job is not software engineering. Maybe not engineering at all. I think _this_ is where AI-generated code might have value.

Also, AI has many other applications than writing code, including non-generative AI applications. These get much less hype and capital: probably because they promise much less and deliver a lot more :-)

How much time is actually spent writing code?

Posted Feb 16, 2026 22:24 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

Granted, there is a lot of Visual Basic etc. that has never been written by more than one person, read by less than one and that person's main job is not software engineering. Maybe not engineering at all. I think _this_ is where AI-generated code might have value.

I fall squarely into the category of people who program for work only occasionally, and I'm skeptical that AI would help that much. The hard part for me has always been understanding the problem I'm trying to solve in enough detail to figure out what a solution looks like. Actually implementing whatever solution I come up with is usually pretty easy in comparison. In practice, understanding the problem in enough detail is an iterative process, where I think I've solved it only to discover there was an unexpected corner case I hadn't known enough to consider when I started out. I think that gets back to something Fred Brooks said in TMMM: a lot of debugging is actually debugging the specification. An AI coding assistant might be nice, but it isn't going to debug the specification for me.

How much time is actually spent writing code?

Posted Feb 16, 2026 22:48 UTC (Mon) by marcH (subscriber, #57642) [Link]

> I think that gets back to something Fred Brooks said in TMMM: a lot of debugging is actually debugging the specification. An AI coding assistant might be nice, but it isn't going to debug the specification for me.

It is well-known that "the customer never knows what they want" and that a large part of the work and value of IT consultants is to "extract" it from them.

AI fans will immediately tell you that of course LLMs can have that sort of discussion - for a MUCH cheaper price.
https://shumer.dev/something-big-is-happening

I have no idea.

How much time is actually spent writing code?

Posted Feb 17, 2026 8:40 UTC (Tue) by taladar (subscriber, #68407) [Link]

It is also not as if you write the prototype to have a prototype, you write it so you figure out the things you need to pay attention to in the architecture of the later full implementation.

How much time is actually spent writing code?

Posted Feb 17, 2026 9:41 UTC (Tue) by Wol (subscriber, #4433) [Link]

> Granted, there is a lot of Visual Basic etc. that has never been written by more than one person, read by less than one and that person's main job is not software engineering. Maybe not engineering at all. I think _this_ is where AI-generated code might have value.

And given my experience working currently as a "VBA Engineer", I'm not sure I agree with you ... in any company of any size, you need specialist end users, "Subject Matter Experts" as we call them, and any AI written code needs reviewing. In my only experience of doing so, I think I reduced about 100 lines of AI code to about 10 lines of company-standards-compliant code.

Cheers,
Wol

Why poison text?

Posted Feb 17, 2026 10:15 UTC (Tue) by paulj (subscriber, #341) [Link] (3 responses)

Yeah, only just noticed the LWN article on the same thing after I made that comment. ;) IIUC they used the code (inc. the very comprehensive test-suite) as a blackbox to functionally replicate its workings.

I remain sceptical. Probing a very functional (i.e., leaves no side-effects on the world) bit of code and writing code that replicates the discovered mappings of inputs to outputs, thus replicating the programme, I mean... you could do that with other algorithms, and probably much more efficiently. E.g., some kind of AFL-fuzz derivative searcher. They used an LLM to basically search the state space instead, transforming the knowledge _of the original source_ encoded into into another language, and using the _original code (inc. test-suite)_ to validate the transforms.

And now the AI-fanboys are touting this as some massive advance and how AI will be writing all the code soon.

I watch the AI-fanboys produce mountains of buggy slop, that they do not understand, and which is atrophying their own coding skills, and I sigh. I give them answers to things they ask me, which they will ignore and instead spend a week with their AI producing half-arsed crap before they come back to me all triumphant with the poor approximation of the answer it has produced. Sigh.

Your last paragraph describing the result of this latest, greatest, amazing advance in AI sounds like it fits that pattern!

The AI-fanboys want to turn coding into something that produces the same buggy results (perhaps worse) as before, except now requiring several racks worth of expensive silicon and kWs of power per "vibe dev".

Sigh, sigh sigh.

Why poison text?

Posted Feb 17, 2026 10:19 UTC (Tue) by paulj (subscriber, #341) [Link] (2 responses)

Sorry, that should be "100+ kW".

Why poison text?

Posted Feb 17, 2026 12:56 UTC (Tue) by Wol (subscriber, #4433) [Link] (1 responses)

Shouldn't that be "100+ kWh"?

Cheers,
Wol

Why poison text?

Posted Feb 17, 2026 13:23 UTC (Tue) by paulj (subscriber, #341) [Link]

No. If people are coding, over whatever amount of time, they're using energy at whatever rate these massively wasteful racks of GPUs consume. kWh is a (slightly odd) unit of energy, the integral of W over a specified period of time. Over unspecified, whatever amounts of time, we can't give an amount of energy used, but we can talk about a rate.

If you wanted to use kWh, you'd have to change my comment to something like "requiring several racks worth of expensive silicon and 800+ kWhs of power per "vibe dev" per day". But... why do that? Just divide the terms in that sentence by 'day', and the "per day" disappears, as does the 'h' in kWh. Or another way: Why pick 'per day'? Why not per week? Or per month? Or per hour? or ....

Better still, instead of doing "X energy per second per hour for each vibe dev per day" - which is just obviously silly in terms of all the units of time being multiplied and divided - just get rid of the extra, redundant additions of time and just say "X energy per second", i.e. W. ;)

Why poison text?

Posted Feb 16, 2026 18:09 UTC (Mon) by mb (subscriber, #50428) [Link] (1 responses)

> The big problem is that it would be hard to insert subtle errors automatically

This is the *perfect* use case for AI.
Subtle errors happen to me all the time when using AI to generate code.
If there's one thing LLM AI models are really really good it, then it's introducing subtle errors.

> A more practical approach would be to keep around all your old, known buggy code and feed that to the bots instead of the fixed versions

That's what everybody does. It's called a git repository.
AI crawlers go down *deep* into the history of git repositories.

Why poison text?

Posted Feb 17, 2026 8:43 UTC (Tue) by taladar (subscriber, #68407) [Link]

But AI is too slow for this use case. You want something efficient so the other people waste their computational effort, you don't want to waste your own computational effort or money.

Why poison text?

Posted Feb 16, 2026 17:44 UTC (Mon) by mb (subscriber, #50428) [Link]

I have no problem with them taking the content and putting it into their model. Sending my content to them is not the problem.

Sabotaging that by feeding incorrect results doesn't solve my problem. They just keep going. Since months I send something like "You have been blocked, because you are probably a bot" back to them. Yet, they simply don't stop crawling. I have hoped that at some point they must stop. But they don't. They do not care about what content they get. Yesterday I sent about 20 GiB of this text out to crawlers.

For me the problem would be solved if they started to crawl the web like every other web search engine in the decades before.
There's *huge* a difference between sending 20 GiB to non-DDoS Google crawler per day and sending 20 GiB to DDoS AI crawler that burn the system to a crisp due to the high amount of concurrent accesses.