IP range flagging [LWN.net]

IP range flagging

Posted Feb 16, 2026 20:02 UTC (Mon) by MarcB (subscriber, #101804)
In reply to: IP range flagging by dagobayard
Parent article: Poisoning scraperbots with iocaine

This is indeed a stupid approach (tbh, I also find the whole concept of this "poisoning" quite pointless and unsustainable). As an example, Reddit blocks the office networks of the company I work at. Likely because they are in our large datacenter AS. But there is an easy workaround: Just get the information you are looking for via Google's AI search...

As to why this poisoning is pointless and unsustainable: AI access patterns are shifting - already have shifted to some degree - from scrapers to agents (I wouldn't be surprised if those agents started acting as scrapers as well). For many websites, especially technical ones, AI agents will likely become a very significant access vector very soon. For things like crates, npm, cpan, pypi, and so on, they might very well become the dominant mode.
Site operators can take an anti-AI stance, but it will ultimately affect their real users and it will not convince them to go back to browsers. Worst-case, the sites will do it like Reddit and get paid by the large AI companies, forever cementing their position in the market...

As a side-note: we literally ran into power problems because of AI bots. They started heavily scrapping countless, ancient web sites. Those sites got more requests in a few hours than they previously got in years - decades in some cases.

to post comments

IP range flagging

Posted Feb 16, 2026 21:56 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

The issue here is reciprocity. Bad AI crawlers _should_ get poisoned results, while well-behaved rate-limited crawlers should get good data.

This way, well-behaved players are rewarded and can outcompete the bad players.

IP range flagging

Posted Feb 17, 2026 21:23 UTC (Tue) by MarcB (subscriber, #101804) [Link] (5 responses)

But that is a mighty ambitious goal. For this to work, we would have to know who actually does the crawling and how the crawled data is traded. As far as I am concerned, the goal primarily is getting rid of the load caused by the crawlers.

For this to make sense, we would also have to be confident that we can reliably tell the good and the bad crawlers apart, 100%. Because it just takes a single undetected request by the bad bot to recognize the poisoned data - or the buyer of the data to just buy more data, from different scrapers and correlate it.

IP range flagging

Posted Feb 18, 2026 0:46 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

Good crawlers don't spam websites and obey robots.txt. Google and other large search providers also publish the IP ranges of their crawlers.

IP range flagging

Posted Feb 18, 2026 12:07 UTC (Wed) by taladar (subscriber, #68407) [Link] (3 responses)

Sure, but to detect that you would need something like a Web Application Firewall that keeps statistics of the requests seen recently and also would need to be able to know which requests from different IPs belong to the same bad crawler, not a trivial task.

IP range flagging

Posted Feb 18, 2026 18:15 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

It's actually an interesting task! Find blocks of IPs that obey robots.txt and moderate the request rate, then whitelist them.

IP range flagging

Posted Feb 19, 2026 14:34 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (1 responses)

And you've now created a market for blessed IP blocks :) .

IP range flagging

Posted Feb 19, 2026 18:21 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

And why not? It's all good as long as they _stay_ well-behaved.