IP range flagging [LWN.net]

IP range flagging

Posted Feb 17, 2026 21:23 UTC (Tue) by MarcB (subscriber, #101804)
In reply to: IP range flagging by Cyberax
Parent article: Poisoning scraperbots with iocaine

But that is a mighty ambitious goal. For this to work, we would have to know who actually does the crawling and how the crawled data is traded. As far as I am concerned, the goal primarily is getting rid of the load caused by the crawlers.

For this to make sense, we would also have to be confident that we can reliably tell the good and the bad crawlers apart, 100%. Because it just takes a single undetected request by the bad bot to recognize the poisoned data - or the buyer of the data to just buy more data, from different scrapers and correlate it.

to post comments

IP range flagging

Posted Feb 18, 2026 0:46 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

Good crawlers don't spam websites and obey robots.txt. Google and other large search providers also publish the IP ranges of their crawlers.

IP range flagging

Posted Feb 18, 2026 12:07 UTC (Wed) by taladar (subscriber, #68407) [Link] (3 responses)

Sure, but to detect that you would need something like a Web Application Firewall that keeps statistics of the requests seen recently and also would need to be able to know which requests from different IPs belong to the same bad crawler, not a trivial task.

IP range flagging

Posted Feb 18, 2026 18:15 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

It's actually an interesting task! Find blocks of IPs that obey robots.txt and moderate the request rate, then whitelist them.

IP range flagging

Posted Feb 19, 2026 14:34 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (1 responses)

And you've now created a market for blessed IP blocks :) .

IP range flagging

Posted Feb 19, 2026 18:21 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

And why not? It's all good as long as they _stay_ well-behaved.