IP range flagging [LWN.net]

IP range flagging

Posted Feb 18, 2026 0:46 UTC (Wed) by Cyberax (✭ supporter ✭, #52523)
In reply to: IP range flagging by MarcB
Parent article: Poisoning scraperbots with iocaine

Good crawlers don't spam websites and obey robots.txt. Google and other large search providers also publish the IP ranges of their crawlers.

to post comments

IP range flagging

Posted Feb 18, 2026 12:07 UTC (Wed) by taladar (subscriber, #68407) [Link] (3 responses)

Sure, but to detect that you would need something like a Web Application Firewall that keeps statistics of the requests seen recently and also would need to be able to know which requests from different IPs belong to the same bad crawler, not a trivial task.

IP range flagging

Posted Feb 18, 2026 18:15 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

It's actually an interesting task! Find blocks of IPs that obey robots.txt and moderate the request rate, then whitelist them.

IP range flagging

Posted Feb 19, 2026 14:34 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (1 responses)

And you've now created a market for blessed IP blocks :) .

IP range flagging

Posted Feb 19, 2026 18:21 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

And why not? It's all good as long as they _stay_ well-behaved.