IP range flagging [LWN.net]

IP range flagging

Posted Feb 18, 2026 12:07 UTC (Wed) by taladar (subscriber, #68407)
In reply to: IP range flagging by Cyberax
Parent article: Poisoning scraperbots with iocaine

Sure, but to detect that you would need something like a Web Application Firewall that keeps statistics of the requests seen recently and also would need to be able to know which requests from different IPs belong to the same bad crawler, not a trivial task.

to post comments

IP range flagging

Posted Feb 18, 2026 18:15 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

It's actually an interesting task! Find blocks of IPs that obey robots.txt and moderate the request rate, then whitelist them.

IP range flagging

Posted Feb 19, 2026 14:34 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (1 responses)

And you've now created a market for blessed IP blocks :) .

IP range flagging

Posted Feb 19, 2026 18:21 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

And why not? It's all good as long as they _stay_ well-behaved.