Volume difference with older crawlers? [LWN.net]

Volume difference with older crawlers?

Posted Feb 16, 2026 11:37 UTC (Mon) by paulj (subscriber, #341)
In reply to: Volume difference with older crawlers? by pizza
Parent article: Poisoning scraperbots with iocaine

Wow, that is incredible. How many unique ASNs?

It will get to the point that you just have to block these abusive ASes in their entirety, every prefix they have.

to post comments

Volume difference with older crawlers?

Posted Feb 16, 2026 12:10 UTC (Mon) by pizza (subscriber, #46) [Link] (1 responses)

> Wow, that is incredible. How many unique ASNs?

Too many to bother with; the IPs I looked up were all assigned to residential ISPs and cell phone carriers.

(probably coming from [borderline] malicious apps or browser extensions on otherwise legit users' systems)

> It will get to the point that you just have to block these abusive ASes in their entirety, every prefix they have.

I think we're already pas the point where that has ceased to be effective.

Volume difference with older crawlers?

Posted Feb 16, 2026 12:45 UTC (Mon) by paulj (subscriber, #341) [Link]

Oh, wow. Ok.

If things really are as dire as painted by a number of people here, in the article and in the comments, then this may ultimately the end of the current Internet architecture, at the L3 -> L4 interface, at least so far as smaller content providers are concerned. Everything will have to go behind something that requires the requester to pay in some way - either by providing some proof of work done (as Anubis does), some other form of proof of extra effort (perhaps registering an account, but that's weak and the abusers will just do that if they need to) or some other micro-payment.

Wow. We live in interesting times.