Volume difference with older crawlers? [LWN.net]

Volume difference with older crawlers?

Posted Feb 15, 2026 13:04 UTC (Sun) by pizza (subscriber, #46)
In reply to: Volume difference with older crawlers? by mb
Parent article: Poisoning scraperbots with iocaine

> If I check my logs I can see that source IP addresses are 99% unique during an AI DDoS.

I routinely see in excess of a million unique IP addresses per wave, each hitting at most two pages over the course of a day or two.

to post comments

Volume difference with older crawlers?

Posted Feb 16, 2026 11:37 UTC (Mon) by paulj (subscriber, #341) [Link] (2 responses)

Wow, that is incredible. How many unique ASNs?

It will get to the point that you just have to block these abusive ASes in their entirety, every prefix they have.

Volume difference with older crawlers?

Posted Feb 16, 2026 12:10 UTC (Mon) by pizza (subscriber, #46) [Link] (1 responses)

> Wow, that is incredible. How many unique ASNs?

Too many to bother with; the IPs I looked up were all assigned to residential ISPs and cell phone carriers.

(probably coming from [borderline] malicious apps or browser extensions on otherwise legit users' systems)

> It will get to the point that you just have to block these abusive ASes in their entirety, every prefix they have.

I think we're already pas the point where that has ceased to be effective.

Volume difference with older crawlers?

Posted Feb 16, 2026 12:45 UTC (Mon) by paulj (subscriber, #341) [Link]

Oh, wow. Ok.

If things really are as dire as painted by a number of people here, in the article and in the comments, then this may ultimately the end of the current Internet architecture, at the L3 -> L4 interface, at least so far as smaller content providers are concerned. Everything will have to go behind something that requires the requester to pay in some way - either by providing some proof of work done (as Anubis does), some other form of proof of extra effort (perhaps registering an account, but that's weak and the abusers will just do that if they need to) or some other micro-payment.

Wow. We live in interesting times.