Volume difference with older crawlers? [LWN.net]

Volume difference with older crawlers?

Posted Feb 15, 2026 5:44 UTC (Sun) by marcH (subscriber, #57642)
Parent article: Poisoning scraperbots with iocaine

> With the invention of large language models (LLMs), text on the web suddenly has an economic value that it didn't previously, which leads to the temptation to ignore those polite requests.

Is ignoring robots.txt and other conventions the only difference with pre-AI crawlers? I (wrongly?) guessed that they were crawling more frequently than before (to constantly educate themselves with the latest news etc.)

to post comments

Volume difference with older crawlers?

Posted Feb 15, 2026 8:09 UTC (Sun) by mb (subscriber, #50428) [Link] (28 responses)

The difference is that AI crawler traffic is a DDoS.

AI crawlers use a massive amount of IP addresses and hammer the systems dozens or hundreds of times per second, but only reuse an IP address once per day or so. Almost all of them also use a normal browser User-Agent header.

This makes it very hard to block an IP address. Even if you managed to detect that one IP address is probably an AI crawler, it won't contact you for another 24 hours or so. So if you want to block based on previous behavior of addresses, you need massive databases and massive firewall tables.
If I check my logs I can see that source IP addresses are 99% unique during an AI DDoS.

Volume difference with older crawlers?

Posted Feb 15, 2026 11:43 UTC (Sun) by marcH (subscriber, #57642) [Link] (9 responses)

Interesting, thanks! But why? Why do they need so much more traffic? Search engines already needed to know everything, no?

Volume difference with older crawlers?

Posted Feb 15, 2026 12:47 UTC (Sun) by excors (subscriber, #95769) [Link] (7 responses)

My unfounded speculation is that crawling the whole web is expensive and it's hard for a young search engine to make money, so they had to figure out how to crawl efficiently and cheaply. Eventually they grew into enormously rich companies, but they still had the technology and the culture of crawling efficiently and cooperating with server owners (via robots.txt, honest UA strings, etc). But now there's an insane amount of money in the AI industry, so everybody and their dog can get billions of dollars for their AI startup and vibe-code their own crawler, and they don't know how to do it efficiently and they don't care - they just care about scraping as many terabytes of text as possible, as quickly as possible, before the whole thing collapses.

More cynically they might recognise it's actually a competitive advantage for them to DDoS sites because it blocks other AI crawlers from getting the same content, and also means end users can't go to the original source and will have to make do with half-fabricated AI summaries of the content, ensuring all the ad revenue goes to the AI companies and not the content producers.

Volume difference with older crawlers?

Posted Feb 15, 2026 13:35 UTC (Sun) by mb (subscriber, #50428) [Link] (6 responses)

I really don't get what their goal is.
The traffic increased up until the point where basically *all* requests started to time out. Including the AI crawler generated requests.

That forced me to put the most expensive part (cgit) behind a simple cookie gate.
That works pretty well, but ever since I have installed this about half a year ago the AI crawlers keep hammering cgit.

Traffic of today is 99.6% hammering on the protected cgit area (and receiving a simple block message as a response).
They keep requesting deep URLs from the cgit, even though they didn't receive a cgit answer ever since I blocked it. So they must be keeping a database of what was there and what to hammer next. But without looking at the actual response.

But at least it doesn't invoke the cgi backend anymore, so the server can still handle it.
Not sure what I want to do, if it keeps increasing, though. And that's the trend.
Load average is at 0.6 and I really do not want to pay for a bigger machine.

Running small servers on the internet is a mess for hobbyists these days.
After E-Mail became a nightmare due to the big corporations putting massive restrictions onto sending servers now it becomes a nightmare to run a simple web server that nobody cared about two years ago.

These two things are currently destroying the Internet as a place where everybody can publish free speech without the help of big corporation platforms.

Volume difference with older crawlers?

Posted Feb 16, 2026 13:44 UTC (Mon) by marcH (subscriber, #57642) [Link] (3 responses)

> After E-Mail became a nightmare due to the big corporations putting massive restrictions onto sending servers now it becomes a nightmare to run a simple web server that nobody cared about two years ago.

Email was doomed from the very beginning because:
- spam existed long before the internet for obvious and well documented economical[*] reasons
- email removed the sender costs and "friction" keeping pre-internet spam to (barely) manageable levels.

Having your inbox open by default to billions of strangers is flawed by design. No other communication tool does that, they all have some sort of approval process (or some way to inflict some cost on the sender side). Most mitigations for email spam involve some sort of approval concepts.

It's not clear yet whether AI will kill the web as we knew it but for sure the Internet was designed by smart "technologists" living in some sort of bubble isolated from economical and other "real-world" aspects.

[*] not just advertisements but also scams and other crime.

Volume difference with older crawlers?

Posted Feb 16, 2026 17:59 UTC (Mon) by mb (subscriber, #50428) [Link] (2 responses)

>Email was doomed from the very

Well, some mail providers act like complete idiots. That has nothing to do with E-Mail as a technical thing.
Want a real word example that happened to me a week ago?

T-Online (Telekom) blocked my personal locked down (accessible only to me and a couple of people I know in person) mail server because I didn't send a mail to t-online addresses for a long time. That made them conclude my address was dialup. This was their "reason" for blocking me.

Sending too much mail -> blocked. You are a spammer.
Sending too little mail -> blocked. You are a dialup even though obviously you aren't.

I do implement every technique needed for modern mail delivery on my system.
That and explaining and proving to them that my system was not dialup and was private and low volume did not help.
They forced me to disclose my private home address in public to fulfill their idiotic corporate processes.

This has nothing to do with E-Mail being problematic.
This is purely caused by idiots doing idiotic things.
Just like AI crawler responsible persons do.

Volume difference with older crawlers?

Posted Feb 17, 2026 8:47 UTC (Tue) by taladar (subscriber, #68407) [Link] (1 responses)

They also now want you to host a website on the domain in your server IP's reverse DNS entry that has some sort of redirect to a website with a contact form for them to use in case of abuse.

I am certainly not going to run a webserver on every single server I run under the server's hostname (which is deliberately chosen to not be any of the services it hosts to make migrating those services easier) just because the T-Online mail admin team is too incompetent to use the abuse@ addresses like everyone else.

Volume difference with older crawlers?

Posted Feb 17, 2026 10:26 UTC (Tue) by Wol (subscriber, #4433) [Link]

T-Online? Germany?

If you start getting spam to that address file a formal complaint under your equivalent of the GDPR, that they leaked your private email address.

Dunno how many legs it has, but given the circumstances any reasonable regulator *should* conclude that they clearly breached the regs.

(Although whether said reasonable regulator actually has sufficient brain cells to *understand* the complaint is another matter...)

Cheers,
Wol

Volume difference with older crawlers?

Posted Feb 23, 2026 23:29 UTC (Mon) by anton (subscriber, #25547) [Link] (1 responses)

They keep requesting deep URLs from the cgit, even though they didn't receive a cgit answer ever since I blocked it.

We have a similar experience with Gforth's ViewCVS or CVSweb; one had been disabled for many years, the other was disabled when the scraperbots caused problems; both were obsolete, because Gforth has switched to git in 2014, with all of the CVS history included.

When I last looked, the scraperbots were still busily accessing various urls for some versions of files in CVS, or various directories with sorting options and the like encoded in the URL, several months after the service had been shut down.

One other thing I noticed is that at least one scraperbot doesn't use the IP protocol properly, and on the old web server we used at the time, every connection by a scraperbot consumed one of the available connections without it being released for a long time, if ever. After the few hundred connections available to the web server had been consumed, it no longer served anything. I don't remember in which state the connection hung, but one may be able to use their connection behaviour to identify them; identifying after the content has been served is probably not be very useful, but maybe the also connect in an unusual way on establishing the connection.

Volume difference with older crawlers?

Posted Mar 8, 2026 18:18 UTC (Sun) by fest3er (guest, #60379) [Link]

Around a year ago, I noticed a huge amount of guests on my forum and decided to take action. (This is a little disjointed; I haven't looked at what I did for some months now.)

On Linux, nf-ct-list is quite useful. Connections with SYN_RECV, CLOSE, CLOSE_WAIT and ESTABLISHED normally should be responded to fairly quickly. If their timers wind down 'too far', I delete the conn which changes its state to INVALID. My firewall drops INVALID packets first thing in mangle:PREROUTING. I also add the IP to my 'banditList' for ipset.

I employ (1) Univ. of Toulouse's blacklists and a couple others to block many IPs I never want traffic to or from, (2) the blacklist of stolen IPs, (3) IPs from would-be members who use invalid email addresses, (4) IPs from my sites' logs (mostly errors and bad traffic entries), (5) Snort alerts, (6) IPs from spam 'Contact Us' messages, and (7) nets from MS and others who claim to be scraping my sites 'to help me'.

Hosts with fewer than three accesses between the previous midnight and 10 minutes ago (humans usually have more than three accesses in ten minutes) get throttled to 2400 baud. I use a few escalating timeouts, up to ipset's ~21 day limit.

Those efforts dropped the 'guest' count on my forum from ~5000 to under 100 and often to under 30. I'm still blocking around 45 000 IPs, dropping connections from internet and rejecting connections to internet. And I'm blocking (via dnsmasq) around 1.3M domains from certain of UT's categories (using "local='/FQDN/'" to 'undefine' them).

It would be nice to find lists of datacenter/cloud IPs to block many of them (or at least recognize them and possibly treat them differently).

It ain't perfect, but it does reduce unwanted traffic.

Volume difference with older crawlers?

Posted Feb 16, 2026 9:21 UTC (Mon) by taladar (subscriber, #68407) [Link]

One aspect is probably users prompting the models to look at specific sites which turns the requests from one per service (e.g. OpenAI) to one per user of the service.

Volume difference with older crawlers?

Posted Feb 15, 2026 13:04 UTC (Sun) by pizza (subscriber, #46) [Link] (3 responses)

> If I check my logs I can see that source IP addresses are 99% unique during an AI DDoS.

I routinely see in excess of a million unique IP addresses per wave, each hitting at most two pages over the course of a day or two.

Volume difference with older crawlers?

Posted Feb 16, 2026 11:37 UTC (Mon) by paulj (subscriber, #341) [Link] (2 responses)

Wow, that is incredible. How many unique ASNs?

It will get to the point that you just have to block these abusive ASes in their entirety, every prefix they have.

Volume difference with older crawlers?

Posted Feb 16, 2026 12:10 UTC (Mon) by pizza (subscriber, #46) [Link] (1 responses)

> Wow, that is incredible. How many unique ASNs?

Too many to bother with; the IPs I looked up were all assigned to residential ISPs and cell phone carriers.

(probably coming from [borderline] malicious apps or browser extensions on otherwise legit users' systems)

> It will get to the point that you just have to block these abusive ASes in their entirety, every prefix they have.

I think we're already pas the point where that has ceased to be effective.

Volume difference with older crawlers?

Posted Feb 16, 2026 12:45 UTC (Mon) by paulj (subscriber, #341) [Link]

Oh, wow. Ok.

If things really are as dire as painted by a number of people here, in the article and in the comments, then this may ultimately the end of the current Internet architecture, at the L3 -> L4 interface, at least so far as smaller content providers are concerned. Everything will have to go behind something that requires the requester to pay in some way - either by providing some proof of work done (as Anubis does), some other form of proof of extra effort (perhaps registering an account, but that's weak and the abusers will just do that if they need to) or some other micro-payment.

Wow. We live in interesting times.

Volume difference with older crawlers?

Posted Feb 15, 2026 23:59 UTC (Sun) by Wol (subscriber, #4433) [Link] (13 responses)

>If I check my logs I can see that source IP addresses are 99% unique during an AI DDoS.

Sounds like a good anti-bot would simply be to delay answering any request from a new IP. 5 seconds? And then the more requests from that IP, the higher the priority goes.

Cheers,
Wol

Volume difference with older crawlers?

Posted Feb 16, 2026 0:55 UTC (Mon) by corbet (editor, #1) [Link] (11 responses)

I don't see how that would help. Humans get grumpy about a five-second delay... The bot just waits, then has the page it was after. Meanwhile your server, which is now trying to keep open five seconds worth of full-on scraper bot traffic, melts down into slag.

Volume difference with older crawlers?

Posted Feb 16, 2026 11:41 UTC (Mon) by paulj (subscriber, #341) [Link] (10 responses)

To be honest, at some point people are going to have to start sueing these highly abusive AI crawlers for the DDoS they are causing. There are laws covering DoS attacks that would apply if these companies are being reckless.

Volume difference with older crawlers?

Posted Feb 16, 2026 12:15 UTC (Mon) by pizza (subscriber, #46) [Link] (9 responses)

> To be honest, at some point people are going to have to start sueing these highly abusive AI crawlers for the DDoS they are causing. There are laws covering DoS attacks that would apply if these companies are being reckless.

So... how do I identify who to sue when they piggyback on residential IPs and pretend to be MacOS 15 (or a Pixel 8 phone, or Chrome on Windows 11 (or, or, or...)

The thing is, each crawler on its own is fine. The problem comes from everyone and their dog having a unique crawler all hitting at the same time.

Volume difference with older crawlers?

Posted Feb 17, 2026 12:05 UTC (Tue) by paulj (subscriber, #341) [Link] (8 responses)

You won't be able to take lines from your logs, and work out exactly which abusive AI-enshitifier is responsible. Or even if you could to some extent, you won't be able to prove an overall pattern of abuse from just that.

What may happen is that, one day, information comes out (by leaks, or just by the hubris of their own self-delusional sense of self-importance - listen to Altman for examples; or perhaps discovery in some other court case not related to web abuse) from one of these abusive AI enshitifiers (abusing the resources of the web, the Internet, the world in terms of energy) that details their shitty and abusive practices and finally provides the rope to hang them with wrt to how abusive they are of the resources of others on the Internet.

We must live in hope.

Volume difference with older crawlers?

Posted Feb 17, 2026 13:17 UTC (Tue) by Wol (subscriber, #4433) [Link]

Don't various ISPs ask for permission to use your router to anonymise web accesses?

Be rather tricky to do, but couldn't a bunch of small website owners sue one of them for damages, on the basis they are actively facilitating unwanted traffic and abuse?

I know I go on about fraud and abuse (small letter initial), but basing it upon the English laws of trespass, using someone else's property when you "knew or should have known" that permission would be refused, is a criminal offense. The mere fact these people are desperate to hide their identity is a blatant admission they know the "knew or should have known" bar is passed.

I don't know how far such a lawsuit would get in the UK (and the fact it would be a criminal suit means the DPP probably wouldn't be interested), but the "anti-social"ness is clear. It's just finding some way of turning the "at someone else's expense" into something you can sue over :-(

Cheers,
Wol

Volume difference with older crawlers?

Posted Feb 17, 2026 13:45 UTC (Tue) by marcH (subscriber, #57642) [Link] (6 responses)

I agree with the immorality but where exactly is the illegality of the abuse? Which "law of the internet" does an abusive, distributed crawler violate? Assuming you find some, what stops crawlers from avoiding those countries and to crawl only from other ones?

If they hijack unwilling client computers then sure, but do we have any indication that it is actually the case? If not then then what else? I mean on what grounds could anyone sue if not?

TCP/IP won against telcos because it was focused on the technical aspects, mostly ignoring the business/economical ones. Afraid that very old naivety is finally hurting a lot; way beyond SMTP.

Volume difference with older crawlers?

Posted Feb 17, 2026 14:37 UTC (Tue) by anselm (subscriber, #2796) [Link]

If you're a user of $AI_COMPANY's free offerings, it probably says somewhere in the 700-page license agreement (that you didn't read when you signed up for the service) that you consent to your computer serving as a proxy for $AI_COMPANY's web crawler .

Volume difference with older crawlers?

Posted Feb 17, 2026 15:48 UTC (Tue) by paulj (subscriber, #341) [Link] (4 responses)

Many jurisdictions have laws that make acts that abuse the resources of another's computer illegal, e.g. to make DDoSes illegal. As one example, in the UK, the "Computer Misuse Act, 1990", in section 3 " Unauthorised acts with intent to impair, or with recklessness as to impairing, operation of computer, etc. " makes it an offence in subsection (2)(a) to impair the operation of any computer; (b) prevent or hinder access to any program or data held in any computer; (c) impair the operation of any such program or the reliability of any such data.

It is an offence if the person making the act intended to cause things, OR they did those acts /recklessly/, i.e. they should have known 2(a) to (c) were likely consequences of their acts.

As Wol says, that these abusive AI-enshitifiers must resort to heavily disguising their DDoS actions just further proves their guilt. They *know* fine well the systems they access do not want this access, they know they are causing problems for those systems, precisely because they obviously have _already been blocked_ from accessing those systems directly; and then they go to the effort of disguising their access and continuing the abuse, recruiting vast armies of other people's computers to continue their abusive behaviour.

One day, these AI-enshitifiers are going to be hit with very large lawsuits. And some of these enshitifiers will turn out to be rather large tech companies, and I hope they end up paying out massive amounts in damages.

Volume difference with older crawlers?

Posted Feb 17, 2026 16:03 UTC (Tue) by paulj (subscriber, #341) [Link]

Oh, I assume the USA has similar laws. The US has easier provision for class-action lawsuits I think (they are - I gather - difficult to take in the UK and Ireland), and many of these AI-enshitifiers are based there.

It just needs a bunch of web content providers to get together, find a lawyer willing to take this on, then advertise the action to recruit even more web content providers who are being abused and then go after some of these awful AI-enshitifier people and give them a good spanking in the courts (and earn the lawyer a nice sum, and maybe a little bit for the abused each).

Volume difference with older crawlers?

Posted Feb 17, 2026 16:55 UTC (Tue) by marcH (subscriber, #57642) [Link] (2 responses)

Thanks! But... if lessons from blatant copyright violations are any indication, I'm afraid they will somehow get away with that too. "Move fast and break things" etc.

Right now they seem rich and powerful enough to dominate that _other_ oligopoly that was rich, powerful and corrupt enough to extend copyright laws to 70 years after the death of the author (lol). That's apparently achieved through a combination of forced "partnerships", buy-outs and other nasty arm-wrestling.

Exactly like with email, who cares about the small players.

Unless... the whole Ponzi scheme falls apart first. Interesting times either way.

Volume difference with older crawlers?

Posted Feb 17, 2026 17:32 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link]

Right now they seem rich and powerful enough to dominate that _other_ oligopoly that was rich, powerful and corrupt enough to extend copyright laws to 70 years after the death of the author (lol). That's apparently achieved through a combination of forced "partnerships", buy-outs and other nasty arm-wrestling.

A big part of this is that the AI evangelists have managed to sell the political elite on the idea that AI is the next big thing, so whichever country dominates AI will gain untold economic, political, and national security advantages. That gives them a plausible, actionable threat to relocate to whichever country does the most to make AI development easy: providing them with access to resources needed for massive AI data centers, letting them scrape copyrighted content as much as they want, etc. The copyright cartel is powerful and is already an economic engine, but they can't plausibly promise to let their host countries rule the world, so the AI industry is coming out on top.

Volume difference with older crawlers?

Posted Mar 15, 2026 7:35 UTC (Sun) by sammythesnake (guest, #17693) [Link]

> oligopoly that was rich, powerful and corrupt enough to extend copyright laws to 70 years after the death of the author (lol).

:-(

Volume difference with older crawlers?

Posted Feb 16, 2026 7:24 UTC (Mon) by mb (subscriber, #50428) [Link]

That just results in the resource consumption on the server to immediately skyrocket.
The port is used. Lots of memory is used already. Some CPU has been consumed already.
Other addresses won't stop hammering, while I hold the connections alive due to the delay.

And real users will be affected the most.

The only thing that really works for me is the exact opposite: Try to handle the request as quickly as possible.