Volume difference with older crawlers? [LWN.net]

Volume difference with older crawlers?

Posted Feb 15, 2026 12:47 UTC (Sun) by excors (subscriber, #95769)
In reply to: Volume difference with older crawlers? by marcH
Parent article: Poisoning scraperbots with iocaine

My unfounded speculation is that crawling the whole web is expensive and it's hard for a young search engine to make money, so they had to figure out how to crawl efficiently and cheaply. Eventually they grew into enormously rich companies, but they still had the technology and the culture of crawling efficiently and cooperating with server owners (via robots.txt, honest UA strings, etc). But now there's an insane amount of money in the AI industry, so everybody and their dog can get billions of dollars for their AI startup and vibe-code their own crawler, and they don't know how to do it efficiently and they don't care - they just care about scraping as many terabytes of text as possible, as quickly as possible, before the whole thing collapses.

More cynically they might recognise it's actually a competitive advantage for them to DDoS sites because it blocks other AI crawlers from getting the same content, and also means end users can't go to the original source and will have to make do with half-fabricated AI summaries of the content, ensuring all the ad revenue goes to the AI companies and not the content producers.

to post comments

Volume difference with older crawlers?

Posted Feb 15, 2026 13:35 UTC (Sun) by mb (subscriber, #50428) [Link] (6 responses)

I really don't get what their goal is.
The traffic increased up until the point where basically *all* requests started to time out. Including the AI crawler generated requests.

That forced me to put the most expensive part (cgit) behind a simple cookie gate.
That works pretty well, but ever since I have installed this about half a year ago the AI crawlers keep hammering cgit.

Traffic of today is 99.6% hammering on the protected cgit area (and receiving a simple block message as a response).
They keep requesting deep URLs from the cgit, even though they didn't receive a cgit answer ever since I blocked it. So they must be keeping a database of what was there and what to hammer next. But without looking at the actual response.

But at least it doesn't invoke the cgi backend anymore, so the server can still handle it.
Not sure what I want to do, if it keeps increasing, though. And that's the trend.
Load average is at 0.6 and I really do not want to pay for a bigger machine.

Running small servers on the internet is a mess for hobbyists these days.
After E-Mail became a nightmare due to the big corporations putting massive restrictions onto sending servers now it becomes a nightmare to run a simple web server that nobody cared about two years ago.

These two things are currently destroying the Internet as a place where everybody can publish free speech without the help of big corporation platforms.

Volume difference with older crawlers?

Posted Feb 16, 2026 13:44 UTC (Mon) by marcH (subscriber, #57642) [Link] (3 responses)

> After E-Mail became a nightmare due to the big corporations putting massive restrictions onto sending servers now it becomes a nightmare to run a simple web server that nobody cared about two years ago.

Email was doomed from the very beginning because:
- spam existed long before the internet for obvious and well documented economical[*] reasons
- email removed the sender costs and "friction" keeping pre-internet spam to (barely) manageable levels.

Having your inbox open by default to billions of strangers is flawed by design. No other communication tool does that, they all have some sort of approval process (or some way to inflict some cost on the sender side). Most mitigations for email spam involve some sort of approval concepts.

It's not clear yet whether AI will kill the web as we knew it but for sure the Internet was designed by smart "technologists" living in some sort of bubble isolated from economical and other "real-world" aspects.

[*] not just advertisements but also scams and other crime.

Volume difference with older crawlers?

Posted Feb 16, 2026 17:59 UTC (Mon) by mb (subscriber, #50428) [Link] (2 responses)

>Email was doomed from the very

Well, some mail providers act like complete idiots. That has nothing to do with E-Mail as a technical thing.
Want a real word example that happened to me a week ago?

T-Online (Telekom) blocked my personal locked down (accessible only to me and a couple of people I know in person) mail server because I didn't send a mail to t-online addresses for a long time. That made them conclude my address was dialup. This was their "reason" for blocking me.

Sending too much mail -> blocked. You are a spammer.
Sending too little mail -> blocked. You are a dialup even though obviously you aren't.

I do implement every technique needed for modern mail delivery on my system.
That and explaining and proving to them that my system was not dialup and was private and low volume did not help.
They forced me to disclose my private home address in public to fulfill their idiotic corporate processes.

This has nothing to do with E-Mail being problematic.
This is purely caused by idiots doing idiotic things.
Just like AI crawler responsible persons do.

Volume difference with older crawlers?

Posted Feb 17, 2026 8:47 UTC (Tue) by taladar (subscriber, #68407) [Link] (1 responses)

They also now want you to host a website on the domain in your server IP's reverse DNS entry that has some sort of redirect to a website with a contact form for them to use in case of abuse.

I am certainly not going to run a webserver on every single server I run under the server's hostname (which is deliberately chosen to not be any of the services it hosts to make migrating those services easier) just because the T-Online mail admin team is too incompetent to use the abuse@ addresses like everyone else.

Volume difference with older crawlers?

Posted Feb 17, 2026 10:26 UTC (Tue) by Wol (subscriber, #4433) [Link]

T-Online? Germany?

If you start getting spam to that address file a formal complaint under your equivalent of the GDPR, that they leaked your private email address.

Dunno how many legs it has, but given the circumstances any reasonable regulator *should* conclude that they clearly breached the regs.

(Although whether said reasonable regulator actually has sufficient brain cells to *understand* the complaint is another matter...)

Cheers,
Wol

Volume difference with older crawlers?

Posted Feb 23, 2026 23:29 UTC (Mon) by anton (subscriber, #25547) [Link] (1 responses)

They keep requesting deep URLs from the cgit, even though they didn't receive a cgit answer ever since I blocked it.

We have a similar experience with Gforth's ViewCVS or CVSweb; one had been disabled for many years, the other was disabled when the scraperbots caused problems; both were obsolete, because Gforth has switched to git in 2014, with all of the CVS history included.

When I last looked, the scraperbots were still busily accessing various urls for some versions of files in CVS, or various directories with sorting options and the like encoded in the URL, several months after the service had been shut down.

One other thing I noticed is that at least one scraperbot doesn't use the IP protocol properly, and on the old web server we used at the time, every connection by a scraperbot consumed one of the available connections without it being released for a long time, if ever. After the few hundred connections available to the web server had been consumed, it no longer served anything. I don't remember in which state the connection hung, but one may be able to use their connection behaviour to identify them; identifying after the content has been served is probably not be very useful, but maybe the also connect in an unusual way on establishing the connection.

Volume difference with older crawlers?

Posted Mar 8, 2026 18:18 UTC (Sun) by fest3er (guest, #60379) [Link]

Around a year ago, I noticed a huge amount of guests on my forum and decided to take action. (This is a little disjointed; I haven't looked at what I did for some months now.)

On Linux, nf-ct-list is quite useful. Connections with SYN_RECV, CLOSE, CLOSE_WAIT and ESTABLISHED normally should be responded to fairly quickly. If their timers wind down 'too far', I delete the conn which changes its state to INVALID. My firewall drops INVALID packets first thing in mangle:PREROUTING. I also add the IP to my 'banditList' for ipset.

I employ (1) Univ. of Toulouse's blacklists and a couple others to block many IPs I never want traffic to or from, (2) the blacklist of stolen IPs, (3) IPs from would-be members who use invalid email addresses, (4) IPs from my sites' logs (mostly errors and bad traffic entries), (5) Snort alerts, (6) IPs from spam 'Contact Us' messages, and (7) nets from MS and others who claim to be scraping my sites 'to help me'.

Hosts with fewer than three accesses between the previous midnight and 10 minutes ago (humans usually have more than three accesses in ten minutes) get throttled to 2400 baud. I use a few escalating timeouts, up to ipset's ~21 day limit.

Those efforts dropped the 'guest' count on my forum from ~5000 to under 100 and often to under 30. I'm still blocking around 45 000 IPs, dropping connections from internet and rejecting connections to internet. And I'm blocking (via dnsmasq) around 1.3M domains from certain of UT's categories (using "local='/FQDN/'" to 'undefine' them).

It would be nice to find lists of datacenter/cloud IPs to block many of them (or at least recognize them and possibly treat them differently).

It ain't perfect, but it does reduce unwanted traffic.