Volume difference with older crawlers? [LWN.net]

Volume difference with older crawlers?

Posted Feb 16, 2026 13:44 UTC (Mon) by marcH (subscriber, #57642)
In reply to: Volume difference with older crawlers? by mb
Parent article: Poisoning scraperbots with iocaine

> After E-Mail became a nightmare due to the big corporations putting massive restrictions onto sending servers now it becomes a nightmare to run a simple web server that nobody cared about two years ago.

Email was doomed from the very beginning because:
- spam existed long before the internet for obvious and well documented economical[*] reasons
- email removed the sender costs and "friction" keeping pre-internet spam to (barely) manageable levels.

Having your inbox open by default to billions of strangers is flawed by design. No other communication tool does that, they all have some sort of approval process (or some way to inflict some cost on the sender side). Most mitigations for email spam involve some sort of approval concepts.

It's not clear yet whether AI will kill the web as we knew it but for sure the Internet was designed by smart "technologists" living in some sort of bubble isolated from economical and other "real-world" aspects.

[*] not just advertisements but also scams and other crime.

to post comments

Volume difference with older crawlers?

Posted Feb 16, 2026 17:59 UTC (Mon) by mb (subscriber, #50428) [Link] (2 responses)

>Email was doomed from the very

Well, some mail providers act like complete idiots. That has nothing to do with E-Mail as a technical thing.
Want a real word example that happened to me a week ago?

T-Online (Telekom) blocked my personal locked down (accessible only to me and a couple of people I know in person) mail server because I didn't send a mail to t-online addresses for a long time. That made them conclude my address was dialup. This was their "reason" for blocking me.

Sending too much mail -> blocked. You are a spammer.
Sending too little mail -> blocked. You are a dialup even though obviously you aren't.

I do implement every technique needed for modern mail delivery on my system.
That and explaining and proving to them that my system was not dialup and was private and low volume did not help.
They forced me to disclose my private home address in public to fulfill their idiotic corporate processes.

This has nothing to do with E-Mail being problematic.
This is purely caused by idiots doing idiotic things.
Just like AI crawler responsible persons do.

Volume difference with older crawlers?

Posted Feb 17, 2026 8:47 UTC (Tue) by taladar (subscriber, #68407) [Link] (1 responses)

They also now want you to host a website on the domain in your server IP's reverse DNS entry that has some sort of redirect to a website with a contact form for them to use in case of abuse.

I am certainly not going to run a webserver on every single server I run under the server's hostname (which is deliberately chosen to not be any of the services it hosts to make migrating those services easier) just because the T-Online mail admin team is too incompetent to use the abuse@ addresses like everyone else.

Volume difference with older crawlers?

Posted Feb 17, 2026 10:26 UTC (Tue) by Wol (subscriber, #4433) [Link]

T-Online? Germany?

If you start getting spam to that address file a formal complaint under your equivalent of the GDPR, that they leaked your private email address.

Dunno how many legs it has, but given the circumstances any reasonable regulator *should* conclude that they clearly breached the regs.

(Although whether said reasonable regulator actually has sufficient brain cells to *understand* the complaint is another matter...)

Cheers,
Wol