Hands-off hosters?

Posted Feb 17, 2025 8:48 UTC (Mon) by taladar (subscriber, #68407)
Parent article: Fighting the AI scraperbot scourge

> Interestingly, scrapers almost never use IPv6

Presumably that is because the kind of hands-off hosters used by scrapers like that invest the minimum amount of effort into their hosting offerings. IPv6 is still seen as something optional by many for some unfathomable reason so they won't invest time in learning how to offer it.

Hands-off hosters?

Posted Feb 17, 2025 13:00 UTC (Mon) by ccr (guest, #142664) [Link]

I wouldn't be so sure about this. I've been experiencing similar scraping issues on my own site (which hosts, among other things, source code repositories) during last year or so. This has been enough to sometimes cause congestion due to hundreds of simultaneous connection attempts.

In my case, I've also noted that the scraping bots do not seem to use IPv6, but also that vast majority of the scraping originated from cloud services of certain nationality. All three of those cloud providers at least document IPv6 capabilities, so I'm not sure if the "blame" can be placed on hosting alone .. of course the scrapers of LWN may be different from who bombard my site, so .. shrug. :)

Eventually, after adjusting throttling settings few times, I decided to simply drop all connections originating from those networks completely in netfilter. By networks, I mean complete ranges assigned to those cloud providers AS. Been quieter after that.