Thanks for your efforts and keep up the good work! [LWN.net]

Thanks for your efforts and keep up the good work!

Posted Jul 10, 2026 16:08 UTC (Fri) by wtarreau (subscriber, #51152)
Parent article: An update on the scraper situation

Hi Jon,

I think that many of us here are totally aware of the problem these bots and proxies are causing to web sites like this one, and the difficulties in fighting them. At least I can say that I haven't noticed anything abnormal on the site here, so your actions were fine from the user experience perspective. Thanks for this!

You're right, it's important never to publicly explain the counter-measures that you apply. Very often some are extremely simple and effective (some easy tricks I've deployed 9 months ago that I imagined would only last one week are still working fine). Also they're often very specific to the site and would hardly adapt to other ones (except for the main principle), so there's little to share by explaining everyone how your specific site is fighting these.

I noticed a 30% drop of traffic on July 2nd, after a 50% one on June 25th that I couldn't explain (mostly attributed to user-agent "sleepbot"). So yes, it seems that such networks are progressively getting dismantled, probably to re-appear somewhere else soon, given that infected browsers (and their unsuspecting users) are just waiting for another C&C to take care of them :-/

I must confess I'm a bit worried about the risk of losing a lot of legit content indexing on the net in the coming years due to installed counter-measures against non-humans. If sites cannot be found via search engines it will become a problem. All this due to AI startups racing in training their own models (or variants).

Maybe it would work better to set up a static central registry of the whole internet's contents that could be scraped by such companies as much as they want without killing small web sites. It could still take a lot of time before we start to see something like this happen though.

to post comments

Thanks for your efforts and keep up the good work!

Posted Jul 10, 2026 17:15 UTC (Fri) by daroc (editor, #160859) [Link] (1 responses)

The central registry you propose sort of exists in the form of Common Crawl: https://commoncrawl.org/

The idea is that you contribute resources to their project, they have _one_ scraper that downloads web content in a polite way (respecting rate limits and robots.txt), and then anyone who wants to can make use of the scraped content without having to re-scrape it.

There are some problems with the approach, but I'm generally pretty happy when I see Common Crawl go by in the server logs because I know that's a bunch of unrelated projects that don't need to send us multiple requests.

Thanks for your efforts and keep up the good work!

Posted Jul 10, 2026 17:46 UTC (Fri) by wtarreau (subscriber, #51152) [Link]

> The central registry you propose sort of exists in the form of Common Crawl: https://commoncrawl.org/

Oh, thanks for the link, I wasn't aware. I'll try to make sure not to block that one!

Thanks for your efforts and keep up the good work!

Posted Aug 6, 2026 23:34 UTC (Thu) by fest3er (guest, #60379) [Link] (1 responses)

In my opinion, they aren't scrapers or 'bots. I call all of them internet bandits, legitimate search engines and archivers notwithstanding; they are robbers and thieves, stealing whatever they can to sell to fences who buy and resell stolen property.

I developed my own bandit blocking methods. At one point, my firewall was blocking around 1.2 million IP addresses (imagine that). I made adjustments. Presently, it blocks 33k IPs, 600 /24s, 8000 /16s, and 700 other-sized net blocks. In fact, I presently have over 2M IPs collected, which break down to 690k /24s and 46k /16s. The IPs break down to:

1.7M with one access,
230k with two accesses each,
194k with 3-10,000 accesses each,
50 with over 10,000 each, and
one with over 65,000 accesses.

Without these methods, phpBB's 'users in the last 5 minutes' would have exceeded 5000, and probably would've been over 10k. There would've been 10-50 established TCP conns and many more in close/wait states.

With my methods, there are only several established TCP conns and 30 or more conns in close or wait states. Traffic over the past week has averaged around 16kB/s.

For sure, my crude methods block legitimate users; I whitelist them should they contact me. I'd like to fine-tune my processes, but don't necessarily have the time or knowledge (like employing statistical processing which should reveal the difference between human and 'bot accesses). I'm tempted to redirect some of them to a honey pot and/or a tar pit.

The legitimate worldwide 'internet services' industry needs to band together to address the problem. Here are a few of the things that came to mind (in no particular order):

Unify world-wide RDAP data; each region seems to record data differently, and seems to record different (or not all) data.
Expand RDAP to make it easy for all internet users to identify the assignees who use IP addresses and netblocks, returning country, and company name where applicable or the person's name.
Expand RDAP to identify standard uses and statuses of netblocks and IPs (end users, VPS farms, cloud server farms, web server farms, search engines, stolen nets, gateway routers, et alia), and require assignees to keep that data up-to-date.
Provide tools (source code) so all internet users can look for general parity between incoming and outgoing traffic. (Have my private network or any of my systems been hijacked?)
Require IP assignees to identify FQDNs that resolve to each of their IPs/nets.
Require FQDN owners to identify all IPs that each FQDN resolves to.
Require cloud providers to monitor their systems for improper use (such as scraping and probing) and block/ban such users.
Prohibit companies from claiming to scrape and probe IP and sites "for your benefit" when they are really doing it for their own benefit and profit.
Require ISPs and others to allow end users to easily assign/relate properly acquired FQDNs to their IPs.

We users of the internet deserve to be able to easily identify who is connecting to our private internetworks. For example, if Acme Systems Engineering is probing my gateway in a harrassing way, I should be able to blacklist all of their IPs. If someone using Peaker Cloud Computing Services is hammering away at my gateway using multiple IPs, I should be able to block of of Peaker's IPs used in their cloud services. The system should be nearly trivial to use and should return data that are easily read by humans and easily parsed by systems.

[This is probably enough for one comment....]

Thanks for your efforts and keep up the good work!

Posted Aug 7, 2026 5:31 UTC (Fri) by anselm (subscriber, #2796) [Link]

Here are a few of the things that came to mind (in no particular order):

[…]

While these proposals are undoubtedly all well-intentioned and make a lot of sense, they sound like way too much hassle to get everyone to comply with. In fact, we should keep things simple and straightforward, and merely require the ubiquitous implementation of RFC 3514. Tadaa! Problem solved.