Volume difference with older crawlers? [LWN.net]

Volume difference with older crawlers?

Posted Feb 23, 2026 23:29 UTC (Mon) by anton (subscriber, #25547)
In reply to: Volume difference with older crawlers? by mb
Parent article: Poisoning scraperbots with iocaine

They keep requesting deep URLs from the cgit, even though they didn't receive a cgit answer ever since I blocked it.

We have a similar experience with Gforth's ViewCVS or CVSweb; one had been disabled for many years, the other was disabled when the scraperbots caused problems; both were obsolete, because Gforth has switched to git in 2014, with all of the CVS history included.

When I last looked, the scraperbots were still busily accessing various urls for some versions of files in CVS, or various directories with sorting options and the like encoded in the URL, several months after the service had been shut down.

One other thing I noticed is that at least one scraperbot doesn't use the IP protocol properly, and on the old web server we used at the time, every connection by a scraperbot consumed one of the available connections without it being released for a long time, if ever. After the few hundred connections available to the web server had been consumed, it no longer served anything. I don't remember in which state the connection hung, but one may be able to use their connection behaviour to identify them; identifying after the content has been served is probably not be very useful, but maybe the also connect in an unusual way on establishing the connection.

to post comments

Volume difference with older crawlers?

Posted Mar 8, 2026 18:18 UTC (Sun) by fest3er (guest, #60379) [Link]

Around a year ago, I noticed a huge amount of guests on my forum and decided to take action. (This is a little disjointed; I haven't looked at what I did for some months now.)

On Linux, nf-ct-list is quite useful. Connections with SYN_RECV, CLOSE, CLOSE_WAIT and ESTABLISHED normally should be responded to fairly quickly. If their timers wind down 'too far', I delete the conn which changes its state to INVALID. My firewall drops INVALID packets first thing in mangle:PREROUTING. I also add the IP to my 'banditList' for ipset.

I employ (1) Univ. of Toulouse's blacklists and a couple others to block many IPs I never want traffic to or from, (2) the blacklist of stolen IPs, (3) IPs from would-be members who use invalid email addresses, (4) IPs from my sites' logs (mostly errors and bad traffic entries), (5) Snort alerts, (6) IPs from spam 'Contact Us' messages, and (7) nets from MS and others who claim to be scraping my sites 'to help me'.

Hosts with fewer than three accesses between the previous midnight and 10 minutes ago (humans usually have more than three accesses in ten minutes) get throttled to 2400 baud. I use a few escalating timeouts, up to ipset's ~21 day limit.

Those efforts dropped the 'guest' count on my forum from ~5000 to under 100 and often to under 30. I'm still blocking around 45 000 IPs, dropping connections from internet and rejecting connections to internet. And I'm blocking (via dnsmasq) around 1.3M domains from certain of UT's categories (using "local='/FQDN/'" to 'undefine' them).

It would be nice to find lists of datacenter/cloud IPs to block many of them (or at least recognize them and possibly treat them differently).

It ain't perfect, but it does reduce unwanted traffic.