Volume difference with older crawlers?

Around a year ago, I noticed a huge amount of guests on my forum and decided to take action. (This is a little disjointed; I haven't looked at what I did for some months now.)

On Linux, nf-ct-list is quite useful. Connections with SYN_RECV, CLOSE, CLOSE_WAIT and ESTABLISHED normally should be responded to fairly quickly. If their timers wind down 'too far', I delete the conn which changes its state to INVALID. My firewall drops INVALID packets first thing in mangle:PREROUTING. I also add the IP to my 'banditList' for ipset.

I employ (1) Univ. of Toulouse's blacklists and a couple others to block many IPs I never want traffic to or from, (2) the blacklist of stolen IPs, (3) IPs from would-be members who use invalid email addresses, (4) IPs from my sites' logs (mostly errors and bad traffic entries), (5) Snort alerts, (6) IPs from spam 'Contact Us' messages, and (7) nets from MS and others who claim to be scraping my sites 'to help me'.

Hosts with fewer than three accesses between the previous midnight and 10 minutes ago (humans usually have more than three accesses in ten minutes) get throttled to 2400 baud. I use a few escalating timeouts, up to ipset's ~21 day limit.

Those efforts dropped the 'guest' count on my forum from ~5000 to under 100 and often to under 30. I'm still blocking around 45 000 IPs, dropping connections from internet and rejecting connections to internet. And I'm blocking (via dnsmasq) around 1.3M domains from certain of UT's categories (using "local='/FQDN/'" to 'undefine' them).

It would be nice to find lists of datacenter/cloud IPs to block many of them (or at least recognize them and possibly treat them differently).

It ain't perfect, but it does reduce unwanted traffic.