email self-hosting

1.5M? That's almost 6000 /24s. With more IP assignees 'publishing' their large assignments in /24s, it becomes a PITA identifying who controls an IP address for both blocking and allowing. Examples: (1) which netblocks do TenCent control (to be blocked)? (2) Which netblocks do Crunchyroll and Netflix use (to be allowed)? (3) Which netblocks do MS' supposedly benevolent scrapers and probers use (to be blocked)?

Let's not confuse 'authorized' or 'controlling' with 'using. Bandits steal and use things they aren't authorized to use or control. Detecting their presence to shut them out is part of the guarding/protection task. Determining and notifying those who are authorized to use and control those remote resources is a different part of the guarding/protection task. Presently, there is no reliable way to identify who controls any block of IP addresses, or even which netblock an IP address is part of. For example, who controls 150.136.168.242? Crunchyroll seems to use it (found via tcpdump), but nothing points back to CR or to CR's upstream. This is just part of the reason I said that DNS/Whois/RDAP are broken, have far outlived their usefulness, and need to have a replacement engineered.

If someone is breaking into (or out of) your home, you don't really care who he is; you want to stop him. Bandits are not authorized to enter my home; they have neither warrant nor probable cause. Just the same, bandits are not allowed to enter (or leave) my network regardless of whether or not they are 'authorized' to use the source IP.

So manual effort and time required. Often more than should be needed.

• If your servers are generally intended for human consumption and many of the requests come from certain cloud providers, block that provider's netblocks and then allow individual IPs as needed. Human users rarely access web sites via cloud providers (AWS, Google, MS, Akamai, etc.)
• Scrape your web error logs. Block any IP that has more than, say, three 'not found' errors in the past 10 days. Human users rarely generate 'not found' errors.
• Scrape your web access logs. Block any IP that has only 1-3 accesses in the past 10 minutes. Human users generally access at least four things in a couple minutes when they visit a web site.
• Scrape your Snort/Suricata alerts; block any IP that generates an alert (chose your rules carefully).
• Scrape your firewall logs; with due consideration, block IPs that trigger DROPs and REJECTs.
• Peruse your conntrack table. Block IPs that don't respond to SYN_SENT and SYN_RECV within, say, 5 seconds; and block IPs that keep conns ESTABLISHED 'too long' (letting the timer run down too far). Also use nf-exp-delete to erase those conntrack entries
• Make use of the Spamhaus stolen netblocks, shadowwhisperer, firehol and other lists.
• Make use of the IPs in the Univ. of Toulouse categorizations (ads, pron, warez, et alia). I block almost 10k of these IPs.
• Add the domains in the Univ. of Toulouse categorizations (ads, pron, warez, et alia) to dnsmasq as local=/domain.to.block/. I currently block 1.3M domains outbound (grows dnsmasq's memory to 100MiB).
• Use ipset entries with escalating timeouts. Some types of entries should be allowed a 15 seconds timeout to allow for human error (e.g., a mistyped URL). After that, bandits (and those deemed non-human) are blocked for five minutes; if they reappear, their corner time is escalated to 10/20/60 minutes, 1 day, 1 week, and 24 days (ipset/netfilter max).

My scripts employ brute force methods. It would be more elegant—and probably more accurate—to employ statistical analysis of the frequency of IPs and their netblocks appearing in error and access logs.

I currently block around 40k IPs and netblocks. I throttle those and another 20k IPs and netblocks down to 2400 baud using HTB. (Yes, those 40k are blocked, but Scheiße happens.)