Thanks for your efforts and keep up the good work!

Thanks for your efforts and keep up the good work!

In my opinion, they aren't scrapers or 'bots. I call all of them internet bandits, legitimate search engines and archivers notwithstanding; they are robbers and thieves, stealing whatever they can to sell to fences who buy and resell stolen property.

I developed my own bandit blocking methods. At one point, my firewall was blocking around 1.2 million IP addresses (imagine that). I made adjustments. Presently, it blocks 33k IPs, 600 /24s, 8000 /16s, and 700 other-sized net blocks. In fact, I presently have over 2M IPs collected, which break down to 690k /24s and 46k /16s. The IPs break down to:

• 1.7M with one access,
• 230k with two accesses each,
• 194k with 3-10,000 accesses each,
• 50 with over 10,000 each, and
• one with over 65,000 accesses.

Without these methods, phpBB's 'users in the last 5 minutes' would have exceeded 5000, and probably would've been over 10k. There would've been 10-50 established TCP conns and many more in close/wait states.

With my methods, there are only several established TCP conns and 30 or more conns in close or wait states. Traffic over the past week has averaged around 16kB/s.

For sure, my crude methods block legitimate users; I whitelist them should they contact me. I'd like to fine-tune my processes, but don't necessarily have the time or knowledge (like employing statistical processing which should reveal the difference between human and 'bot accesses). I'm tempted to redirect some of them to a honey pot and/or a tar pit.

The legitimate worldwide 'internet services' industry needs to band together to address the problem. Here are a few of the things that came to mind (in no particular order):

• Unify world-wide RDAP data; each region seems to record data differently, and seems to record different (or not all) data.
• Expand RDAP to make it easy for all internet users to identify the assignees who use IP addresses and netblocks, returning country, and company name where applicable or the person's name.
• Expand RDAP to identify standard uses and statuses of netblocks and IPs (end users, VPS farms, cloud server farms, web server farms, search engines, stolen nets, gateway routers, et alia), and require assignees to keep that data up-to-date.
• Provide tools (source code) so all internet users can look for general parity between incoming and outgoing traffic. (Have my private network or any of my systems been hijacked?)
• Require IP assignees to identify FQDNs that resolve to each of their IPs/nets.
• Require FQDN owners to identify all IPs that each FQDN resolves to.
• Require cloud providers to monitor their systems for improper use (such as scraping and probing) and block/ban such users.
• Prohibit companies from claiming to scrape and probe IP and sites "for your benefit" when they are really doing it for their own benefit and profit.
• Require ISPs and others to allow end users to easily assign/relate properly acquired FQDNs to their IPs.

We users of the internet deserve to be able to easily identify who is connecting to our private internetworks. For example, if Acme Systems Engineering is probing my gateway in a harrassing way, I should be able to blacklist all of their IPs. If someone using Peaker Cloud Computing Services is hammering away at my gateway using multiple IPs, I should be able to block of of Peaker's IPs used in their cloud services. The system should be nearly trivial to use and should return data that are easily read by humans and easily parsed by systems.

[This is probably enough for one comment....]