Thanks for your efforts and keep up the good work!
Thanks for your efforts and keep up the good work!
Posted Jul 10, 2026 17:15 UTC (Fri) by daroc (editor, #160859)In reply to: Thanks for your efforts and keep up the good work! by wtarreau
Parent article: An update on the scraper situation
The central registry you propose sort of exists in the form of Common Crawl: https://commoncrawl.org/
The idea is that you contribute resources to their project, they have _one_ scraper that downloads web content in a polite way (respecting rate limits and robots.txt), and then anyone who wants to can make use of the scraped content without having to re-scrape it.
There are some problems with the approach, but I'm generally pretty happy when I see Common Crawl go by in the server logs because I know that's a bunch of unrelated projects that don't need to send us multiple requests.