Ok, bye bye [LWN.net]

Ok, bye bye

Posted Jul 24, 2026 0:40 UTC (Fri) by dskoll (subscriber, #1630)
In reply to: Ok, bye bye by IanKelling
Parent article: Codeberg: Protecting our FLOSS commons from LLMs

The scraper bots are a scourge. I had to password-protect my self-hosted Forgejo instance because it was being hammered. And I have no idea why the bots think it's a good idea to scrape the Web interface to git instead of just doing a git clone.

to post comments

Ok, bye bye

Posted Jul 26, 2026 13:05 UTC (Sun) by cladisch (✭ supporter ✭, #50193) [Link] (1 responses)

> And I have no idea why the bots think it's a good idea to scrape the Web interface to git instead of just doing a git clone

Because HTML has no standardized mechanism to redirect a scraper to a git repository. It might be possible to set <link rel="alternate" …>, but nobody uses it.

And, of course, those bots are vibe coded; neither the bots nor their human handlers have the ability to evaluate the returned data for correctness or usefulness. The only quality measure they have is the amount of pages and returned data, so they have no incentive to add special code to handle git repositories.

Ok, bye bye

Posted Jul 26, 2026 15:57 UTC (Sun) by pizza (subscriber, #46) [Link]

> Because HTML has no standardized mechanism to redirect a scraper to a git repository. It might be possible to set <link rel="alternate" …>, but nobody uses it.

If crawlers are capable of semantically parsing out source code from a sea of HTML, they're capable of finding the "clone this repository' links.