Why not clone? [LWN.net]

Why not clone?

Posted Aug 29, 2026 11:23 UTC (Sat) by jpeisach (subscriber, #181966)
In reply to: Why not clone? by magfr
Parent article: Ryabitsev: Creepy crawlies

> Apparently the answer is that the AI trainers are dumb.

Yeah. They probably want the money or are following the hype train.

Which, by the way, why can't they just scrape the GitHub mirror?

to post comments

Why not clone?

Posted Aug 29, 2026 11:47 UTC (Sat) by mote (guest, #173576) [Link]

> Which, by the way, why can't they just scrape the GitHub mirror?

There was a recent article similar to this from Wikipedia, the author was pointing out they (WMF) made the whole thing downloadable by design with an easy click, but LLMs are instead just hammering the webheads scraping it like it's 1995. The problem goes beyond just git out there in the wilds of LLM scraper land. Apathy, indifference towards the shared commons from the LLM operators, a snapshot of humanity at large really. Lot of people out there just don't care about you, me or the public good.

Why not clone?

Posted Aug 29, 2026 14:04 UTC (Sat) by rgmoore (✭ supporter ✭, #75) [Link] (5 responses)

Which, by the way, why can't they just scrape the GitHub mirror?

Because that requires more human time and effort. Every site they might want to scrape handles things a bit differently: it has its own API, mirrors to its own favorite mirror, etc. They could access all the information efficiently by following all those rules, but it would require a large team of programmers coding rules specific to each source of information. Instead, they seem to have decided that having access to huge bot networks means they don't have to worry about efficiency or rules following, and they can just brute force their way through by pretending to be millions of individual users accessing the information one web page at a time.

Why not clone?

Posted Aug 29, 2026 19:32 UTC (Sat) by marcH (subscriber, #57642) [Link]

Yeah, adjusting to the specifics of each server would be a lot of code to write. And it's not like these companies know how to write code /s

Why not clone?

Posted Aug 30, 2026 5:42 UTC (Sun) by roc (subscriber, #30627) [Link]

Seems like AI agents could easily write all the code for this.

Why not clone?

Posted Aug 31, 2026 8:14 UTC (Mon) by taladar (subscriber, #68407) [Link] (2 responses)

Maybe some sort of header RFC to standardise pointing people to more efficient download methods for the same data would be a good idea?

Why not clone?

Posted Aug 31, 2026 17:11 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

It might not hurt to come up with a standardized way of sending bots to the polite download method, but I am skeptical it would help that much. The bots ignore robots.txt, and their use of residential proxy networks to mask their access patterns is a sign the people running them know they aren't welcome. It seems unlikely that people who act that way will take a polite hint.

Why not clone?

Posted Sep 1, 2026 7:14 UTC (Tue) by taladar (subscriber, #68407) [Link]

If it was just about being polite, sure, but a lot of crawler operators actually do care about efficiency because it saves them time and money too.

Why not clone?

Posted Sep 1, 2026 7:44 UTC (Tue) by skx (subscriber, #14652) [Link] (1 responses)

> Which, by the way, why can't they just scrape the GitHub mirror?

They're probably doing that as well.

No doubt there are multiple systems each scraping their own set of preferred/discovered targets, with no central coordination at all.

Why not clone?

Posted Sep 3, 2026 15:30 UTC (Thu) by JanC_ (subscriber, #34940) [Link]

Even if it were the same scraper system, the web pages would be different, so they would not detect that e.g. GitHub is showing the same content as Codeberg in a different way.