Why not clone? [LWN.net]

Why not clone?

Posted Aug 29, 2026 10:35 UTC (Sat) by magfr (subscriber, #16052)
Parent article: Ryabitsev: Creepy crawlies

Why does the crawlers crawl?
Would it not be more efficient for them to clone the repository and then pull once in a while rather than running a continuous overload attack on your web interface?

to post comments

Why not clone?

Posted Aug 29, 2026 10:48 UTC (Sat) by magfr (subscriber, #16052) [Link] (10 responses)

Ok. I should have read it first.
Apparently the answer is that the AI trainers are dumb.

Why not clone?

Posted Aug 29, 2026 11:23 UTC (Sat) by jpeisach (subscriber, #181966) [Link] (9 responses)

> Apparently the answer is that the AI trainers are dumb.

Yeah. They probably want the money or are following the hype train.

Which, by the way, why can't they just scrape the GitHub mirror?

Why not clone?

Posted Aug 29, 2026 11:47 UTC (Sat) by mote (guest, #173576) [Link]

> Which, by the way, why can't they just scrape the GitHub mirror?

There was a recent article similar to this from Wikipedia, the author was pointing out they (WMF) made the whole thing downloadable by design with an easy click, but LLMs are instead just hammering the webheads scraping it like it's 1995. The problem goes beyond just git out there in the wilds of LLM scraper land. Apathy, indifference towards the shared commons from the LLM operators, a snapshot of humanity at large really. Lot of people out there just don't care about you, me or the public good.

Why not clone?

Posted Aug 29, 2026 14:04 UTC (Sat) by rgmoore (✭ supporter ✭, #75) [Link] (5 responses)

Which, by the way, why can't they just scrape the GitHub mirror?

Because that requires more human time and effort. Every site they might want to scrape handles things a bit differently: it has its own API, mirrors to its own favorite mirror, etc. They could access all the information efficiently by following all those rules, but it would require a large team of programmers coding rules specific to each source of information. Instead, they seem to have decided that having access to huge bot networks means they don't have to worry about efficiency or rules following, and they can just brute force their way through by pretending to be millions of individual users accessing the information one web page at a time.

Why not clone?

Posted Aug 29, 2026 19:32 UTC (Sat) by marcH (subscriber, #57642) [Link]

Yeah, adjusting to the specifics of each server would be a lot of code to write. And it's not like these companies know how to write code /s

Why not clone?

Posted Aug 30, 2026 5:42 UTC (Sun) by roc (subscriber, #30627) [Link]

Seems like AI agents could easily write all the code for this.

Why not clone?

Posted Aug 31, 2026 8:14 UTC (Mon) by taladar (subscriber, #68407) [Link] (2 responses)

Maybe some sort of header RFC to standardise pointing people to more efficient download methods for the same data would be a good idea?

Why not clone?

Posted Aug 31, 2026 17:11 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

It might not hurt to come up with a standardized way of sending bots to the polite download method, but I am skeptical it would help that much. The bots ignore robots.txt, and their use of residential proxy networks to mask their access patterns is a sign the people running them know they aren't welcome. It seems unlikely that people who act that way will take a polite hint.

Why not clone?

Posted Sep 1, 2026 7:14 UTC (Tue) by taladar (subscriber, #68407) [Link]

If it was just about being polite, sure, but a lot of crawler operators actually do care about efficiency because it saves them time and money too.

Why not clone?

Posted Sep 1, 2026 7:44 UTC (Tue) by skx (subscriber, #14652) [Link] (1 responses)

> Which, by the way, why can't they just scrape the GitHub mirror?

They're probably doing that as well.

No doubt there are multiple systems each scraping their own set of preferred/discovered targets, with no central coordination at all.

Why not clone?

Posted Sep 3, 2026 15:30 UTC (Thu) by JanC_ (subscriber, #34940) [Link]

Even if it were the same scraper system, the web pages would be different, so they would not detect that e.g. GitHub is showing the same content as Codeberg in a different way.

Why not clone?

Posted Aug 29, 2026 12:15 UTC (Sat) by MarcB (subscriber, #101804) [Link] (6 responses)

That really is the main question that the original article is not answering. It simply makes no sense to do this for training. The Linux kernel is not *that* valuable (only a small fraction of software development is at the system level - don't get fooled by the inherent bias of this site). Even if it were, cloning is obviously so much more efficient; even more so for the data consumers themselves.

I wonder is this already is "agentic" AI, potentially controlling a real browser. This would equally fit the pattern of just 4-5 requests and explain how they defeat countermeasures. Those things pick up GIT commits referenced anywhere and can decide too pick them up to have a closer look. This can easily find some "random old fork". It might be worth checking what the possible trail might have been for a number of those requests (but it could be hard, because of all the anti-bot/scraper measures everywhere :-).

Why not clone?

Posted Aug 29, 2026 22:40 UTC (Sat) by JanC_ (subscriber, #34940) [Link] (5 responses)

My guess is that these are just dumb people who are randomly scraping the internet with dumb scrapers to feed their attempt at training AI models. They obviously have no clue what git or a commit is.

Why not clone?

Posted Aug 31, 2026 8:16 UTC (Mon) by taladar (subscriber, #68407) [Link] (1 responses)

It is much more likely that these are just people using (inference) some coding agent to solve some problem that requires information from those particular commits.

Why not clone?

Posted Sep 1, 2026 12:55 UTC (Tue) by smurf (subscriber, #17840) [Link]

Dream on. The volume is much too high for that idea to be remotely valid. Also, the traffic is too spiky.

Why not clone?

Posted Sep 3, 2026 5:15 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (2 responses)

How can they be dumb enough to do that, but also smart enough to use headless Chrome (or whatever) to bypass Anubis? It doesn't make sense to me.

Why not clone?

Posted Sep 3, 2026 13:41 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

I wonder if this recent DailyWTF isn't applicable. They're measuring 4xx results without looking at why they're getting them and the metrics numbers "just need to go up", and running a full browser "makes number go up" and is seen as a suitable solution.

Why not clone?

Posted Sep 3, 2026 15:33 UTC (Thu) by JanC_ (subscriber, #34940) [Link]

They already have to use browsers because so many sites show no content at all without JavaScript & various other browser technologies.