Why not clone? [LWN.net]

Why not clone?

Posted Aug 31, 2026 8:14 UTC (Mon) by taladar (subscriber, #68407)
In reply to: Why not clone? by rgmoore
Parent article: Ryabitsev: Creepy crawlies

Maybe some sort of header RFC to standardise pointing people to more efficient download methods for the same data would be a good idea?

to post comments

Why not clone?

Posted Aug 31, 2026 17:11 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

It might not hurt to come up with a standardized way of sending bots to the polite download method, but I am skeptical it would help that much. The bots ignore robots.txt, and their use of residential proxy networks to mask their access patterns is a sign the people running them know they aren't welcome. It seems unlikely that people who act that way will take a polite hint.

Why not clone?

Posted Sep 1, 2026 7:14 UTC (Tue) by taladar (subscriber, #68407) [Link]

If it was just about being polite, sure, but a lot of crawler operators actually do care about efficiency because it saves them time and money too.