Why not clone? [LWN.net]

Why not clone?

Posted Sep 9, 2026 14:31 UTC (Wed) by anton (subscriber, #25547)
In reply to: Why not clone? by JanC_
Parent article: Ryabitsev: Creepy crawlies

Solution: Protect your site from scrapers by showing nothing when JavaScript is enabled, and the site when it is disabled.

to post comments

Why not clone?

Posted Sep 10, 2026 3:31 UTC (Thu) by raven667 (subscriber, #5198) [Link] (2 responses)

I'm trying to think how that could even work, having the initial request return a nearly empty document with a meta refresh to redirect to a new URL that also has javascript which redirects to a tarpit, so if the JS runs first you redirect to nowhere, but if you ignore the JS then you redirect to a page with real content. The URL for the content page will still have to be in the initial page, so the scraper might still log it for future scraping, maybe you have to record the source IP along with a token and deny any request for that page which comes from the "wrong" IP with a valid token. You'd need to be prepared to store a value for a significant number of addresses.

Why not clone?

Posted Sep 10, 2026 8:02 UTC (Thu) by anton (subscriber, #25547) [Link] (1 responses)

Yes, it only works if the scrapers really only do what a browser with JavaScript enabled does. And if this defense gains any popularity, the scrapers will look at the non-JS path. Blocking IPs that follow the JS path is probably not what we want:

Scrapers will probably use a different IP for following the non-JS path (if they follow it).
Humans who read on the JS path a graphic that says "Please access this page again with JS disabled to see its content" will actually access the non-JS path from the same IP.

Why not clone?

Posted Sep 10, 2026 8:05 UTC (Thu) by zdzichu (subscriber, #17118) [Link]

Humans greeted with "please disable JS" banner will never ever return to the site again.