Why not clone?
Why not clone?
Posted Sep 10, 2026 3:31 UTC (Thu) by raven667 (subscriber, #5198)In reply to: Why not clone? by anton
Parent article: Ryabitsev: Creepy crawlies
I'm trying to think how that could even work, having the initial request return a nearly empty document with a meta refresh to redirect to a new URL that also has javascript which redirects to a tarpit, so if the JS runs first you redirect to nowhere, but if you ignore the JS then you redirect to a page with real content. The URL for the content page will still have to be in the initial page, so the scraper might still log it for future scraping, maybe you have to record the source IP along with a token and deny any request for that page which comes from the "wrong" IP with a valid token. You'd need to be prepared to store a value for a significant number of addresses.
