Not just crawlers, but the end product too

Not just crawlers, but the end product too

While the bulk of it is probably crawlers, something I observed when I used Claude for my work is that by default when you ask it to find documentation or code on something it will try to download repositories file by file, often piping a curl through head or grep and then download it again if it didn't find what it wanted... Even when you had the path to your own checkout in your CLAUDE.md. You can instruct it to download whole repos with git instead to a custom path, which it will follow most of the time, but not always, specially as a session grows long. The defaults are crappy and the machine not great at following instructions. If you are vibecoding rather than paying attention to the assistant, you'll find you add a lot of unnecessary load to websites, just like scrappers.