another possible source of scraper traffic

I've worked for a company that considers itself to be an ai company for the past 1.5 years (not one of the labs, but a smaller startup), and I can maybe add a bit of detail about another possible source of this traffic.

As context, one of the big advances in making LLMs useful over the past couple years is the idea of an "agent loop". Instead of asking the LLM to directly answer a question or to write some code, it iteratively calls "tools" to interact with the world, bouncing back and forth between inference and tool calls. For example, the agent might run some shell commands then based on the result look up documentation then based on that content write a script and etc.

So the actual traffic source is agents aggressively looking up documentation to complete their goals. In the case of something like Claude Code or Codex these requests come directly from a person's computer (on a residential internet connection), and I'm not sure to what extent the software identifies itself in the UA or respects robots.txt. In the case of something like ChatGPT or Claude web (or anything else hosted) I wouldn't be surprised if they use residential proxy services to ensure they can access content. This might also go part of the way to explaining some requests to several decade old articles - in my own (non-ai-assisted) work I've found LWN articles to be tremendously useful, so I would not be surprised if agents are looking up old pages to try to figure out how to (eg) write or fix a driver.