Review?

I find that the difficulty is mostly not understanding the change, as a lot of the LLM driven PRs I see are actually fairly trivial, but rather the process of trying to carry out the review and interact with the nominal submitter.

The difference is that a human junior developer will (hopefully) learn from review comments, not just in terms of acting on then for the change under review, but also in terms of doing better with their next change.

An LLM will never do that, because it can't, so every change it makes is like a freshly minted junior developer and you're starting over from scratch every time. Sure in theory you could provide previous reviews as context but nobody's actually going to burn tokens on that.

Now if there's a human sitting between the LLM and the reviewer then maybe they will learn and will do a better job of prompting the LLM and reviewing it's output next time round but all too often any human that does exist is just copy-pasting the review comments back into the LLM and then copy-pasting the result back into the pull request and essentially just acting as a stenographer that adds no value to the process at all.

Frankly it's utterly soul destroying trying to review most of the LLM driver PRs we get for the OpenStreetMap web site code and I at least have mostly given up even trying.