This is just a re-expression of a problem graphics developers have had for fifteen years or more, ever since video cards started getting dedicated RAM with relatively access latencies from the CPU: you can store stuff in VRAM and it's really fast to manipulate with the GPU and to display, or you can store it in main memory and bash it with the CPU and it's much slower to display, but if you mix the two, you get incredible sloth. Back when the offscreen pixmap cache didn't have a defragmenter (pre X.org 1.6.0) it tended to get too fragmented to put any useful amount of text into... and text scrolling, on a Radeon 9250, then took several *seconds* per screen. That was entirely because of repeated CPU<->VRAM data transfers. They're *slow*.
(Of course, KMS should fix all of this by giving us a proper memory manager for VRAM-plus-main-memory. It doesn't seem to be there yet, though: I still see occasional scrolling slowdowns when the pixmap cache gets too fragmented and the defragmenter hasn't kicked in yet.)