I guess he was talking about big-iron border gateways. I could imagine that their performance suffers compared to IPv4 simply because of the longer addresses and more bookkeeping required. Cache tables and memory bandwidth don't magically increase when you switch.
And since this kind of hardware is a huge investment, it tends to be run at high loads even for standard use cases (i.e. IPv4). Switch to v6 and you suddenly max out your infrastructure. Sure, you could buy new routers, but where is the economic sense in that?