Won't happen. It's been proven time and time again since the 80s: it's all about process. If you care about performance, you must go with the high-volume, off the shelf CPU. Apple learned it with PowerPC, Intel/HP learned it with Itanium, and Sun has learned it multiple times over. We'll see if Larrabee and Cell can buck this trend but they're looking pretty weak so far.
Let's say you're Sun, you want a fast server chip, and you willing to give up single-core speed (what everyone else wants) to go massively multicore. You like the differentiation so you burn a ton of engineer time creating a custom chip and all the infrastructure that goes along with it.
Once you're shipping, you'll find that Intel has already smoked you. Happens every time. They just take a rusty, decades-old design, update it, shrink it, and fab it on a process you won't be able to use for years. And now their $400 chip blows the doors off your custom $1600 silicon for all but the most synthetic workloads.
I agree, we'll see servers become far more specialized to host modern workloads more efficiently. But, when it happens, the successful products will be x86 based. Alas.