Honestly? Nothing's wrong with Xen. Most of the interesting things are done in hardware these days anyway so there isn't a huge difference between any hypervisor out there.
But what most people don't understand about Xen is that it's not "part of Linux". The bits that were merged into the kernel in recent years are guest-enablement features only. It's a full blown Operating System that has no relationship to Linux at all. It's a microkernel design based on an old research project (search for Nemesis Micorkernel if you're interested). Linux only runs as a guest under Xen.
Xen has its own scheduler, own MMU, own set of device drivers. By constrast, there is no such thing as the "KVM scheduler". KVM is just a small layer that adds virtualization support to Linux. *Linux is the hypervisor*.
I prefer KVM over Xen for the virtualization for the same reason I prefer Linux over FreeBSD for running Apache, or Linux over <insert custom OS> for whatever workload you can think of.
History has shown that collaborating on a general purpose OS wins time and time again over special purpose boutique OSes. That's why many of our cell phones and DVD players run Linux along with most of the Top 500 supercomputers.
You can always make the argument "but you can build a better scheduler for XYZ workloads". It's a short sighted world view that almost always loses over time.