The ability to run KVM as non-root is already in libvirt. In Fedora 12 all 'qemu:///system' connections run VMs under a dedicated 'qemu' user account, while 'qemu:///session' connections run VMs under the UID of the user using that connection.
Also in Fedora 12, /dev/kvm has mode 0666 out of the box, allowing qemu:///session uses to use KVM acceleration.
The libvirt security architecture that deals with sVirt is modular allowing arbitrary security plugins. The Ubuntu devs have got an impl using AppArmour. It would also be possible to write an impl that ran each VM as a unique user ID.