It's safe. Access to /dev/kvm doesn't give any access to other virtual machines.
Of course, if a process has access to another process (via kill(2) or ptrace(2)) it can affect or access data belonging to that process. So if you run all virtual machines as the same user, you need to further isolate them. I believe sVirt does that with its random selinux contexts. but I'm no selinux expert.