That was the experience of the Solaris developers: they wanted a secure and lightweight mechanism, so they adapted code that was already in the kernel to provide security isolation and simplified it to produce the appearance of virtual machines.
The initial solaris "zones" were extremely lightweight, since their code paths were already being executed. And they had their security designed in from (before!) the beginning.
They were then extended with resource management to create full "containers".
Right from the beginning, they were simple and elegant, and adding the third generation of resource controls didn't mess them up at all (:-))
From that experience, I think a distribution with Linux containers would be a particularly good base from which to create a secure desktop.