Actually what you are describing is currently one of the hot areas in VM systems right now, virtual desktop deployments. My guess, based on your comment about 64GB servers, is that you haven't much exposure to "Enterprise" VM systems like ESX, Xen or KVM. Just for a point of reference I dug around and looked at some ESX 4.1 VDI desktop benchmarks. The benchmark I found was for the storage system but the test harness they used had a density of 78 VMs/server (dual socket/6-core 96GB RAM, pretty average) and they could have gone higher but they wanted low latency and low memory/cpu contention. The VMs in the test were running Windows.
So your example of 50 VM hosts that there is no way you could do this isn't true, running desktop VMs at that level of density isn't even cutting edge and can be done on a modest dual-socket system, probably worth around $10-15k whereas 78 $500 desktops would be almost $40k.
In fact from a security perspective running desktops as virtual machines has some other benefits too in that may systems are run from snapshots off a central, read-only system image so infected machines can be easily and completely rolled back to a known good state.