I don't remember the actual numbers, but the nxagent and Xvnc processes tended to be at the top of the 'top' output when sorted by memory use. And the (res - shared) values were consistently better for nxagent than for Xvnc. NX is doing something more efficiently. I'm not sure what.
Perhaps this is due to the fact that NX has multiple process (NX agent and NX proxy)? You'll need to count both for a valid comparison.
In any case, the Xvnc memory usage is no deal. On my Fedora 11 system, the local Xorg now consumes 63 MiB (resident), while Xvnc only consumes 34 MiB.