Indeed, VNC has nothing to do with X. I can't seemlessly copy/paste text, moving a windows is horribly slow in VNC, and it's not compatible with the X applications that already run on other systems. It's not the way to replace X. In my opinion, the proper way to do that is to have the network not between X layers but next to them. In short, applications should be able to more or less directly communicate with hardware, and an X network server should be available as any other application (think Xceed and equivalent under other OSes). Dynamic libraries should provide the alternate API needed for local applications to use the X network protocol to render on remote displays.
I can say I'm using X remotely on a daily basis, including between various systems. It would be a big functional loss if networking would simply be removed. That would be one reason to switch to a more open system :-/