No,
X supports running remote applications, just like FireFox supports running JavaScript(Angry Birds). FireFox doesn't let Angry Birds take over your box, why should X be allowed to pass sensitive information to remotely connected clients? Keep in mind that you don't -necessary-(not at all, not ever) trust root on the remote host to be able to read all your passwords.
I know that it's not easy to be secure, even Browsers have had issues with loading some image files. Leaving doors like this open because "There are other security issues" is not acceptable! There are always other security issues and what kind of world do you think we'd have if that excuse worked. Sudo doesn't really need to verify passwords because we already know the user has logged in, network VPNs need to be encrypted but there is no point in verifying the data isn't forged because no one would have the key used to make an encrypted packet. I hope I've made my point, but I'll try and get a few examples more on topic. Email servers can forward the BCC header to every one, it'll be removed on the receiving end by the users MUA. Passwords can be saved along with there hashes, because no one could ever read the shadow file. Hmm, I'm still not happy with any of these. Ahh, SSH doesn't need to be encrypted because there is telnet. Self signed server certificates are just as good as any other because no one really knows what they have installed for trusted CA certs. I really like this last one, a lot.