Yes. You have to privileged code handling input, either living in the kernel or in a privileged process. It's possible to write secure code, it's just very hard when you have as much code as the X server does (or...for that matter, the linux kernel, but that's a different issue). If you put the input server in a separate privileged process, you have to secure a lot less code, so the problem becomes feasible.
And note that it doesn't actually have to run as root, just as a priv user.