In the name of cross-OS portability, for the last 2 decades, X has done it's own hardware
management. [*]
The effect of this is that while X is nicely portable from one OS to another, the portability
from one generation of hardware to the next got severely compromised, and more than a few
times X needed expanding or changing for this reason. And even more times, the OS kernel, such
as Linux, grew nasty workarounds to keep older versions of X working on newer HW or newer
kernels.
Lets face it:
It's the kernels job to manage system resources; doing that twice is a bad mistake. It's also
a kernels job to abstract hardware to a large degree..
For all intents and purposes, this makes the current X part of the kernel (logically). In that
light, the current split is rather bad, and needs to be fixed (and is getting fixed
thankfully).
[*] Yes I know this is getting fixed.