to answer your final question (why are drivers in X instead of the kernel)
it basicly boils down to historic reasons, it's always been done that way.
X works on many different systems with different kernels and it used the same drivers on all
of them.
there's work being done currently to define an interface between kernel drivers and X that
will shift this boundry significantly, but they will need to keep much of the old code around
to support other (and older) systems