As I understand it, originally, XFree86 was supposed to be OS agnostic. The problem, however,
if you put the driver code into the kernel, X becomes very much reliant on that code. Since
everyone had wildly different ways of writing kernel drivers, it was decided that the best
way to keep OS-agnosticism would be to have the drivers in userspace, made and maintained by
the XFree86 project.
As time went on and X became more and more irrelevant except for unix-alikes, historical
inertia, laziness, and plain old pig-headedness from the XFree86 owners kept the aging,
increasingly brittle userspace kernel methodology in place.
It's no accident that people started talking seriously about video drivers in the kernel only
after XFree86 became a zombie project and x.org was transfered to a saner build system.