Well, those are good points. Undoubtedly, there is a certain elegance to doing things in the kernel, and it is very convenient. Even though kernel code is _risky_ it is also easy - you can do whatever you want in a straight-forward manner, maintain shared state and synchronization between processes for free, etc.
I am mostly concerned about the first part - the "riskiness". Adding more code to the kernel invariably will mean more bugs. Graphics code is very complex (at least the one I have worked with).
Fundamentally, putting stuff in the kernel offers only two true benefits: ability to maintain cheap shared state, and the ability to force changes to distributions in a centralized manner. (The latter is sad but true). Any other benefits are incidental.
So, if it is not terribly complex or inefficient, it makes sense to move code out of the kernel, when possible.
Of course it is absolutely impossible with some kinds of code - network and disk drivers, file systems, etc. However there are also other kinds, like USB, where user mode drivers are possible and even desirable. Graphics is somewhere between.