"I have to disagree. First, it is not only the mode setting that goes into the kernel."
The memory management makes a lot of sense, too -- nobody is asking for a userspace virtual-memory subsystem, and the graphics memory space is just as complex and in need of kernel maangement these days as system RAM. Nothing else is going into the kernel that hasn't already been there for many years.
"Designing the whole architecture so that the kernel can display an oops message when it crashes is absurd."
That is not the point of KMS at all, only a side benefit that some people think is neat.
The point of KMS is that the kernel can reset the graphics mode at any time if need be, and those times happen often enough to be important. When X crashes for example, which in the current architecture often requires a power cycle to get your display working again. Or take suspend/resume, where trying to coordinate with userspace processes is really hard and still after all these years of suspend/resume work fails pretty damn often for quite a few people.
"For one, you hardly need extremely low latency for that, so polling once every 200ms in user mode would probably be sufficient."
Except that that blows nuts for laptops (and low-power desktops) where we're trying to reduce and/or eliminate polling entirely so the CPU can spend most of its time in a low-power state. Hotplug is most attractive to laptop users, too. If my laptop is just sitting there showing a static screen (say I'm reading a page) I don't want it waking up the CPU at all if it can be helped. Battery life is possibly the single most important thing for many people with laptops... a laptop with no charge is just a big paperweight if you don't have a power outlet nearby, which is pretty often for travelers and even students.
"Take mode settings. If you had a user mode daemon doing it, it would work equally well."
Except for suspend/resume where any kind of userspace synchronization is a whore.
If you want a microkernel approach, there are plenty of OSes out there for you to comfortably use. Linux is not and never has been one of them, and things move into the kernel in the Linux world when the advantages for doing so are clear and the disadvantages can only be summed up as "some people don't like it on principle because they're afraid of code in kernel space."