I have to disagree. First, it is not only the mode setting that goes into the kernel. And mode setting is really the last thing that needs to be there. You certainly do not have multiple applications trying to set different modes _at the same time_.
The kernel and X also never access the hardware at the same time. Designing the whole architecture so that the kernel can display an oops message when it crashes is absurd.
The thing about handing monitor hot-plug interrupts is also a red herring. For one, you hardly need extremely low latency for that, so polling once every 200ms in user mode would probably be sufficient. Even if you wanted to do it with an interrupt, there is no need for everything else to be in the kernel - handle the interrupt in the kernel and notify user mode.
Low latency interactions between DMA and interrupt handling do need to be in the kernel, however graphics drivers rarely have those.
So, I am aware of some of the arguments used to justify this, and I don't find them entirely convincing. There are clearly some advantages (especially if you have multiple 3D apps), but I am surprised that everybody is accepting this as if it is the only possible and clearly superior solution.
There is the unfortunate tendency of trying to put stuff in the kernel, simply because the kernel is the only common part that can be relied on to be in every Linux system. Take mode settings. If you had a user mode daemon doing it, it would work equally well. However, will every distribution take it up? Who will maintain it and in what repository? So, the problem is at least partly political/organizational and that really worries me. Can there be another UDEV?