Kroah-Hartman: AF_BUS, D-Bus, and the Linux kernel

Yes! You shouldn't let any userspace application access the GPU directly, because these days we have an expectation that userspace shouldn't be able to compromise the kernel. The alternative is to have signed userspace, and that's not an acceptable option. So, unless you're producing a microkernel which has a separation between drivers and the rest of userspace (which Linux doesn't have), the correct line to draw is the one where only the kernel gets to drive hardware that can overwrite the rest of the OS.