Well, the kernel being the only bit of software that manages the hardware is just dogma. You do it when it makes sense, and don't do it when it doesn't.
I am not saying that it doesn't in this case, but it is not as clear cut as people seem to thing. For one, drawing sometimes requires talking directly to the hardware. Not every graphics chip has a DMA command queue in system RAM, and even even for the ones that do, it is not always the best solution.
The command queue could be a FIFO in the chip. So, in some cases X has to talk to the hardware in order to draw. If we insist the kernel to be the only one touching the hardware, we have to make a kernel mode transition for every drawing command, which is not ideal.