OpenGL is a massive abstraction, what might make sense is for the kernel to provide an
abstraction that is more low level and closer to the hardware than OpenGL that a generic
OpenGL implementation can be built on.
This is essentially what the Gallium3D people are working on making, although I can't recall
how much of that stuff is gonna be in kernel.
Modern 3D hardware can be driven primarily by writing to it's memory so the driver can
actually run in userspace for little or no penalty, possibly even requiring fewer transitions
from kernel to userspace and so actually being faster, this however applies slightly less to
older less general graphics hardware.
To find out lots about this sort of thing poke around on the nouveau wiki since the direction
taken by them is likely to be the direction taken by other open source drivers, and they are
moving some things into the kernel that are at the moment userspace only things.