Ya.. I originally thought the pure OpenGL environment would of been the way to go, but there are a couple problems with going pure-OGL server.
1. Doing 2D graphic acceleration for applications on OpenGL is more difficult/irritating then originally thought. EXA was developed as a replacement for XAA and is actually able to run quite well on 3D hardware.
2. OpenGL is only designed for a application API. There is nothing in the standard for managing displays, video memory, etc. Mode setting and whatnot is actually VERY VERY hard to get right.
Then people realized also that they will want to use the GPU for other tasks like OpenCL, shader languages, video acceleration, codec acceleration, etc.
So the thoughts went from "OpenGL for everything" to extending the existing DRI protocol used by OpenGL to allow drivers that support multiple APIs (gallium) then putting a unified memory management scheme and mode setting into the kernel.