The ultimate solution would be to not have the server do any of the rendering at all.
Instead of the server doing the rendering all the rendering is handled by the clients. The clients themselves talk to the GPU via whatever API they feel like using and then render their output to a off screen buffer.
Then all the 'display server' has to do at this point, besides handle the grunt work of doing things like handling inputs and modesetting, is flip the client buffer and composite the display.
With a little bit effort then it should be possible to kill the 'display server' and then have another 'display server' be able to grab the client application's output buffer.
Taking that and extending the idea further it should then be possible for you to take the client application's buffer it's rendering to, perform some sort of efficient streaming image compression, and then send that output over the network.
It may need slightly more bandwidth then traditional X, but it would still benefit from accelerated rendering (especially if your image compression can be accelerated) and would be virtually unaffected by latency... which is with X is it's Achilles's heal.
You could then have a single application on your desktop and then seamlessly transfer it to your phone or anything else running on any network. No special effort needed on the side of the application developer and you still get acceleration and all that happy stuff.