> Afaics, there were essentially two reasons for putting the rendering code in the X server
X server supports both client-side and server-side rendering.
But Xorg is much more than just rendering.
1. Xorg == standards. Stable standards supported everywhere, literally (Windows, MaxOS, I've heard even Android has some java-based X11 implementation, maybe not as good as Xorg, but still).
2. Xorg == hardware support. It works everywhere, including small mobile/ebook devices and large multi-monitor systems.
3. Xorg == software support. During its long lifetime it got all the nifty tools you can think about (xrandr, xinput2, xautomation, xdotool, xbindkeys, xvkbd, xnest, x2x, x11vnc, etc).
4. Xorg == infinite personalisation. You can choose any window manager or desktop environment you like. Or you can combine them. Thousands of themes are waiting for you.
5. Xorg == development support. Every toolkit supports Xorg.
6. Xorg == stability. You can release your closed-source binary game today and you can be sure, that it will still work for the next 10 years.
(#6 is not that strong now, Wayland fans talk a lot about Xorg being bad, and that gives some bad points to Xorg and commercial Linux development, sigh...)
At the same time X11 was never rotten. On the contrary, it was always among the first to innovate. First remote graphical terminals, first desktops with multiple workspaces, multi-monitor systems, first multiseats, first 3D-desktop, first multi-touch (MPX, yeah, it was done before iPhone :)) — most (if not all) of these things were first done under X-server.
> Otoh, client-side rendering is faster because it avoids the protocol overhead
You should probably like DirectFB a lot. It's direct, client-side, and has no server, so it completely avoids any overhead possible. :)