> Because it's still useful in many cases (e.g. between machines on the same LAN) to run remote X clients over an SSH tunnel.
That is why you would dynamically load it - use a library derived from the X server code to draw into something like a Wayland frame buffer when local, and use the X wire protocol when using an X server over a network connection. As long as you are on the same machine, a large class of X applications should run much faster, with a higher degree of security, and so on.
Even without a compositor, this is arguably the way it should have been done from the beginning. What is the point of protecting access to the frame buffer on a typical embedded system? It just slows things down.