I don't think it will make much of a difference because display servers are moving out of the hardware management business. Any new display server is going to, or should, be using standardized application APIs for it's foundation.
So it should not matter what display server you are using. The display server needs to conform to the APIs provided by low level system libraries and the kernel, not the other way around.
With X windows in Linux the display server had to have it's own driver to drive the hardware directly, which is a idiotic design and is a disaster. It's one of the major reasons open source drivers got so far behind.