Wayland doesn't use any drivers directly. Instead it relies on OpenGL and EGL, but needs the ability to run OpenGL and EGL directly on the hardware. This is now possible with all the open source drivers using gallium. I have tried wayland and it works well if you use the open source drivers. Even llvmpipe works ok.
The problem is closed source drivers. If the closed source drivers would be modified to not require X but run directly on the hardware, then wayland would work for them too with no changes to wayland needed. But no one can see the code so no one knows.