A framebuffer driver is just that.. you take information and you write it out to a framebuffer. It's a memory region. It does not get you any sort of OpenGL acceleration or hardware scaling or accelerated YUV to RGB conversion or anything like that. Your worse off, efficiency-wise, with a plain FB driver then you were back in VGA graphics days.
Xorg provides a framework for 2D and 3D acceleration. It's almost always going to have better performance then a simple FB driver unless something is severely broken.
If you don't want to use Xorg then you have to replace it with something else.
Wayland is a possibility and it provides it's own 2D and 3D acceleration whatnot. But it's not mature.