It's not so much the voodoo I find annoying, it's the instability that binary drivers cause.
Systems where I use binary drivers (nvidia or ati) to be able to play 3D games will crash as much as Windows. The cause is usually some interaction between OpenGL, video overlay and Compiz. I don't even try to do more advanced stuff like multi-head or tv-out.
On the other hand, if I stick to default open-source drivers, thing hardly ever go sour. So I've been avoiding nvidia like the plague, and stayed with Intel for basic desktop with compiz, and Ati for 3D gaming.
This is good news, it means I backed the right horse for once :-)