It's actually impossible even on well-tested platforms with good support (read: Windows). Even with such a standardized and clean API as DirectX (3D, Compute) is.
Linux sucked for anything graphics related since forever and NVidia has been a lone saviour for a number of years. That problem isn't new. KDE can't "fix them in the drivers" because nobody - not even major commercial Linux players like Red Hat - was able to do that either. Hell, even Intel can't, despite having its own drivers for its own hardware.
That's all is pretty sad, actually. With the appearance of GPGPU more than 4 years ago, the world is slowly moving to some hybrid computing model where main CPU cores are used only for task-parallel programming, which is a minor challenge, while data-parallel stuff - the most compute-intensive one - is being done on simpler, but more numerous GPU-like cores.
Good news is that the future indeed belongs to multicore systems. Bad news is that it won't be symmetric, but rather asymmetric multiprocessors with most of performance-critical code being written in "virtual" instruction set like NVIDIA's PTX which heavily relies on drivers to translate it to hardware-specific representation.
GPU drivers nowadays aren't just for graphics. We're abstracting away the processors (which is a good thing), but unlike Java etc we avoid losing the speed that much. Unfortunately we are introducing a layer of drivers which will soon be needed to run even the most basic software. And there's no open source solution for that, not even on a horizon - seeing how open source drivers are struggling to support the functionality of pre-GPGPU era...