" the extremely fast CPUs available everywhere, including mobile devices."
You're slightly over-estimating the power of many mobile CPUs. And seriously underestimating the amount of audio data that some apps need to work with (and hence the amount of effort necessary for conversion).
If GStreamer were to work for every possible use case it needs to work for, you'd pretty much be forced to work with 64-bit floating point audio everywhere. That is ass-slow on many modern CPUs, especially many mobile CPUs that have no FPU and even a lot of desktop CPUs still in common use.
The extra code isn't hurting you, but it's helping a lot of other people. I'm all for cutting out bloat, but "necessary code" never counts as bloat.