Yep. This is why -mfpmath=sse doesn't actually change the ABI. :)
(But with regard to the flag which *does*, everything you say is true. glibc's hwcaps mechanism will allow you to implement 'slight variation on instruction set', allowing some but not all libraries to have alternate versions for various hwcaps, plus 'tls' as a now-obsolete special case. On x86, this tends to get used to compile different x86-32 binaries for machines supporting versus not supporting the CMOV instruction; on e.g. SPARC64, it is (or was) used to provide alternate versions of libraries for the SPARCv9 32-bit instruction set, which is much like x32 except ABI-compatible with the usual SPARC 32-bit ABI -- all the SPARCv9 registers plus integer multiply and divide instructions, imagine that! You can't use the hwcaps mechanism to support different ABIs though, because nothing stops a hwcapped library calling a non-hwcapped one, or vice versa.)