Yes, I agree... exactly why my comments were all discussing ABI backward-compatibility. Your example of the sched_setaffinity(2) call is, as you say, a great example of my point... ensuring that a backward-compatible ABI is available involves a lot of tricky footsie-playing with the linker etc.
API compatibility is a completely different animal: if you're going to use Linux-specific APIs then I suppose you just have to fix your code if they change in the future. If you don't want to do that, restrict yourself to POSIX and other standard APIs. I don't really worry about this, to be honest.