The old adage says make things as simple as possible, but no simpler. Calling abort() is often _too_ simple. Returning error codes, and per thread error flags are nothing new. It has been standard practice for ages. It works, and is not that complicated.
Yes, overcommit is a problem. When (if) you get a out-of-memory error, it means the system is pretty ill, possibly dying. The proper solution in that case varies from program to program, and even within a program. Sometimes you should free up resources, sleep a while, and try again. Sometimes aborting is the right thing to do. And yet sometimes you should ask the user how to proceed. That's the reason why fixing this policy in a library is bad design.