>That's all very well if you don't care about abstractions or compatibility.
I had to stare at this for a while to figure out what you were saying...
I think you mean, "that's all very well if you aren't claiming to return a Foo but in fact returning a FooBar (a subclass of Foo chosen at runtime)"? That's a good point. In practice I find that the only times dynamic dispatch (virtual methods) pull their weight in C++ are when using the "strategy" pattern, and then it's usually the caller passing the subclass instance down and references work fine, but that's for my code in my projects; YMMV. I'm happy to add that to the list of exceptional cases where (smart) pointers are called for.
But in response to the rest of your comment, giving the calling code the ability to allocate the object does not mean that you have violated any abstractions. If my code says "give me a rectangle", and someone changes the size of a rectangle in the header file, then my code will allocate a differently sized object from then on. Binary (as opposed to source) compatibility is a little more work, but it just requires the usual pimpl technique that you usually need anyway if you want binary compatibility in a C++ library.
And of course there are always trade-offs, but that doesn't mean this is a zero-sum game. Pointer arithmetic/memory management is one of the most error-prone aspects of classic C/C++ programming (probably only threads are worse); eliminating it will, all else being equal, reduce your bug rate -- or at least, it worked for me.
It's also true that the other way you can segfault is by walking off the end of an array (well, arguably this is a special case of pointer arithmetic, but whatever). The answer to that is even easier: along with not using pointers, don't use arrays :-). Avoiding pointers in C++ takes some tricks, but avoiding arrays (and unchecked wrappers around them) is trivial and worthwhile. It won't stop you making indexing mistakes, but it can turn unreliable crashes/random memory corruptions into reliable clean exits that print the offending file and line number.