OOM happens when you're out of commit. If you're doing it right, you paid for the commit for the pages you'll need when you loaded the image, so making the backtrace tables resident should still be possible.
> a good thing too, it's fantastically annoying as even very-much-not-OOM systems can find fork() failing because there's not enough swap to back every page that might get dirtied in the new address space, even if it's only going to exec() and throw them all away
I disagree: strict commit accounting makes a system more predictable in practice. If you find fork failing, you should either add more swap (which won't actually get used, as you note, except in the worst case) or change your program to use vfork or posix_spawn instead, both of which don't have the intrinsic commit-accounting problems of fork.