Yes exactly. Generally, library functions which immediately allocate storage will be returning a pointer to that storage: they can return NULL. Library functions which *call* such functions may have more complex work to do, including possibly unwinding partially-completed work. If *this* would require memory allocation, you may need to find a different algorithm, or arrange to do all allocations first (I've done both at various times).
But saying "I won't bother, I'll just abort" is a disservice to your users, no matter *how* hard it is to handle OOM.