Avoiding the OOM killer with mem_notify
Posted Jan 31, 2008 17:06 UTC (Thu) by jzbiciak
(✭ supporter ✭
Parent article: Avoiding the OOM killer with mem_notify
How effective can this be, though, for many C programs? If I malloc a bunch of memory, perhaps as caches, and then am asked to free it, that doesn't magically release pages back to the OS. Now, if malloc uses mmap for some of the larger allocations, those can be released back to the OS by munmap. But, for the general sbrk managed heap, I have to free stuff near the end of the heap before I can ask for my brk to be lowered. There's no guarantee I can do that.
For this to be useful, whatever I malloc needs to have an additional level of indirection in user space, so I can move the objects I wish to keep and then compact the heap. Otherwise, simply freeing stuff up won't be enough.
It may be useful to compare/contrast this to the HURD's approach, which is simply to force user space to do its own VM management. There, the kernel and user-space dicker about physical pages only, and user space figures out how best to handle the burden when a given app wants more pages than the OS can give it. The answer could be garbage collection, discarding caches, swapping or whatever makes sense to a given application.
The main thing is that the app knows way ahead of time that real RAM is in short supply, and avoids getting into the overcommitted state entirely. And since the kernel isn't doing the swapping, it seems like you wouldn't get into situations where you need to free memory so you have enough memory so that you can write out pages and the like. Example: Imagine that to wake an app so it can free some pages, you have to bring it in from swap, but swap is too full to write any dirty anonymous pages out. If your policy is that each app self-swaps, this should never happen since the OS guarantees it'll have enough pages to do its work, and user space will just muddle along with what its given. (In theory, it seems like a user space app could get by with just a few pages... a couple executable pages and a couple data pages.)
I'm guessing mem_notify will try to wake apps sufficiently far ahead that it can avoid those "need RAM to free RAM" situations in practice, but setting proper thresholds seems like it ought to be rather tricky.
to post comments)