Toward more predictable and reliable out-of-memory handling [LWN.net]

Toward more predictable and reliable out-of-memory handling

Posted Dec 21, 2015 13:07 UTC (Mon) by sorokin (guest, #88478)
Parent article: Toward more predictable and reliable out-of-memory handling

Do I understand correctly that all problems with OOM handling in Linux are caused by memory overcommitting? Is it true that memory overcommitting is required for two reasons: (1) because fork is used to create child processes and (2) because shared objects are mapped as MAP_PRIVATE?

Can't the (1) be fixed by using something like posix_spawn and can't the (2) be fixed by mapping as MAP_PRIVATE only GOT and PLT and MAP_SHARED everything else in shared objects? As I understand in this case it will be possible to default to vm.overcommit_memory=2 and forget about oom-killer.

to post comments

Toward more predictable and reliable out-of-memory handling

Posted Dec 23, 2015 0:39 UTC (Wed) by eternaleye (guest, #67051) [Link] (1 responses)

Well, the first issue is that while those two points are part of why, they're far from the whole story.

In particular, there are most certainly programs which allocate large amounts of address space without necessarily using that memory in the end.

In addition, changing every use of fork to posix_spawn is... a decidedly nontrivial undertaking.

Besides that, failing to check the return value of malloc() is a pervasive issue, which has been worsened by the default-overcommit behavior on Linux has trained people that "malloc() never fails."

And then, if the user program DOES check the return value of malloc(), what are its options?
- Drop some internal caches
- Loop forever
- Die

i.e., the exact same issue as the OOM killer, with the added downside of not having the ability to take advantage of global information - say, page cache pages being clean.

And while it may _seem_ as though there's a benefit there, where the allocating process takes the hit, it's a false fairness - after all, a 3.9 GB allocation may have just succeeded for another program, and now your shell wants one measly page and can't get it.

Toward more predictable and reliable out-of-memory handling

Posted Apr 15, 2016 23:27 UTC (Fri) by oak (guest, #2786) [Link]

> there are most certainly programs which allocate large amounts of address space without necessarily using that memory in the end.

For example programs that create threads. While the stack in main thread grows dynamically, rest of the threads have a fixed sized stack. While one can specify what sized stack threads get, most programs don't.

By default, thread stack sizes on Linux are currently 8MB. If you have a lot of programs that use a lot of threads, without overcommit that would eat quite a bit of RAM.