LWN: Comments on "Toward a smarter OOM killer"

death by swap

misiu_mp — 2009-12-05T17:19:38+00:00

That was a description of trashing.
I would presume that executables do not make up much of the used memory. So reusing their pages will probably not be much gain.

Trashing is what happens when processes get their pages continuously swapped in and out as the system schedules them to run. Thats when everything grinds to a halt because each context switch or memory access needs to swap out some memory in order to make place for some other memory to be read in from the swap or the binary.
That can possibly happen when the total working set (actively used memory) of the busy processes exceeds the amount of ram, or more realistically, when the swap (nearly) runs out so there is nowhere to evict unused pages to free up ram - leaving space for only small chunks to run at a time.
Usually (in my desktop experience) soon after that the oom killer start kicking in, which causes the system to trash even more (as the oom have needs too) and it takes hours for it to be done.
When it happens I usually have no choice but to reboot loosing some data, so for me the oom killer has been useless and over-commitment the root of all evil.

Using up swap alone does not affect performance much, if you dont access whats on the swap. If you continuously do that - that's trashing.

The existence of OOM is one of the few really stupid things in Linux

jch — 2009-11-23T15:19:36+00:00

> How does Solaris deal with mapping shared libraries without overcommitting?

Shared libraries are backed by filesystem data, so a read-only map of a shared library does not involve overcommit.

posix_spawn is stupid as a system call

jch — 2009-11-23T15:08:38+00:00

This is analogous to the *at system calls (openat, fstatat, ...) that have been introduced in Linux and included in the latest revision of POSIX.

death by swap

dlang — 2009-11-19T18:33:28+00:00

I don't understand how that is any different than thrashing the swap.

in both cases you have to read from disk to continue, the only difference is if you are reading from the swap space or the initial binary (and since both probably require seeks, it's not even a case of random vs sequential disk access)

death by swap

nye — 2009-11-19T16:46:46+00:00

>my recommendation is that if you are using a normal hard drive (usually including the SSD drives that emulate normal hard drives), allocate a 2G swap partition and leave overcommit enabled (and that's probably a lot larger than you will ever use)

FWIW, I agree, except that I'd make it a file instead of a partition - it's just as fast, and it leaves some flexibility just in case.

I use a 2GB swapfile on machines ranging from 256MB to 8GB of RAM - it may be overkill but that much disk space costs next to nothing. I wouldn't want to set it higher, because if I'm really using swap to that extent, the machine's probably past the point of usability anyway.

death by swap

makomk — 2009-11-19T13:29:57+00:00

On the other hand, too small a swap partition can also cause "death by swap". Once the swap partition fills up, the only way to free up memory is to start evicting code from RAM - and that utterly kills the responsiveness of the system, as less and less RAM becomes available for the running apps' code. The system enters a kind of death spiral, where the more memory the application allocates, the more slowly it and every other application runs.

death by swap

dlang — 2009-11-18T18:37:30+00:00

there used to be a technical reason for twice ram.

nowdays it depends on your system and how you use it.

if you use the in-kernel suspend code, the act of suspending will write all your ram into the swap space, so swap must be > ram

if you don't use the in-kernel suspend code you need as much swap as you intend to use. How much swap you are willing to use depends very much on your use case. for most people a little bit of swap in use doesn't hurt much to use and by freeing up additional ram results in a overall faster system. for other people the unpredictable delays in applications due to the need to pull things from swap is unacceptable. In any case, having a lot of swap activity if pretty much unacceptable for anyone.

note that if you disable overcommit you need more swap or allocations (like when a large program forks) will fail so you need additional swap space > max memory footprint of any process you intend to allow to fork (potentially multiples of this). With overcommit disabled I could see you needing swap significantly higher than 2x ram in some conditions.

my recommendation is that if you are using a normal hard drive (usually including the SSD drives that emulate normal hard drives), allocate a 2G swap partition and leave overcommit enabled (and that's probably a lot larger than you will ever use)

if you are using a system that doesn't have a normal hard drive (usually this sort of thing has no more than a few gig of flash as it's drive) you probably don't want any swap, and definantly want to leave overcommit on.

death by swap

pimlottc — 2009-11-18T16:47:28+00:00

STOP USING the old "TWICE RAM" guideline!

You know, I've been hearing this lately, but the problem is there seems to be no consensus on what the guideline should be. Some swear by no swap at all, while others say running without at least some is dangerous. No one seems to agree on what an appropriate amount is. Until there is a new accepted rule of thumb, everyone will keep using the old one, even if it's wrong.

death by swap

pimlottc — 2009-11-18T16:12:34+00:00

irssi can't start on a system with 256MB RAM and no swap? Seriously?

death by swap

michaeljt — 2009-11-16T13:50:48+00:00

>> If you get to the point that you're stealing a page from a process simply because that process is over its quota of real memory, you should steal ALL that process' pages. It can't fit its working set into memory, so it isn't going to make decent progress, so the memory you do give it is wasted.
>I suppose I see three cases here. One is that the page was part of the process' working set at an earlier point in time, but no longer is. In that case swapping it out is the right thing to do. The other is that the process is in control, but it's working set is bigger than the available memory. Then I agree that there is a good case for putting it on hold until enough memory is available, although that is a non-trivial problem which is somewhat outside of the scope of what I am trying to do. And the third case is the one that I am interested in - a runaway process which will eventually be OOMed. In this case, the quota will stop it from trampling on the working set of every other process in memory in the meantime.
Actually case 2 could be handled to some extent by lowering the priority of a process that kept on swapping for too long.

Encouragement encouragement encouragement

michaeljt — 2009-11-16T13:45:37+00:00

Ahem, I haven't thought that far ahead yet :) So far it is one of a few small projects I have lined up for whenever I have time, but I was posting here in order to get some feedback from wiser minds than my own before I made a start.

A suggestion

jlmassir — 2009-11-15T20:03:07+00:00

Killing a child if there is no memory for a fork-exec is kinda cruel.

A suggestion

Gady — 2009-11-14T20:52:23+00:00

Killing the child process if it uses more than 10% is kinda cruel. There are no rules against the child doing that. What should be done is that in this case the memory is allocated, and if that cannot be done, then the child is killed.

Per-process memory limits?

efexis — 2009-11-14T12:11:39+00:00

You can do this using cgroups Resource Counters / Memory Resource Controller. This lets you see the memory usage estimates for each group (they're estimates because of shared libraries etc), change the limits on the fly, and also change which group a process belongs to (with new processes belonging to the group of its parent process by default).

Encouragement encouragement encouragement

efexis — 2009-11-13T22:32:15+00:00

I (for one) would be most interested in your work. The systems I manage are very binary in whether they're behaved or not, because I have configured them to behave (ie, how much memory is available, decide how much to give to database query caching etc, so everything just works). I try to keep swap file around 384Meg whether the system has 1G or 8G of RAM because that's a nice size to swap stuff out that you don't need to keep in memory, but using disk as virtual RAM is just way too slow, I'd prefer processes be denied memory requests than have them granted at the cost of slowing the whole system down. But all in all, because everything's set up for the amount of memory available, the only time I will get into OOM situations is when there is a runaway process (I manage systems for hosting small numbers of database driven websites, some of them may be developed on windows systems and then moved to the linux system, most are written in PHP which has a very low bar of entry, and so developers often do not have a clue when it comes to writing scalable code).

So, what I would want is something that assumes that most of the system is being well behaved, but will quickly chop off anything that is not, and will stop the badly bahaved stuff from dragging the well behaved stuff down with it. The well behaved stuff quite simply doesn't need managing; that's my job. The badly behaved stuff needs taking care of quickly, by something that your idea seems to reflect *perfectly* (it's not often you read someones ideas and your brain flips "that's -exactly- what I need").

How would I find out if you do get chance to hammer out the code that achieves this? Is there an non-LKML route to watch this (please don't say twitter :-p )

The existence of OOM is one of the few really stupid things in Linux

rlhamil — 2009-11-12T09:50:45+00:00

While optimistic overcommit might be _statistically_ better, it's not deterministic enough
for my liking. And my liking is that everything other than /dev/random is _totally_
deterministic (neglecting external input of course).

(I'd argue that overcommit-by-default is an invitation to denial of service attack, and, if
likely victims were more or less predictable, might be a "covert channel" as well.)

Solaris doesn't do overcommit, but does also offer MAP_NORESERVE, so that individual
mmap() operations can opt out of a reserve, in which case a write to a private mapping
(copy-on-write from a file) can cause the process to receive SIGSEGV or SIGBUS; see

http://docs.sun.com/app/docs/doc/816-5167/mmap-2?l=en&...
(the online version of the mmap() man page for Solaris 10)

I think that all that's missing is:
* a system call to turn on or off similar behavior for heap and/or stack, and to
turn on or off _implicit_ MAP_NORESERVE on all private mappings for that process
and its subsequently forked children (reset on exec)
* a shared library feature to implement system policy specifying which executables
should be be subject to overcommit, with a settable default for all not explicitly specified
* an OS default of no overcommit
* no OOM killer needed

Distros could supply default policy that opted for overcommit on chronically hoggish
(and typically not critical to system integrity) apps such as browsers. People might e.g.
not mind their browser dying a few more times than it would anyway, but might be very
glad to be sure that their X server (desktop user) or database server process was safe from
nondeterministic behavior possibly triggered by another process.

That gets overcommit out of the OS, and pushes the decisions into user space. A process
could always override policy with the system calls, but it would have to know what it was
doing to do that.

The only limitation with implementing the defaults for an executable in the dynamic linker
is that it wouldn't be able to allow overcommit for static executables. If that was a serious
limitation, a new mechanism would be needed to push the policy settings into the kernel,
and execve() (or equivalents) would have to implement them, which is IMO more comprehensive
but otherwise uglier.

Per-process memory limits?

etrusco — 2009-11-12T06:38:10+00:00

(continuing my monologue...)
It would be nice, nonetheless, to be able set the limits from different process (e.g. a window manager) instead of resorting to a launcher and having to define the limit upfront...

Per-process memory limits?

etrusco — 2009-11-12T05:58:09+00:00

Oops, never mind. I should have RTFM before posting :-$

Per-process memory limits?

etrusco — 2009-11-12T05:51:10+00:00

Sorry, I meant a different limit for each process, AFAICT setrlimit/sysctl work globally?

A suggestion

jlmassir — 2009-11-12T05:17:56+00:00

Maybe then the solution to this problem would be:

1. Never allow overcommit when calling malloc
2. Allow overcommit on fork/exec, but kill the child process if it tries to
write to more than 10% of its virtual size.

This way, buggy programs that malloc too much memory and never use them
would be fixed and fork bombs would be killed, while still allowing to do do
system calls between fork and exec.

What do you think?

Per-process memory limits?

oak — 2009-11-11T20:30:22+00:00

It would also be nice if setrlimit() actually would work properly.

...For something else than VmSize limit which is useless for processes
that mmap() files.

death by swap

hppnq — 2009-11-09T12:34:09+00:00

You may want to look at Documentation/cgroups/memory.txt. Otherwise, it seems there is no way to enforce RSS limits. Rik van Riel wrote a patch a few years ago but it seems to have been dropped.

Personally, I would hate to think that my system spends valuable resources managing runaway processes. ;-)

Per-process memory limits?

hppnq — 2009-11-09T09:22:25+00:00

It would be nice if you could trigger actions other than 'die, fiend!' on exceeding a limit

Some attempts to exceed a limit specified using setrlimit are met with an error rather than a kill, and I think all the signals delivered can be handled if appropriate measures are taken. Whether that is a good idea is another question. ;-)

death by swap

michaeljt — 2009-11-09T09:02:01+00:00

> If you get to the point that you're stealing a page from a process simply because that process is over its quota of real memory, you should steal ALL that process' pages. It can't fit its working set into memory, so it isn't going to make decent progress, so the memory you do give it is wasted.
I suppose I see three cases here. One is that the page was part of the process' working set at an earlier point in time, but no longer is. In that case swapping it out is the right thing to do. The other is that the process is in control, but it's working set is bigger than the available memory. Then I agree that there is a good case for putting it on hold until enough memory is available, although that is a non-trivial problem which is somewhat outside of the scope of what I am trying to do. And the third case is the one that I am interested in - a runaway process which will eventually be OOMed. In this case, the quota will stop it from trampling on the working set of every other process in memory in the meantime.

While we are on the subject, does anyone reading this know where RSS quotas are handled in the current kernel code? I was able to find the original patches enabling them, but the code seems to have changed out of recognition since then.

posix_spawn is stupid as a system call

nix — 2009-11-08T23:34:09+00:00

Yeah, breaking the entire installed base of Linux apps would probably be a
*bad* move :) I think, if you wanted to do this, you'd have to introduce a
huge pile of new syscalls and reimplement the old ones as thin wrappers
(inside the kernel so as not to force everyone to upgrade glibc) calling
the new ones.

The existence of OOM is one of the few really stupid things in Linux

nix — 2009-11-08T23:32:16+00:00

Yeah, but forking to exec immediately afterwards is the common case. If
you make that some weird nonportable new variant, 90% of programs are
never going to use it, and none of the rest will until some considerable
time has passed (time for this call to percolate down into the kernel and
glibc --- and try getting this call past Ulrich, ho ho.)

(Anyway, we *have* fork_to_exec_soon(). It's called vfork().)

posix_spawn is stupid as a system call

epa — 2009-11-08T21:26:18+00:00

From a purist point of view, all these 'new' calls are generalizations of the existing ones taking an extra pid argument, so they can just replace them, with the old ones provided by the C library; of course in the real world there is such a thing as backward compatibility :-p.

The existence of OOM is one of the few really stupid things in Linux

epa — 2009-11-08T21:21:52+00:00

I don't think it matters much if a few slightly-buggy applications use the wrong variant. If 90% of userspace including the most important programs such as shells passes the right hint to the kernel, the kernel can make better decisions than it does now, and the need for the OOM killer will be reduced. It's a similar situation with raw I/O, for example: a disk-heavy program such as a database server might know that it will scan through a large file just once. Ordinarily this file's contents might clog up the page cache and evict more useful things. To help get more consistent performance, apps can be coded to hint to the kernel that it needn't bother to cache a particular I/O request. The default is still to cache it, and it's not catastrophic if one or two userspace programs haven't been tuned to use the new fancy hinting mechanism.

but forget the fork then exec situation for a moment and consider the real fork situation. for a large app, most of the memory will never get modified by either process, and so even there it will almost never use the 2x memory that has been reserved for it.

Very true, but of course there's no way for the kernel to know this. I expect most apps would prefer the fork to either succeed for sure, or fail at once if not enough memory can be guaranteed. There may be a few where optimistically hoping for the best and perhaps killing a random process later is the ideal behaviour.

Per-process memory limits?

nix — 2009-11-08T11:56:58+00:00

We *have* setrlimit(). It would be nice if you could trigger actions other
than 'die, fiend!' on exceeding a limit, but it's not essential: you could
have a parent monitoring process which spots the kill and takes
appropriate action (though figuring out which limit the child exceeded, or
whether it exceeded a limit at all or was just randomly killed by the
admin, might be harder).

Per-process memory limits?

etrusco — 2009-11-08T03:19:37+00:00

Not exactly related, but since the article talks about "badly behaved Firefox", it would be kind of nice to have syscalls to set a VM (or RSS, etc) size limit for a process... (and trigger a callback, or suspend or kill the process)

(Sure, an interested application could limit its own heap usage through its memory allocator, but this couldn't stop a DoS from a virus/code injection.)

death by swap

giraffedata — 2009-11-07T17:06:33+00:00

but back in the 70's they realized that most of the time most programs don't use all their memory at any one time. so the odds are pretty good that the page of ram that you swap out will not be needed right away.

I think you didn't follow the scenario. We're specifically talking about a page that is likely to be needed right away. It's a page that the normal page replacement policy would have left alone because it expected it to be needed soon -- primarily because it was accessed recently.

But the proposed policy would steal it anyway, because the process that is expected to need it is over its quota and the policy doesn't want to harm other processes that aren't.

What was known in the 70s was that at any one time, a program has a subset of memory it accesses a lot, which was dubbed its working set. We knew that if we couldn't keep a process' working set in memory, it was wasteful to run it at all. It would page thrash and make virutally no progress. Methods abounded for calculating the working set size, but the basic idea of keeping the working set in memory or nothing was constant.

death by swap

dlang — 2009-11-07T10:01:11+00:00

if the program you are stealing the memory from needs it to make progress, you would be right.

but back in the 70's they realized that most of the time most programs don't use all their memory at any one time. so the odds are pretty good that the page of ram that you swap out will not be needed right away.

and the poor programming practices that are commone today make this even more true

death by swap

tdwebste — 2009-11-07T03:39:30+00:00

On embedded devices I have constructed processing states with runit to control the running processes. This simple but effective long term scheduling to avoid out memory/swapping, works well when you know in advance what processes will be running on the device.

death by swap

giraffedata — 2009-11-07T01:21:03+00:00

If you get to the point that you're stealing a page from a process simply because that process is over its quota of real memory, you should steal ALL that process' pages. It can't fit its working set into memory, so it isn't going to make decent progress, so the memory you do give it is wasted. You're also wasting the swap I/O it's doing. After a while, after other processes have had a chance to progress, you can swap them out and give the first process the memory it needs. If you can't do that because it's run amok and simply demands more memory than you can afford, that's when you kill that process.

Algorithms for this were popular in the 1970s for batch systems. Unix systems were born as interactive systems where the idea of not dispatching a process at all for ten seconds was less palatable than making the user kill some stuff or reboot, but with Unix now used for more diverse things, I'm surprised Linux has never been interested in long term scheduling to avoid page thrashing.

why not posix_spawn()?

cmccabe — 2009-11-06T23:23:09+00:00

> Any good reasons why vfork() should be avoided?

Ah. Found it.

https://www.securecoding.cert.org/confluence/display/secc...

> Due to the implementation of the vfork() function, the parent process is
> suspended while the child process executes. If a user sends a signal to
> the child process, delaying its execution, the parent process (which is
> privileged) is also blocked. This means that an unprivileged process can
> cause a privileged process to halt, which is a privilege inversion
> resulting in a denial of service.

clone(CLONE_VM) + exec might be the win...

Colin

why not posix_spawn()?

cmccabe — 2009-11-06T22:55:44+00:00

> Any good reasons why vfork() should be avoided?

The manual page installed on my system says that copy-on-write makes vfork unecessary. It concludes with "it is rather unfortunate that Linux revived this specter from the past." :)

However... it seems like the results you've posted show quite a substantial performance gain for vfork + exec as opposed to fork + exec, for processes with large heaps.

Maybe the "preferred" way to do this on Linux would be using clone(2)??

why not posix_spawn()?

mikov — 2009-11-06T19:28:08+00:00

I am not sure what you mean. Unless I am missing something, vfork() is much more flexible and easier to use than posix_spawn().

If your purpose is to call exec() after fork(), you should just be able to mechanically replace all forks() with vforks() and get a big boost.

Toward a smarter OOM killer

dlang — 2009-11-06T19:17:31+00:00

one thing that would be very nice to have show up in top (and similar tools) is how much ram _would_ be needed to satisfy all possible COW splits so that admins could get an idea of how overcommitted the kernel currently is.

I wouldn't be surprised to find that much of the time the ratio of overcommit rises significantly shortly before the OOM killer kicks in.

The existence of OOM is one of the few really stupid things in Linux

dlang — 2009-11-06T19:14:25+00:00

and if an application misuses this and does fork_intend_to_exec_soon() and then doesn't exec soon, what would the penalty be?

if applications can misuse this without a penalty they will never get it right (especially when using it wrong will let their app keep running in cases where the fork would otherwise fail)

but forget the fork then exec situation for a moment and consider the real fork situation. for a large app, most of the memory will never get modified by either process, and so even there it will almost never use the 2x memory that has been reserved for it.

why not posix_spawn()?

khc — 2009-11-06T19:12:37+00:00

you are right that the speed of fork() is seldom noticeable in a GUI program, but it bites me all the time in daemons (big daemon wanting to launch many processes, one by one, do do some tasks). vfork() is too limiting because all you can do after is exec(), but sometimes you do want to extra flexibility that posix_spawn can provide.

I have to admit that I have never checked to see if posix_spawn fits my need, though. Since I only care about linux and posix_spawn on linux is the same as fork()/.../exec(), it's useless for me anyway.

LWN: Comments on "Toward a smarter OOM killer"

death by swap

The existence of OOM is one of the few really stupid things in Linux

posix_spawn is stupid as a system call

death by swap

death by swap

death by swap

death by swap

death by swap

death by swap

death by swap

** Encouragement encouragement encouragement **

A suggestion

A suggestion

Per-process memory limits?

** Encouragement encouragement encouragement **

The existence of OOM is one of the few really stupid things in Linux

Per-process memory limits?

Per-process memory limits?

Per-process memory limits?

A suggestion

Per-process memory limits?

death by swap

Per-process memory limits?

death by swap

posix_spawn is stupid as a system call

The existence of OOM is one of the few really stupid things in Linux

posix_spawn is stupid as a system call

The existence of OOM is one of the few really stupid things in Linux

Per-process memory limits?

Per-process memory limits?

death by swap

death by swap

death by swap

death by swap

why not posix_spawn()?

why not posix_spawn()?

why not posix_spawn()?

Toward a smarter OOM killer

The existence of OOM is one of the few really stupid things in Linux

why not posix_spawn()?

Encouragement encouragement encouragement

Encouragement encouragement encouragement