Stopping unwanted OOM killer experiences [LWN.net]

There has, in recent times, been a small increase in the number of complaints from users who have seen processes killed by the kernel in response to an out-of-memory (OOM) situation. The only problem is that the system should not have been quite that hard up for memory at the time. Even if the user is doing something which requires completely irrational amounts of memory ("yum update", say), it seems like the system should have been able to muddle along without killing low-priority processes, like the ssh server. These unwanted OOM killer experiences have driven a few developers to take a closer look at what was going on.

Marcelo Tosatti has been working on the problem for a bit; he put together a patch which tries to avoid invocations of the OOM killer if things might get better soon. The idea is that, while a full scan of a memory zone may have failed to turn up any free pages, it may have kicked I/O into motion that will, very soon, make some pages free. So the OOM killer is kept in its cage until the no-memory situation has persisted for a few seconds. Marcelo reported that this patch improved things significantly for his test cases.

It turns out, though, that the real problem was elsewhere; the token-based thrashing control patch appears to be the real culprit. This patch, remember, tries to reduce system thrashing in memory-constrained situations by exempting one process at a time from the page reclaim mechanism. That process will, in theory, make use of its sheltered time to make some real progress before the token moves on and its pages are, once again, subject to eviction. The token-based mechanism has been shown to truly improve the situation when memory is tight.

Until it gets too tight, as it turns out. A process which needs a page, but which does not hold the token, may find that all of the (otherwise) reclaimable pages belong to the process currently holding the token. The unlucky process thus finds no pages to grab, and pushes the big red OOM button. The system is not truly out of memory, however; it has simply been told that all the good pages are temporarily off limits.

Rik van Riel put his finger on the problem, and Andrew Morton put together a simple patch to fix it. Essentially, the VM subsystem will now ignore the swap token when finding reclaimable pages gets too hard. During normal operation, the token-based mechanism holds sway, but it can be set aside as a preferable alternative to killing random processes in the system. The patch appears to have solved the problems without taking away the benefits of the token-based approach.

Marcelo acknowledged that this was the right fix, grumbled that he had wasted a bunch of time, and promised "Next time I should be looking into the easy stuff before trying miraculous solutions." It was his work, however, which shone a light on the problem in the first place, and led to its eventual solution.

(Log in to post comments)

Stopping unwanted OOM killer experiences

Posted Nov 18, 2004 11:12 UTC (Thu) by rwmj (subscriber, #5474) [Link]

I've been hit by the OOM killer recently too.

There seems to be a simple and obvious fix: allow the user to specify a list of processes and a priority for each. The OOM killer would kill low-priority processes first.

On my system, the list would look something like this:

10 X                    # killing X should be a final resort
 8 firefox-bin          # don't have session management
 6 apache               # development server, doesn't matter if it is killed
 4 wineserver           # generally running IE, so unimportant
 2 gaim, ical, xpostit  # no state, doesn't matter if killed
 0 artsd, esd, kdeinit  # I try to kill these regularly anyway, but they
                        # still manage to pop up somehow

Rich.

Stopping unwanted OOM killer experiences

Posted Nov 18, 2004 21:09 UTC (Thu) by tkreagan (subscriber, #4548) [Link]

Doesn't it strike anyone as a little crazy the way kernel development keeps getting backed into more and more special cases? It seems like things are turning into an unworkable mess, with OOM killers tripping over thrash-swappers tripping over pluggable schedulers, two separate volume management solutions, and an endless number of constantly changing kernel data structures.

I realize that there are significant benefits to letting the kernel evolve on its own, but am I the only one who thinks its time to focus on pruning, bug fixes, and validating existing structures before we take off on the next flight?

Or have I spent too much time in the OpenBSD world?

Stopping unwanted OOM killer experiences

Posted Nov 27, 2004 16:56 UTC (Sat) by riel (subscriber, #3142) [Link]

Alternatively, you might not have spent enough time looking at the OpenBSD VM, which has its own share of special cases ;)

Every VM I've seen (and yes, I have looked at all the BSDs, XNU, Mach and others) are chock full of special case handling. Take a look at vm/vm_glue.c next time you're in FreeBSD land...

Yes, the swap token thrashing prevention code has a few corner cases, but it doesn't require anywhere near the number of magic constants that traditional Unix and BSD memory scheduling algorithms require.

Stopping unwanted OOM killer experiences

Posted Nov 19, 2004 21:18 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

I sure hate to see people working on tuning the OOM killer, as if it's a normal part of memory management. The OOM killer is just a less drastic form of panic. When it runs, the system is broken. There are even cases where a panic would be better.

I understand that it's very difficult to get memory management right, and as a band-aid, the OOM killer can be better than a reboot in the same way that an oops is often more convenient than a full panic.

The OOM killer should never run when there are pageouts in progress. I don't care how slow they are. If the pageout device is broken and the pageout is actually indefinite, that should be handled like any pageout I/O error. If things are just slow, a user space process that monitors performance and kills some processes to speed up others would be appropriate. The kernel should kill processes only when it is backed into a kernel-level corner, like where it doesn't have enough swap space to back the virtual memory it has created and is thus deadlocked. And users ought at least to have the option of allocating resources so the kernel doesn't get deadlocked.