Not logged in
Log in now
Create an account
Subscribe to LWN
LWN.net Weekly Edition for May 23, 2013
An "enum" for Python 3
An unexpected perf feature
LWN.net Weekly Edition for May 16, 2013
A look at the PyPy 2.0 release
also, actually _using_ swap can be extremely painful, sometime more painful than simply crashing the system (at least that you can have watchdogs for and failover/reboot)
in theory you are right, the system is unpredictable with overcommit enabled. in practice it is reliable enough for _many_ uses
Taming the OOM killer
Posted Feb 5, 2009 8:22 UTC (Thu) by brouhaha (guest, #1698)
swap isn't free, especially on embedded systems
In an embedded system, I wouldn't expect there to be a large number of processes sleeping between a fork() and exec(). If there were, that would most likely be a sign of serious problems, so having a fork() fail under such circumstances seems like a good thing.
also, actually _using_ swap can be extremely painful, sometime more painful than simply crashing the system
If you're concerned about system performance degrading because of excessive swap usage, there's no reason why you can't have a user-space process to act as a watchdog for that problem, which may occur for reasons other than memory "overcommit".
I've been involved in engineering a number of embedded products that had to have high reliability in the field, and I would not dream of shipping such a product with the kernel configured to allow memory overcommitment. Even though you can _usually_ get away with it, "usually" isn't good enough. There simply needs to be enough memory (and/or swap) to handle the worst-case requirements of the system. Otherwise it _will_ fail in the field, and thus not be as reliable as intended.
Posted Feb 5, 2009 8:33 UTC (Thu) by dlang (✭ supporter ✭, #313)
if you could allocate the address space but then tell the kernel "don't really use it" you may be ok, but how is that different from the current overcommit?
you _are_ overcommiting (compared to what is acceptable to the system's performance) and counting on the efficiancies of COW to keep you from actually using the swap space you have comitted.
the only difference is that you overcommit up to a given point (at which time your allocations start failing, which may also cause the system to 'fail' as far as the user is concerned)
i fully agree that there are situations where disabling overcommit is the right thing to do. However, I am also seeing other cases where allowing overcommit is the right thing to do.
Posted Feb 5, 2009 9:07 UTC (Thu) by brouhaha (guest, #1698)
if you could allocate the address space but then tell the kernel "don't
really use it"
you may be ok, but how is that different from the current
The system should not be designed or configured such that the kernel can fail to provide memory that has been committed, because there is NO reasonable recovery mechanism for that. It is far easier to handle memory allocation problems gracefully when the error is reported at the time of the attempt to commit the memory, rather than at some random future time.
Posted Feb 5, 2009 9:16 UTC (Thu) by dlang (✭ supporter ✭, #313)
there are times for that, but there are also times when 99.999% reliability is good enough.
Posted Feb 5, 2009 10:23 UTC (Thu) by epa (subscriber, #39769)
(On a related note, the kernel is free to refuse any request for extra memory, and can do so for its own reasons. So for example if a process needs to fork() then the memory allocation would normally succeed, on the assumption that the extra memory probably won't be used, but provided there is enough swap space to back it up just in case. Whereas an explicit memory allocation 'I want ten gigabytes' could, as a matter of policy, be denied if the system doesn't have that much physical RAM.)
Posted Feb 5, 2009 21:03 UTC (Thu) by dlang (✭ supporter ✭, #313)
but if you have large programs that may need to fork, it's not nessasarily the case that it's 'a slightly larger amount of swap'. I've seen people arguing your point of view toss off that a large system should be willing to dedicate a full 1TB drive just to swap so that it can turn overcommit off. in practice, if you end up using more than a gig or so of swap your system slows to a crawl
Posted Feb 5, 2009 22:26 UTC (Thu) by epa (subscriber, #39769)
It is rather a problem that merely adding swap space as available means it can then be used by applications just as willingly as physical RAM. Perhaps a per-process policy flag would say whether an app can have its memory allocation requests start going to swap (as opposed to getting 0 from malloc() when physical RAM is exhausted). Then sysadmins could switch this flag on for particular processes that need it.
Posted Feb 6, 2009 0:45 UTC (Fri) by nix (subscriber, #2304)
Posted Feb 5, 2009 12:42 UTC (Thu) by michaeljt (subscriber, #39183)
Posted Feb 5, 2009 15:44 UTC (Thu) by hppnq (guest, #14462)
I would rather have my system slow to an unusable crawl if I was confident that it would come out of it again at some point. Even then, I can still press the reset button, which is what I have usually ended up doing in OOM situations anyway.
On your home system this makes some sense, but all this goes out the window once you have to take service levels into account.
Posted Feb 6, 2009 7:46 UTC (Fri) by michaeljt (subscriber, #39183)
Posted Feb 6, 2009 8:58 UTC (Fri) by dlang (✭ supporter ✭, #313)
you probably already have tools in place to detect processes dieing and either restart them (if the memory preasure is temporary) or failover to another box (gracefully for all the other processes on the box)
Posted Feb 5, 2009 21:06 UTC (Thu) by dlang (✭ supporter ✭, #313)
if you are willing to hit reset in this condition then you should be willing to deal with the OOM killer killing the box under the same conditions.
Posted Feb 6, 2009 8:00 UTC (Fri) by michaeljt (subscriber, #39183)
Posted Feb 12, 2009 19:14 UTC (Thu) by efexis (guest, #26355)
Personally I have swap disabled or set very low, as a runaway process will basically mean I lose contact with a server, unable to log in to it or anything, until it has finished chewing through all available memory *and* swap (causing IO starvation, IO being the thing I need to log in and kill the offending task) until it hits the limit and gets killed.
Everything important is set to be restarted, either directly from init, or indirectly from daemontools or equivalent, which is restarted by init should it go down (which has never happened).
Posted Feb 13, 2009 23:33 UTC (Fri) by michaeljt (subscriber, #39183)
Posted Feb 14, 2009 0:03 UTC (Sat) by dlang (✭ supporter ✭, #313)
how much swap did you allocate? any idea how much was used?
enabling overcommit with small amounts of swap will allow large programs to fork without problems, but will limit runaway processes. it's about the textbook case for using overcommit.
Posted Feb 16, 2009 9:04 UTC (Mon) by michaeljt (subscriber, #39183)
Definitely too much (1 GB for 2 GB of RAM), as I realised after reading this: http://kerneltrap.org/node/3202. That page was also what prompted my last comment. It seems a bit strange to me that increasing swap size should so badly affect system performance in this situation, and I wondered whether this could be fixed with the right tweak, such as limiting the amount of virtual memory available to processes, say to a default of 80 percent of physical RAM. This would still allow for large processes to fork, but might catch runaway processes a bit earlier. I think that if I find some time, I will try to work out how to do that (assuming you don't answer in the mean time to tell me why that is a really bad idea, or that there already is such a setting).
Posted Feb 16, 2009 15:38 UTC (Mon) by dlang (✭ supporter ✭, #313)
Posted Feb 17, 2009 8:23 UTC (Tue) by michaeljt (subscriber, #39183)
Indeed. I set ulimit -v 1600000 (given that I have 2GB of physical RAM) and launched a known bad process (gnash on a page I know it can't cope with). gnash crashed after a few minutes, without even slowing down my system. I just wonder why this is not done by default. Of course, one could argue that this is a user or distribution problem, but given that knowledgeable people can change the value, why not in the kernel? (Again, to say 80% of physical RAM. I tried with 90% and gnash caused a noticeable performance degradation.) This is not a rhetorical question, I am genuinely curious.
Posted Feb 17, 2009 8:29 UTC (Tue) by dlang (✭ supporter ✭, #313)
the distro is in the same boat. if they configured it to do what you want, they would have other people screaming at them that they would rather see the computer slow down than have programs die (you even see people here arguing that)
Posted Feb 17, 2009 14:27 UTC (Tue) by michaeljt (subscriber, #39183)
It does take a decision though - to allow all programmes to allocate as much RAM as they wish by default, even if it is not present, is very definitely a policy decision. Interestingly Wine fails to start if I set ulimit -v in this way (I can guess why). I wonder whether disabling overcommit would also prevent it from working?
Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds