| |||||||||||||
More memory - less OOM situationsMore memory - less OOM situationsPosted Jun 10, 2010 13:12 UTC (Thu) by NAR (subscriber, #1313)Parent article: Another OOM killer rewrite complaints about poor choices are less common than they once were I think the main reason for this is that nowadays we have more memory in our desktops than the KDE/GNOME applications plus firefox uses and even when gnome-panel manages to allocate half GB and the system starts swapping, the user has a good chance to kill the process manually instead of relying on the OOM killer.
(Log in to post comments)
More memory - less OOM situations Posted Jun 10, 2010 13:38 UTC (Thu) by zlynx (subscriber, #2285) [Link]
Have you ever tried to operate a Linux system that is out of memory and swapping?
It isn't impossible but it can easily take 15 seconds and more between commands. It can take a minute to switch to a text console, another minute to log in, then more minutes to locate and kill the offending process.
Usually its easier to reboot the machine.
More memory - less OOM situations Posted Jun 10, 2010 14:33 UTC (Thu) by NAR (subscriber, #1313) [Link]
Actually yes. Usually it happens on one morning when I turn off the screen lock and I notice that it takes too long time to change workspaces. CPU usage is practically 0 and the system has just started to swap, so it's not that hard to kill firefox and opera. I don't like to lose the shell histories, the 100+ opened files in vim, etc. so I try to avoid logging out, even though I have to do it every other month when the desktop environment leaked too much memory.
More memory - less OOM situations Posted Jun 10, 2010 18:17 UTC (Thu) by iabervon (subscriber, #722) [Link]
Isn't that situation actually more like that you've got bad processes which have used up nearly all of RAM, but they're asleep, and other processes that use some memory have run overnight, and pushed parts of sleeping processes into swap? That is, you're noticing at a point in memory consumption when the total usage is more than RAM and less than swap, but the system's active set fits in RAM, so the only symptom is that things have to be swapped back in, and you notice and fix the problem. This would suggest that the reason people are happier in general is that margin between where swap gets used at all and where the system is thrashing is bigger, and the size of the one-time delay on waking up is big enough that people notice and respond.
More memory - less OOM situations Posted Jun 11, 2010 20:45 UTC (Fri) by giraffedata (subscriber, #1954) [Link] This would suggest that the reason people are happier in general is that margin between where swap gets used at all and where the system is thrashing is bigger, and the size of the one-time delay on waking up is big enough that people notice and respond. It doesn't look to me like that margin is the difference. I think it's just the level at which swap gets used at all. In the old days, you operated normally in the aforementioned margin, so never knew you were close to the upper edge of it, and also had no separate reason to go shoot memory hogs manually. Today, as soon as you enter the margin, you're both bothered and warned of memory excesses, so you do the manual OOMK.
More memory - less OOM situations Posted Jun 14, 2010 10:58 UTC (Mon) by alsuren (guest, #62141) [Link]
I've always blindly followed the old recommendation of having twice as much swap as RAM, so I don't think I've ever hit the OOM condition. Swappy-swappy-death is no fun though. Is there a way to make the OOM killer detect swappy-swappy-death and kick in then?
Maybe we should all just disable swap entirely, and then forkbomb our machines periodically to see how good the OOM killer really is.
More memory - less OOM situations Posted Jun 17, 2010 22:47 UTC (Thu) by efexis (guest, #26355) [Link] I do, either disable or severely limit swap. The only time I ever tend to hit OOM is when something's misbehaving (infinite loop + memory leak, fork bomb etc) so I get my system back once swap's exhausted, no matter how much swap there is, it'll never be bigger than infinite, so making it big seems to have the single effect of delaying how quickly I can recover the system.But best thing I've found to do is just put everything in seperate cgroups with memory limits set at around 80% by default, so no single thing can take out the whole system. OOM killer works just great then, killing where needs to be killed. Example cgroup report: $ cgroup_report CGroup Mem: Used %Full apache 244,396K 24.43% compile 524K 0.03% mysql51 81,340K 8.32% mysql55 85,756K 8.78% netserv/courier/authdaemond 16,444K 3.28% netserv/courier/courier 52,452K 5.24% netserv/courier/esmtpd-msa 260K (no limit) netserv/courier/esmtpd 2,768K 0.55% netserv/courier/imapd 785,568K 78.55% netserv/courier/pop3d 8,492K 0.84% netserv/courier 0 netserv/courier/webmaild 0 netserv 176,320K 12.03% network 300K 0.03% rsyncd 264K 0.10% sshd 282,904K 28.29% system/incron 0 system 15,588K 1.55% system/watchers/sif 1,144K 3.66% system/watchers 0 . 6,936K (no limit)Systems without cgroups though, yikes, when they go wrong they make my head hurt. I'm sure cgroups are the answer to everything :-p
More memory - less OOM situations Posted Jun 19, 2010 1:52 UTC (Sat) by giraffedata (subscriber, #1954) [Link] The old way to do this is with rlimits. I've always set rlimits on vsize of every process -- my default is half of real memory (there's a bunch of swap space in reserve too). Before Linux, rlimits (under a different name) were the norm, but on Linux the default is unlimited and I think I'm the only one who changes it. Rlimits have a severe weakness in that a process just has to fork to get a whole fresh set of limits, but they do catch the common case of the single runaway process.
More memory - less OOM situations Posted Jun 20, 2010 8:53 UTC (Sun) by efexis (guest, #26355) [Link]
Yep I used to set it in my own code to protect the rest of the system from my mistakes, and still do when am forced to use a system without memory cgroups, but many of the servers I help to maintain host sites for people developing in php, who 'coincidentally' (seeing as this is respectable lwn not slashdot!) have no clue about the underlying system or scalability, and they do all kinds of crazy things!
Also I guess I don't really understand the internals of how rlimits work, what they apply to, things like the allocating of costs of shared memory etc, whereas I followed the development of cgroups so have a better understanding. So do processes inherit the parents limits but get their own counters? So you set say 10M data limit, you can end up with three processes with 5M each just fine? I think that's what you meant by new set of limits, or do you mean forked procs get reset to the hard limit or no limit? I guess this is with the exception of the user wide process limit?
It's hard to tell the answers to these just from the man pages which is all the looking I've done so far, 'tho a few mins experimenting I could figure it out, but any main/quick details you wouldn't mind sharing off the top of your head I would be grateful, there're still a couple of older webservers without cgroup support I'm trying to help family with so rlimits would no doubt be helpful!
Cheers :-)
More memory - less OOM situations - rlimits Posted Jun 20, 2010 17:16 UTC (Sun) by giraffedata (subscriber, #1954) [Link] I guess I don't really understand the internals of how rlimits work But what you're asking about is externals. I don't know much about the internals myself. Some of this is hard to document because the Linux virtual memory system keeps changing. But I think this particular area has been neglected enough over the years that the answer that I researched about 10 years ago is still valid. The rlimit is on vmsize (aka vsize). Vmsize is the total amount of virtual address space in the process. That includes shared memory, memory mapped files, memory mapped devices, and pages that use no memory or swap space because they're implied zero. Procps 'ps' tells you this figure with --format=vsize (it's also "sz" in ps -l, but in pages, while "vsize" is in KiB). A new process inherits all its parent's rlimits, so a parent with a 10M limit can use 5M and fork two children that do likewise and use a total of 15M just fine. The user-wide process rlimit is an exception to the idea that a process can extend its resource allocation through the use of children, but it's not an exception to the basic idea that a child inherits the parent's full limit -- it's just a bizarre definition of limit intended to hack around the basic weakness of rlimit that we've been talking about. Apparently, fork bombs were a big enough problem at some time to deserve a special hack.
More memory - less OOM situations Posted Jun 17, 2010 23:38 UTC (Thu) by efexis (guest, #26355) [Link] Is it definitely leaky, as opposed to perhaps it coincides with backups or something io intensive that can fill your page cache, vacating stuff out to swap.One horrible but quick fix for that is to have something like this set to run in the morning so everythings loaded back into memory ready for you to use it:
#!/bin/bash
getnum() {echo $2}
cached=$( getnum `grep ^Cached: /proc/meminfo` )
memFree=$( getnum `grep ^MemFree: /proc/meminfo` )
swapTotal=$( getnum `grep ^SwapTotal: /proc/meminfo` )
swapFree=$( getnum `grep ^SwapFree: /proc/meminfo` )
swapped=$(( $swapTotal - $swapFree ))
available=$(( $cached + memFree ))
[[ "$available" > "$swapped" && "$swapped" > 0 ]] && {
echo "Loading everything back from swap"
swapoff -a
echo "Enabling swap again"
swapon -a
}
And of course if it isn't that then it won't bother doing anything so at least there should be nothing to lose! More elegant fixes depend on your setup.
|
|||||||||||||