Weekly Edition
Return to the Press pageSponsored link
Vyatta –
Linux & Open Source
Alternative to Cisco –
Advanced Routing,
Firewall, VPN, QoS..
Free Download ->
| |||||||||||
When Linux Runs Out of Memory (O'ReillyNet)
O'ReillyNet looks
at what happens when a Linux system runs out of memory. "Perhaps
you rarely face it, but once you do, you surely know what's wrong: lack of
free memory, or Out of Memory (OOM). The results are typical: you can no
longer allocate more memory and the kernel kills a task (usually the
current running one). Heavy swapping usually accompanies this situation, so
both screen and disk activity reflect this."
(Log in to post comments)
When Linux Runs Out of Memory (O'ReillyNet) Posted Dec 2, 2006 0:40 UTC (Sat) by azhrei_fje (guest, #26148) [Link] As someone who teaches Linux Internals :) let me say that this article is a bit out of date (although essentially correct). I was a bit dismayed by the simplistic descriptions of how things worked, but I can forgive that: perhaps I have too much background in the topic and I'm being picky. But when the author stated that there was a kernel patch to allow the user process to consume the entire 4GB on a 32-bit processor, I stopped reading. (This has been a core component of the kernel for a long time now. As in, "measured in years.") I'm sorry, but I rate the article "-1: Informative". :-/
When Linux Runs Out of Memory (O'ReillyNet) Posted Dec 3, 2006 20:29 UTC (Sun) by ken (subscriber, #625) [Link] We have a visitor from the future. As far as I know the kernel.org kernel only allows the 3-1 split.
When Linux Runs Out of Memory Posted Dec 2, 2006 5:12 UTC (Sat) by Tara_Li (subscriber, #26706) [Link] I've been hitting this one a lot less recently than I had been - mostly because Mozilla is killing itself off before it can actually bring the OOM killer into play. Of course, talking to the Mozilla people about "memory leaks" is useless - that's not a memory leak, that's a feature! And you can turn it off with this particular config setting - except that doesn't seem to work.
I turned to setting a ulimit -v on my mozilla process. That way it kills itself off well before making my machine descend into swap hell. A gigabyte works well - keeps the gig of RAM and the half-gig of swap from going insane.
Still, I have to wonder - why is Mozilla dying when it can't allocate more memory? Why doesn't it put up a note "Sorry: Can't open new tab or window, or load that page - please close other tabs or windows to free up memory!" (Or perhaps throw away some of that famous fast-back/fast-forward cached crap?) Programs shouldn't die just because they hit a minor stumbling block like can't allocate more memory! Especially interactive programs that could actually have the user do something to alleviate the problem!
Doesn't help that Linux doesn't actually have a decent way to tell how much memory a process is using! You've got the %MEM, VSZ and RSS columns from "ps aux". You've got VIRT, RES, and SHR from "top". You've got a *BUNCH* of things from pmap, especially with the -d and -x options. And everything you find on the Internet will either pick *one* column, and tell you that's the one that matters - or it'll tell you that none of the numbers is worth the CPU time it took to come up with them. Everyone goes on and on about the value of shared libraries - why does my seamonkey process come up with only 3576K of shared memory according to "pmap -d" out of 724732K mapped?
Memory usage? Posted Dec 2, 2006 6:29 UTC (Sat) by socket (guest, #43) [Link] Doesn't help that Linux doesn't actually have a decent way to tell how much memory a process is using!Okay, I'll bite. I've been using Linux for waaaay too long to not have a good answer for this. I have no idea how to interpret the various memory-related output of ps, top, and (new to me) pmap. Can somebody refer us to practical information about memory usage on Linux? I'm okay with a complicated answer, so long as it's a good answer.
Memory usage? Posted Dec 2, 2006 8:43 UTC (Sat) by scottt (subscriber, #5028) [Link] I use the sum of writable and X server memory.This number ignores the address space taken up by code in executables and shared libraries.
pmap is new to me, also Posted Dec 2, 2006 18:52 UTC (Sat) by pr1268 (subscriber, #24648) [Link] pmap really must be new, since it's the first shell command I've come across in ages that doesn't have a man page. For those curious, I'm running Slackware 11. pmap is located in /usr/bin and is included in Slackware as part of the procps-3.2.7-i486-1 package. And, yes, I do have the man pages package installed also. Just curious, in reference to Mozilla/Firefox's greedy memory usage: how do other Web browsers (Konqueror, Opera, etc.) compare? Is Seamonkey as bad as Mozilla/Firefox? Thanks.
Re: pmap is new to me, also Posted Dec 3, 2006 3:12 UTC (Sun) by ldo (subscriber, #40946) [Link] >pmap really must be new, since it's the first shell command I've come>across in ages that doesn't have a man page. It does on both my SuSE and Gentoo systems.
pmap is new to me, also Posted Dec 3, 2006 15:11 UTC (Sun) by k8to (subscriber, #15413) [Link] I compared mozilla and firefox a long time ago, and firefox was a substantial, but not overwhelming win.
I compared firefox 1.0 and firefox 1.5 respectively against opera, konqueror, and epiphany, all of which were significant wins. Where it really gets nuts is just how much X server memory gets used over time.
Memory usage? Posted Dec 3, 2006 3:26 UTC (Sun) by bk (guest, #25617) [Link]
function mem() {
if [ -z $1 ]; then
echo "usage: mem [process]"
return
fi
TPID=$(pidof $1)
if [ -z $TPID ]; then
echo "error: unknown process"
return
fi
pmap -d $TPID |grep -E "mapped|${TPID}"
}
I prefer to look at the 'writeable/private' number, but keep in mind that it doesn't include shared objects which can be significant.
The moral of the story is that there is no simple answer because it isn't a simple problem.
Memory usage? Posted Dec 3, 2006 17:30 UTC (Sun) by Tara_Li (subscriber, #26706) [Link] I'm not looking for a 1-number solution!
I'd be happy with:
-------
XXXXX KB shared with other processes through libraries
XXXXX KB mapped into process memory space for communication with devices
XXXXX KB locked against paging out
-------
The -v option might offer add in other technical details, when run as root a -a option might summarize this information for all running processes.
I'm willing to handle complicated! I can't find anyway to find out what the complications *ARE* short of going through the kernel source code - and I'm not *THAT* much of a hacker.
-------
Picking on Firefox/Seamonkey is strictly because I'm tired of restarting it every day just to keep things under control. I currently use Seamonkey 1.0.5 (from the Fedora Extras repository), with the SessionSaver extension (no, it's not causing the situation - I got it *BECAUSE* of this situation!). Shortly before crashing, it's common for pmap to show the Seamonkey process using 1.1-1.3GB. After restarting, and all tabs finishing reloading (including the back/forward caches for the tabs, *DESPITE* having browser.sessionhistory.max_total_viewers set to 0 [I can tell by the cookies that get asked to be set when I restart]), Seamonkey is using perhaps 300MB or so.
I picked Mozilla (using Mozilla as an umbrella term for both Firefox and Seamonkey) to comment on because it seems to be one of the more extreme cases.
Memory usage? Posted Dec 3, 2006 12:41 UTC (Sun) by aleXXX (subscriber, #2742) [Link] Lubos Lunak about measuring memory consumption:http://www.kdedevelopers.org/node/1445 http://www.kdedevelopers.org/node/1567 Alex
Memory usage? Posted Dec 4, 2006 1:07 UTC (Mon) by k8to (subscriber, #15413) [Link] I was disappointed, reading through this, to realize it's even harder to look at these values than I realized. Even Lubos Lanak doesn't seem to grasp the full picture.
He worries about memory which is swapped to disk not being included in the figure that suggests the everyman's "memory footprint" of a program, but programs may often leave data lying around that they don't need for a while (or perhaps ever), which the kernel eventually migrates to swap. Since these pages aren't fighting to stay in ram, they won't have a detrmental effect on memory availability for other programs, and so often you don't want to include these pages in a footprint value. Of course they are pages that the program has allocated, so when looking at program behavior, you may want them still.
It all depends.
What I find truly amazing is the bittorrent client Azureus which is written in java. At least running on java 1.4.2, it was capable of retaining hundreds of megabytes in real ram even while the system was near memory exhaustion, and and while it as doing _no real work_. How many objects do you need to send a message to the tracker every few minutes to find out there are no new peers, and how do you spread these objects out over (200M/4k) 500 pages?
Memory usage? Posted Dec 4, 2006 22:02 UTC (Mon) by mitchskin (subscriber, #32405) [Link] There was a nice comment on Slashdot a while ago (that I can't find now) that said that one of the reasons that Java processes tend to have a large resident set is that the GC touches lots of pages. Even with a generational compacting collector I would think that the GC is going to be touching a large fraction of the heap in the time scale over which the VM system is deciding what to keep resident.
Memory usage? Posted Dec 5, 2006 22:55 UTC (Tue) by dicej (subscriber, #36115) [Link] On the other hand, copying collectors can actually improve locality of reference by grouping linked data together. This increases the probability that closely-related data will occupy a minimal set of pages. So a garbage-collected system that uses a lot of linked data structures can potentially use the VM system (not to mention cache) more efficiently than manual memory management.
As for Java, even simple programs end up using a large portion of its class library indirectly, which may help account for its resident set size.
When Linux Runs Out of Memory Posted Dec 2, 2006 7:56 UTC (Sat) by khim (subscriber, #9252) [Link] Still, I have to wonder - why is Mozilla dying when it can't allocate more memory? Why doesn't it put up a note "Sorry: Can't open new tab or window, or load that page - please close other tabs or windows to free up memory!" To put up such a note you need... right: still more memory :) You can't win. When there are no memory essintially any operation can blow. Even something like "a = b" is not guaranteed (if a was allocated but never used then there are possibility that kerbel just lazily allocated address space and not memory). It is possible to write wokring program - but it's very-very-very hard: 10 (or may be 100) times harder then "normal" program. And if guys can not even write the program without buffer overflow errors... The interesting thing is: why can not we do it even "Windows way" (actually it was invented many years before Windows, but Windows does it today): monitor quota and when it's 95% full - warn the user. Not 100% bullet-proff, but will work in a lot of cases... It's done with quote in disk, but why not for quota in RAM ? You don't even need this code in Mozilla itself...
When Linux Runs Out of Memory Posted Dec 2, 2006 22:27 UTC (Sat) by JoeBuck (subscriber, #2330) [Link] Mozilla caches pages in memory. If a memory allocation request fails, it can free cache and try again. It can also keep an "emergency supply" around to deal with error reporting if it can't obtain memory any other way.
Re: Out of Memory Posted Dec 3, 2006 3:10 UTC (Sun) by ldo (subscriber, #40946) [Link] >To put up such a note you need... right: still more memory :) You can't win. Yes you can. A standard technique from MacOS days (going back over 20 years) was to preallocate a reserved memory pool specifically for emergencies like this. In those days, 32 kiB was considered enough.
When Linux Runs Out of Memory Posted Dec 3, 2006 12:36 UTC (Sun) by aleXXX (subscriber, #2742) [Link] "do it the Windows way" ?At work I have a Linux and a Windows XP machine, the Windows machine has been mainly used for Outlook etc. Recently I also did a lot of compiling on XP. Man, does that suck ! On Linux I can have run "make -j4" on a single processor machine and my other applications still behave properly, a bit slower, but no significant degredation in "Quality of Service". I mean, when I switch between GUI apps, the still react immediately to my actions. Now the same thing last week under XP, this means, not the same, but only *1* compile process running and at the same time trying to do something else. The whole box felt bascially like blocked. Everything was slow and sluggish as hell. It took seconds for things to react. almost unusable. I can't even say "worse then under Linux", because under Linux (and FreeBSD etc.) it's not problem at all, but under XP it was almost unusable. If any Linux kernel ever behaved so bad this would be fixed within the next few weeks. Alex
When Linux Runs Out of Memory Posted Dec 3, 2006 19:28 UTC (Sun) by drag (subscriber, #31333) [Link] Well that is scedualing issues, probably not so much memory management.
Remember how back in Redhat 8 days and such when distros typically raised the X server's priority to increase the responsiveness of the GUI? Well obviously Linux doesn't have that issue anymore.
I remember when the preemptive-ness features were aviable for me to use back when 2.6 kernel came out... I was amazed that I could no longer tell when my CPU was pegged at 100% utilization. Everything was still very responsive. Before I could tell just by moving the mouse around.
In the issues of proccess scedualing, especially when it comes to how XP handles multiple proccessors sucks badly for a multitasking desktop environment when compared to Linux.. Although I am not realy in a big position to tell since it's been ages since I actually used XP for any length of time other then when fixing it for friends and family.
When Linux Runs Out of Memory Posted Dec 3, 2006 21:13 UTC (Sun) by HenrikH (guest, #31152) [Link] Not neccesarily scheduling issues, on Windows, all memory is put to swap first and only sent to RAM if used, and if not used for a short while -> to swap again (atleast that is the bahavier that I see on the Windows systems that I administrate). This probably makes it easier to halt the malloc() and add more swap when running out of memory while in the Linux case even the Kernel itself can have troubles finding enough free memory to even page to disk when memory is low. A kind of trade-off I suppose.
When Linux Runs Out of Memory Posted Dec 3, 2006 21:16 UTC (Sun) by HenrikH (guest, #31152) [Link] Problem is that it isn't Mozilla that trows itself out when it has no more memory to allocate. The main issue here is that malloc only returns NULL when libc has detected an error in the free memory tables, i.e. NULL is never returned as a result of no memory available to Mozilla has no way to know that it is running out of memory.
Instead of malloc returning NULL. the kernel sends the OOM to the Mozilla process.
When Linux Runs Out of Memory Posted Dec 4, 2006 20:36 UTC (Mon) by oak (subscriber, #2786) [Link] Just turn off the overcommit feature from the /proc. Mozilla willthen just die earlier, but it will die to malloc() returning NULL (as I doubt all of Firefox handles that gracefully), not to OOM-killer killing it when Mozilla uses the memory it got (for the first time).
Can programs mark memory volitile? Posted Dec 2, 2006 20:17 UTC (Sat) by aashenfe (subscriber, #12212) [Link] Maybe What is need for programs like Firefox/Sea monkey is a way to allocate volatile memory. So Firefox could allocate volatile pages. If the kernel gets into a situation where it needs memory, it could just throw out those pages. When you use the back button, Firefox would make a system call to make the memory non volitile again. If it returns false then the page is gone. Firefox could just download it, or if was a post or something, prompt to refresh. A slower back button is rare cases is better than a thrashing machine.
Maybe Firefox could then be a little more aggressive about memory use with less adverse effect.
Other programs might benefit also.
Can programs mark memory volitile? Posted Dec 3, 2006 0:26 UTC (Sun) by khim (subscriber, #9252) [Link] More aggressive ? It can gobble gigabytes already! No need to make it even bigger memory hog...
Can programs mark memory volitile? Posted Dec 3, 2006 3:14 UTC (Sun) by aashenfe (subscriber, #12212) [Link] Yes, it's true that Firefox can take a lot of memory.
The main point is a lot of the memory it uses is basically cache, and the kernel should be able to treat it the same as it treats it's own cache.
Throw it out quickly when needed.
Can programs mark memory volitile? Posted Dec 3, 2006 15:29 UTC (Sun) by drag (subscriber, #31333) [Link] And another thing about Firefox (and I expect other Mozilla-based projects) is that they are self-rendering..
I mean that the icons, the toolbars and other such things are rendered along with the webpage. At least I beleive so..
So what happens is that although Firefox isn't a huge huge memory hog, what happens is that the memory it does use is unique to that instance of Firefox.
So depending on the environment your using this can have a very big or a very small impact.
The examples I've heard about is in regards to setting up X terminals..
However on a very multiuser environment this has less of a impact since it is's libraries and such are easy shared by many multiple instances from many multiple users. In comparision to firefox there isn't nearly as much unique information per user as there is with OO.org. So ultimately you can have situations were you have a large number of users and they can all be using OO.org fine, but when a significant number of people try to use Firefox you can end up with memory exaustion pretty easily.
Then when you through in things like flash animation it can easily bring a powerfull system to it's knees.
So for this reason it's recommended that if your running a 'thin client's it would probably be better to just have firefox locally run on the terminals rather then on the application server.
At least that is my own understanding of it. Don't have any personal experiance with managing lots of users in a distributed X environment.
I suppose if your having memory problems then maybe it would be easier to run Epiphany instead of a Mozilla browser. I don't know about Seamonkey, but with my system Epiphany has a lot less tendancy to consume ram.
Although naturally webpages are just going to eat a lot ram, especially X server memory.
With konqueror it's even more memory efficient then either of those two, but it tends to hammer my cpu (not a whole lot, but it is enough that it turns on my cpu fan in my laptop, which starting firefox and epiphany won't) and with kde dependancies that starts up with it it's ultimate memory usage is huge compared to running Epiphany.
Can programs mark memory volitile? Posted Dec 4, 2006 20:47 UTC (Mon) by oak (subscriber, #2786) [Link] > So Firefox could allocate volatile pages. If the kernel gets into> a situation where it needs memory, it could just throw out those pages. > When you use the back button, Firefox would make a system call to make > the memory non volitile again. Why not just use BSD allocator which uses madvise() to tell about this stuff to kernel? No need to modify Firefox, just use an allocator that suits Firefox better... FYI: Besides X server resource usage, I think Firefox problem really is heap fragmentation, and I've seen mentions that the memory usage isn't as bad on non-Linux systems so an alternative to the Glibc allocator (which is good general allocator) might serve Firefox better.
Can programs mark memory volitile? Posted Dec 4, 2006 21:55 UTC (Mon) by rafdz (guest, #41427) [Link] BSD allocator uses madvise on a page only when all the chunks it contains have been freed with free().also BSD allocator rounds allocation requests to powers of two but doesn't add per chunk header as glibc allocator does.so fragmentation may vary with size of allocated chunks:glibc will behave badly when chunks are small and BSD will behave badly when size of a chunk is just above a power of two.
Re: When Linux Runs Out of Memory Posted Dec 3, 2006 3:07 UTC (Sun) by ldo (subscriber, #40946) [Link] How odd. LWN already posted a perfectly good description of the OOM killer over two years ago.
|
|||||||||||