| |||||||||||||
Requesting 'real' memoryRequesting 'real' memoryPosted Feb 2, 2008 2:00 UTC (Sat) by giraffedata (subscriber, #1954)In reply to: Requesting 'real' memory by njs Parent article: Avoiding the OOM killer with mem_notify
Turning off overallocation just means that you get a different error handling API How is getting killed by the OOM Killer an error handling API? Turning off overallocation means you get an error handling API where you had none before. It certainly doesn't stop you from running the system out of memory. Turning off overallocation for one process doesn't stop you from running the system out of memory; that's why the OOM Killer is still there. But he only kills other processes. The connection is that if Process X is not overallocating memory (swap space), then the OOM Killer is guaranteed to be able to relieve memory pressure without having to kill Process X. You can't say that about an overallocating process. Think of it as two separate pools of swap space; one managed by simple allocation; the other with optimistic overallocation and an OOM Killer. A process decides which one works best for it. all you have to do to force allocation is to touch pages after you allocate them, No, that's not enough. In overallocating mode, the swap space does not get allocated until the kernel decides to steal a page frame. By then, the process is in no position to be able to deal with the fact that there's not enough swap space for him.
(Log in to post comments)
Requesting 'real' memory Posted Feb 3, 2008 1:39 UTC (Sun) by njs (guest, #40338) [Link] Ah, I see, that's not what overallocation means. It has nothing to do with swap. As far as the memory manager is concerned, the total amount of memory available in the system is RAM + swap -- if you have 1G ram and 2G swap, then you have 3G total memory. This 3G total is distributed among processes. If overallocation is turned off, then each time a process calls malloc() (well, really mmap()/sbrk()/fork()/etc., but never mind), either some pages from that 3G are shaved off and reserved for that process's use, or if there are not enough pages remaining then the syscall fails. If overallocation is turned on, then malloc() never actually allocates memory. What it does instead is set up some virtual pages in the process's address space, and then the first time the process tries to write anywhere on each of those not-really-there pages, the process takes a fault, the memory manager allocates one of those 3G of pages, sticks it in place of the fake page, and then finally allows the write to continue. The upside of this is that if a process malloc()'s a big chunk of memory and then only uses part of it, it's as if they only malloc()'ed exactly what they ended up needing. The downside is that since the actual allocation is now happening somewhere in the middle of a single user-space cpu instruction, there's no way to signal an error back to the process if the allocation fails, and so in that case the only thing you can do is wake up an OOM-killer. > The connection is that if Process X is not overallocating memory (swap space), then the OOM Killer is guaranteed to be able to relieve memory pressure without having to kill Process X. Which means that this just isn't true. Memory pressure is caused by actually-allocated pages, and the only difference between a process using overallocation and one that isn't is that the overallocating process may have some virtual pages set up to trigger allocation sometime in the future. Whether such pages exist has no bearing whatsoever on memory pressure *now*. The only way the OOM-killer can relieve memory pressure is to kill off processes that are using memory, and Process X qualifies.
Requesting 'real' memory Posted Feb 3, 2008 2:29 UTC (Sun) by giraffedata (subscriber, #1954) [Link] Though you start out saying overallocation has nothing to do with swap, your second sentence shows that it is strongly related to swap, saying that swap space is half the equation in determining how much memory is available to allocate. But what overallocation are you talking about? You describe it like some well-defined Linux function. Are you talking about something that Linux implements? AFAIK, Linux does not implement a per-process overallocation mode, and we were talking about what should be. It's clear how the mode should work: The same way it does in AIX, which is what I described. "Turning off" overallocation has to mean that once you've allocated memory, you can use it and can't be killed for lack of memory. Otherwise, why bother having the mode? And the way to do that is to allocate swap space to the process at the moment you allocate the virtual addresses to it. You could alternatively permanently allocate some real memory to the process, but that would be really wasteful (and we already have a means to do that: mlockall()). BTW, it's not helpful to talk about memory not being actually allocated by malloc (brk/mmap/etc). malloc() does actually allocate virtual memory. Allocating swap space and allocating real memory are separate things that exist in support of using virtual memory which has been allocated. I also don't think "virtual" amounts to "fake." It's a different form of memory.
I assume you mean pages of real memory. Filling up of real memory is the primary cause of memory pressure, but it's easy to relieve that pressure: just push data you aren't using out to swap space and free up the real memory. When swap space is full, that's when the pressure backs up into the real memory and the OOM Killer is needed. That's why I say swap space is the key to giving guaranteed-usable virtual memory to a process.
Requesting 'real' memory Posted Feb 3, 2008 4:06 UTC (Sun) by njs (guest, #40338) [Link] Obviously we're totally talking past each other, but I'll try one more time... >Though you start out saying overallocation has nothing to do with swap, your second sentence shows that it is strongly related to swap, saying that swap space is half the equation in determining how much memory is available to allocate. Well, sure, swap is, by any measure, an important part of a VM system, but that doesn't mean it's "strongly related" to any other particular part of a VM system. My point is that from the point of view of overallocation, the difference between swap and physical RAM is just irrelevant. > But what overallocation are you talking about? You describe it like some well-defined Linux function. Are you talking about something that Linux implements? Yes. I'm talking about "overallocation" or "overcommit", which in this context is a technical term with a precise meaning. When people are talking about it in this thread, they are referring to a particular policy implemented by the Linux kernel and enabled by default. Evidentally you haven't encountered this particular design before, which is why I described it in my previous comment... >AFAIK, Linux does not implement a per-process overallocation mode, and we were talking about what should be. No, AFAIK it doesn't, but it does support a global overallocation/no-overallocation switch, and it's obvious what it would mean to take that switch and make it process-granular. Maybe there's yet another policy that Linux should implement, but if you want to talk about that then trying to redefine an existing term to do so will just confuse people. >And the way to do that is to allocate swap space to the process at the moment you allocate the virtual addresses to it. Huh, so is this how traditional Unix works? Is this tight coupling between memory allocation policy and swap management policy the original source of that old advice to make your swap space = 2xRAM? I've long wondered where that "rule" came from. I guess I've heard before that in traditional Unix all RAM pages are backed on disk somewhere, either via the filesystem or via swap, but I hadn't thought through the consequences before. I'm guessing this is the original source of confusion. Linux doesn't work like that at all; I'm not sure there's any modern OS that does. In a system where RAM is always swap-backed, having 1G RAM and 2G of swap means that all processes together can use 2G total; in Linux, they can use 3G total, because if something is in RAM it doesn't need to be in swap, and vice-versa. (What happens in your scheme if someone is running without swap? I bet there are people in this thread who both disable overallocation and run without swap (hi Zooko!).) "Allocated memory" in my comment really just means "anonymous transient data that the kernel has committed to storing on the behalf of processes". It can arrange for it to be stored in RAM, or in swap, or whatever. There is no such thing as "allocated swap" or "allocated RAM" in Linux. (Except via mlockall(), I guess, if you want to call it that, but I don't think calling it that is conceptually useful -- it's more of a way to pin some "allocated memory" Does that make my previous comment make more sense?
Requesting 'real' memory Posted Feb 4, 2008 9:43 UTC (Mon) by giraffedata (subscriber, #1954) [Link] but it does support a global overallocation/no-overallocation switch, and it's obvious what it would mean to take that switch and make it process-granular. That would be only slightly different from the scheme I described. The only difference is that the global switch lets you add a specified amount of real memory to the size of swap space in calculating the quota. If you could stop the kernel from locking up that amount of real memory for other things, you could have the OOM-proof process we're talking about, with less swap space. I think the only reason I haven't seen it done that way is that swap space is too cheap to make it worthwhile to bring in the complexity of allocating the real memory. If I were to use the Linux global switch, I would just tell it to consider 0% of the real memory and throw some extra disk space at it, for that reason.
What happens in your scheme if someone is running without swap? I bet there are people in this thread who both disable overallocation and run without swap They don't have that option. The price they pay to have zero swap space is that nothing is ever guaranteed to be free from being OOM-killed. Which is also the case for the Linux users today who disable overallocation and run without swap.
Requesting 'real' memory Posted Feb 5, 2008 13:44 UTC (Tue) by filipjoelsson (subscriber, #2622) [Link] > But what overallocation are you talking about? You describe it like some > well-defined Linux function. Are you talking about something that Linux > implements? AFAIK, Linux does not implement a per-process overallocation > mode, and we were talking about what should be. I think the overallocation he talks about is on the userspace level. When I'm programming an application to read, store and analyze data on a laptop (a field application - the user wants to have preliminary analysis right away), I can make a big fat allocation to use for cache. I store the data in a database, which also uses a big fat cache. Until now, I have been had to either make both caches no larger than a quarter the size of the RAM (roughly), since swapping out really defeats the point of the cache. If I had bigger caches, I could just hope that the sum of the actually used memory would not be larger than RAM (or else it'd start swapping out, or if I run without swap - trigger the OOM). With this patch, I can safely overcommit - and when I get the notification, I can cut down on the caches and survive. The analysis step of the app will be slower, but my user will not have been near a crash - and the analysis would have been slower anyway because of swapping. One difference is that he can run without swap in a much more safe manner. Anyway, this is not at all a case of a partial crash. You could argue that it is a case of sloppy programming, but why should I reinvent memory managing? I'd much rather let the kernel do that for me. Oh, and lastly - on embedded platforms, you often have to do without swap. Swapping to flash does not strike me as a good idea.
Requesting 'real' memory Posted Feb 8, 2008 5:00 UTC (Fri) by goaty (guest, #17783) [Link] Don't forget about stack! For heavily multi-threaded processes on Linux, stack is usually the biggest user of virtual memory. As an aside, I know some operating systems use a "guard page" stack implementation, where writes into the first page off the bottom (top?) of the stack trigger a page fault, which allocates another page worth of real memory, and also moves the "guard page". The benefit being that virtual memory use is much closer to actual memory use, and turning off overcommit is much more viable. The downside is that the ABI requires the compiler to generate code to probe every page when it needs to allocate a stack frame larger than the page size. Which ironically can end up using more real memory than Linux's virtual-memory-hungry approach.
Requesting 'real' memory Posted Feb 8, 2008 21:23 UTC (Fri) by nix (subscriber, #2304) [Link] Linux has used a guard-page stack implementation since forever. It's transparent to userspace: the compiler doesn't need to do a thing. (Obviously when threading things get more complex.)
|
|||||||||||||