Sponsored link
Vyatta –
Linux & Open Source
Alternative to Cisco –
Advanced Routing,
Firewall, VPN, QoS..
Free Download ->
| |||||||||||
Short topics in memory managementShort topics in memory managementPosted Mar 7, 2007 6:18 UTC (Wed) by jcm (subscriber, #18262)Parent article: Short topics in memory management
I'd like to see a real memory reservation system in Linux, using kernel events to notify userland of memory pressure and having applications dynamically adjust their own caches/etc. as needed.
I brought this up on lkml a while ago, and suggested a combination of netlink events passed over D-BUS to listening applications as one way to achieve this - for example, telling Firefox that memory pressure means it needs to stop using ludicrously large in-memory caches of the last 10 pages visited so the rest of the user session isn't killed by the thrashing to disk that follows.
Having briefly played with this, I lost interest due to lack of time (and perhaps also a lack of intimate knowledge of the VM system) but I'd like to revive this if anyone is interested in helping out.
Jon.
(Log in to post comments)
Short topics in memory management Posted Mar 7, 2007 9:02 UTC (Wed) by k8to (subscriber, #15413) [Link] I'm ignorantly curious about the problems you're trying to address. Isn't the solution to "the firefox problem" to cache almost all data on disk? Linux seems really great about keep disk data in memory when there isn't pressure and reclaiming it when necessary. Am I misunderstanding that problem, and is what you are trying to address really a different problem?
Short topics in memory management Posted Mar 7, 2007 11:28 UTC (Wed) by njs (subscriber, #40338) [Link] The cache's he's referring to in the firefox case are explicitly the in-memory ones -- while in principle an app could write such caches out to disk and trust the OS to make accessing that data as efficient as reasonable, in practice this may require major restructuring of your app. Firefox *does* cache pages on disk, of course, but it also has a smaller in-memory cache of fully parsed and rendered pages; the point of this is to avoid the re-rendering (and re-executing javascript, etc.) overhead of going to the normal disk cache (which just stores html files), and there's no reasonable way to take giant in-memory object graphs and write them do disk directly.
Now OTOH one would hope Linux would be clever about reclaiming memory by writing such memory out to swap... though various things could thwart this, e.g., if Firefox has a garbage collector tromping over those pages and keeping them resident, or if its memory is sufficiently fragmented that objects that are just being kept around as a cache, and objects that are in active use, happen to share pages.
These are just potential problems, though; I have no idea if they come up in practice. The firefox I'm typing this in seems to be about 300 megabytes total, but about 100 of those have been pushed out to swap, even during active use on a lightly loaded system without much memory pressure. Now, of course, it might be that those 100 megabytes are unused because they were actually leaked or something, but it is at least suggest that the "firefox problem" is not as bad as the original poster thinks.
Short topics in memory management Posted Mar 7, 2007 13:35 UTC (Wed) by dion (subscriber, #2764) [Link] I think you might be right.
Firefox probably thinks we are living in 1970 when there wasn't anything called virtual memory, so it allocates gobs of RAM for caches and does explicit file io.
A much better way would be to have just one cache in one huge memory mapped file, that way there is never any explicit io and the kernel is free to write stuff out to disk whenever it needs to.
It would make a persistent cache a bit tricky, you could never do anything with the cached files using normal tools and crashing would be interesting, though.
Someone working on firefox should take a peek at Varnish, which is incredibly fast because it lets the kernel do all the hard memory/io work.
Stuck with 1970s VM? Posted Mar 7, 2007 15:57 UTC (Wed) by im14u2c (subscriber, #5246) [Link] Don't forget, Firefox has to run on Windows, too. ;-)
Stuck with 1970s VM? Posted Mar 7, 2007 16:38 UTC (Wed) by ibukanov (subscriber, #3942) [Link] As far as I remember Windows got memory-mapped files starting at least with Windows-95. Surely the API is different, but it is straightforward to isolate the difference behind a tiny wrapper. The best thing about it is that since the memory mapping removes explicit reads/writes from the application, the end result is more portable code.
Stuck with 1970s VM? Posted Mar 7, 2007 16:44 UTC (Wed) by im14u2c (subscriber, #5246) [Link] *whoooosh*
That's the sound of a tongue-in-cheek jab at Windows going over your head. :-)
Short topics in memory management Posted Mar 7, 2007 16:50 UTC (Wed) by ibukanov (subscriber, #3942) [Link] > It would make a persistent cache a bit tricky,
The trick is to use relative offsets instead of pointers in the data that goes into the mapped file. It has an extra benefit of a more secure code since it is easier to add bound-check for the offsets then checking the pointers.
Short topics in memory management Posted Mar 7, 2007 17:47 UTC (Wed) by zlynx (subscriber, #2285) [Link] Checking offset bounds may look easier than a pointer, but it isn't really. A pointer is nothing but an offset from memory location 0, after all.
if( pointer > mmap_base && pointer < mmap_end ) ...
Using an unsigned int offset instead saves you from checking the lower bound, but you lose that savings again the instant you segment your memory into different uses, like using some for text and some for graphics.
Short topics in memory management Posted Mar 8, 2007 23:57 UTC (Thu) by im14u2c (subscriber, #5246) [Link] "A pointer is nothing but an offset from memory location 0, after all." Well, these days it is on most architectures...
Short topics in memory management Posted Mar 7, 2007 17:49 UTC (Wed) by zlynx (subscriber, #2285) [Link] One problem with large mmaps is fragmented virtual memory. It is very possible with 32-bit addressing to end up in a situation where you *cannot* allocate a single block of virtual. Especially if you are doing things where you grow your maps, reposition them, etc. Then you need crazy stuff like a mmap defragmenter where you flush it all to disk, unmap it and remap everything again.
All the problems could be solved of course, but who is going to rewrite all that code? Again.
Short topics in memory management Posted Mar 7, 2007 18:14 UTC (Wed) by dion (subscriber, #2764) [Link] Well, the solution is to "not do that, then".
If I were to implement a browser cache in a memory mapped file then the mapping would be the first thing to get allocated and it would never change.
|
|||||||||||