| |||||||||||||
replaced by new patch from ..replaced by new patch from ..Posted Sep 29, 2005 5:56 UTC (Thu) by hisdad (subscriber, #5375)Parent article: Swap prefetching
I saw it the other day, riel or molnar IIRC.
Apparenty huge improvements to interactivity while doing huge file copies.
That might make this irrelevant because if never get swapped out in the first place.
--John
(Log in to post comments)
replaced by new patch from .. Posted Sep 29, 2005 7:08 UTC (Thu) by ncm (subscriber, #165) [Link] Would somebody please explain the above comment, comprehensibly?
For my part, I would prefer a much more aggressive prefetcher. Any page that's unused is wasted -- providing it can be reclaimed quickly because there's also a copy somewhere on disk. Similarly, any page of swap that doesn't mirror an otherwise-unbacked page in RAM is wasted, and slows down reclaiming that page for some other use.
Throughput's nice for benchmarks and kernel compiles, but most of us suffer far more from abysmal latency than from marginally-reduced throughput.
Andrea Arcangeli's per-task-predictive-write-throttling Posted Sep 29, 2005 10:34 UTC (Thu) by farnz (guest, #17727) [Link] I believe he's referring to Andrea Arcangeli's per-task-predictive-write-throttling.If I've understood it properly, the patch measures the rate at which each task is writing to disk; if maintaining that rate would cause the kernel to start flushing buffers via pdflush in the near future, the task's timeslice is used to flush instead. The result is that tasks doing the odd write here and there aren't affected, since they don't cause enough dirty pages. Tasks like cp, which dirty lots of pages, get paused to write out these pages (cleaning them), and making these pages eligible for eviction. This reduces the memory pressure cp-type tasks can induce without killing their performance; cp would eventually pause for the writeout, once it had dirtied as much RAM as it could. The patch just brings this pause forward, so that it doesn't dirty too much RAM before it writes out.
Andrea Arcangeli's per-task-predictive-write-throttling Posted Sep 29, 2005 10:41 UTC (Thu) by nix (subscriber, #2304) [Link] It's a good idea, that patch, definitely. For a pathological example, my backup process involves multi-hundred-Mb copies to a packet-written CD-RW. This generally floods almost all of memory with dirtied pages and then flushes them to this rather slow device, making the machine slow and swappy for a quarter-hour or so until the flush is done: with predictive write throttling, I'd expect to see a steady trickle instead, at something close to it.
(Even the current behaviour is much better than Linux-2.4, where the fact that block devices didn't have their own queues led to the entirety of X and everything I was doing freezing solid within a second or two of the flush beginning; the system was more swappy than normal because memory was filled with all those dirty pages, and a tiny and otherwise-unnoticeable bit of swapping or paging was stuck behind the vast heap of stuff destined for the CD-RW.)
replaced by new patch from .. Posted Sep 29, 2005 10:36 UTC (Thu) by nix (subscriber, #2304) [Link] The patch you're discussing is intended to stop processes that would dirty many pages from filling up memory with dirty pages far faster than the device can emit them, and only then blocking; instead it forces them to block sooner, before so much of memory is filled up by its dirtied pages.
This is orthogonal to the patch under discussion, which is arranging that when something *has* filled memory and then freed it all again, that useful stuff gets put back in there sooner rather than later: after all, memory can be filled by many things other than dirtied pages awaiting flushing (e.g., large ld(1) runs ;) )
replaced by new patch from .. Posted Sep 29, 2005 13:50 UTC (Thu) by liljencrantz (subscriber, #28458) [Link] These algorithms may be orthogonal in what they do, but the problems they solve have a strong overlap.
The writeout on dirty pages patch fixes filter-type programs that read/write out huge amounts of data, but never use the same data for more than a short period of time. This is somewhat related to the new instructions in the next-generation consoles that support writing out data without touching the caches, only that the kernel autodetects such uses, so the program doesn't have to explicitly tell the OS what data not to put in the cache. This patch should in theory work very well for backup programs, indexers, media players and many other types of cache killers.
The swap prefetching patch would also solve the issue of filter-type programs, though significantly less efficiently, since a filter program would first force the entire system to swap out and then slowly 'swap in' during half a minute or so. But swap prefetching also fixes a slightly different type of problem, namely when an application uses a huge amount of memory and then exits (or at least free()s the memory). This includes applications that do some rather complicated things during initialization, like OpenOffice, as well as memory-hungry 'one-shot' programs, like yum.
replaced by new patch from .. Posted Sep 29, 2005 21:08 UTC (Thu) by nix (subscriber, #2304) [Link] It doesn't fix reading-cache-killers: they'll still push other data out of the cache.
But it handles the other half of the job. :)
replaced by new patch from .. Posted Sep 29, 2005 21:35 UTC (Thu) by giraffedata (subscriber, #1954) [Link] The early writing of dirty pages patch doesn't actually address, except in an incidental way, the problem of single-use pages, either read or write. If you read or write a gigabyte of virtual memory on a .5 gigabyte system, you will sweep physical memory with the patch just like without. To solve that problem, you need to change the cache replacement policy, not the prewriting policy.Actually, as VM changes so frequently, I don't know just what the present cache replacement policy is; maybe it's already sweep resistant; my point is that early writing isn't about that. Early writing makes it so all those page frames wasted with pages that will never be used again are at least clean, so when they do get reclaimed, the reclaimer doesn't have to wait. The wait for page laundry is shifted to the guy diryting pages away from the innocent bystander competing for real memory.
replaced by new patch from .. Posted Sep 29, 2005 23:24 UTC (Thu) by farnz (guest, #17727) [Link] AIUI, the big gain of Andrea's early write-out patch is that the VM has a strong preference for evicting clean pages (nearly free) over dirty (expensive I/O needed). Because Andrea's code stops streaming writes like cp from dirtying lots of one-use pages, it makes the pages that cp is dirtying much more attractive to the VM when it's hunting for freeable pages.
replaced by new patch from .. Posted Sep 29, 2005 20:15 UTC (Thu) by hisdad (subscriber, #5375) [Link] Ah yes, angeli. so i didn't recall correctly afterall!
I had mostly thought of swap in the case of large copies and not considered
It will be interesting to see what they are like in practice.
--John
|
|||||||||||||