2.6 swapping behavior [LWN.net]

2.6 swapping behavior

Posted May 6, 2004 21:47 UTC (Thu) by iabervon (subscriber, #722)
Parent article: 2.6 swapping behavior

I think the issue is really that stuff used for a minute five hours ago is preferred to stuff used for an hour six hours ago. Stuff that's of lasting significance is more likely to be needed again after a period of the system being idle, although it may be good to evict while the system is busy.

Ideally, things would get swapped out while updatedb ran, and then swapped back in when nothing had used the memory cached for updatedb. But it wouldn't just be program memory getting swapped back in; it would be clever to pull into cache files and directories that get used a lot, so that (for example), your Mozilla cache would be in memory again when you got up.

(Log in to post comments)

2.6 swapping behavior

Posted May 7, 2004 7:59 UTC (Fri) by njhurst (guest, #6022) [Link]

I don't understand why updatedb needs so much cache memory? Surely it only needs to keep a stack of inodes from root to the current point in the filesystem in memory. Once it has looked at a file that file's memory should be returned to the pool immediately. I don't know how to force the kernel to do this though.

(This is obviously updatedb specific information, but maybe it would be easier to fix updatedb than everything else?)

2.6 swapping behavior

Posted May 7, 2004 21:53 UTC (Fri) by addw (guest, #1771) [Link]

Trouble is that the kernel doesn't know that the updatedb is not going to look at those files ever again (well, 'till it runs again tomorrow). But the blocks from the file system are left in memory on the grounds that something recently used it likely to be used again in the near future.

Simple prediction doesn't always work.

2.6 swapping behavior

Posted May 13, 2004 0:19 UTC (Thu) by njhurst (guest, #6022) [Link]

I agree, my point is just that maybe some thought could be put into making updatedb more well behaved, rather than trying to get that behaviour directly out of the kernel?

I think it is allowable to have user space programs try to optimise their behaviour with the kernel :)

2.6 swapping behavior

Posted May 14, 2004 11:17 UTC (Fri) by forthy (guest, #1525) [Link]

IMHO the initial priority of a just-allocated or just-loaded buffer is too
high. That's why memory hogs (which claim a page once and never look at it
again) swap out everything else in Linux, and updatedb also does the same
thing. The pages of OpenOffice, which have been used and reused all day
long over and over again have lower priority for the kernel than a newly
allocated use-once page.

But as long as kernel developer stick their thumbs in their ears and sing
"lalala", this won't change.

Another note: updatedb *only* reads the names of a file system. Well, it
also checks if a name is a directory, and that forces it to read the
entire inode. The real problem here is that updatedb tries to solve
something the file system should do itself (especially if you think of it
like Hans Reiser does). Why is it impossible for the file system to keep
the file names in a database like updatedb, and answer queries like locate
directly? Hint: BeOS did something like that.

2.6 swapping behavior

Posted May 11, 2004 1:28 UTC (Tue) by mcelrath (guest, #8094) [Link]

updatedb (and many other applications) need to be using O_DIRECT or some other flag that indicates explicitly that files will be read exactly once, and putting the file in the buffer cache isn't necessary.

There is no way for the kernel to predict that some process named 'updatedb' will read every file exactly once, but another process named 'mozilla' likes to read the same file over and over. It's up to the application to specify that.

AFAIK O_DIRECT is not the appropriate flag for this, because read/write buffers must be page aligned to use it. An O_NOCACHE flag has been proposed before (especially by streaming video folks) but has not been added, though I did see an implementation once. I think an O_NOCACHE or O_READONCE is the solution to this...