| |||||||||||||
Optimizing Linux with cheap flash drivesOptimizing Linux with cheap flash drivesPosted Feb 20, 2011 13:45 UTC (Sun) by nix (subscriber, #2304)In reply to: Optimizing Linux with cheap flash drives by Tara_Li Parent article: Optimizing Linux with cheap flash drives
I really wish I could turn off caching on some files - there's no good reason to read-cache an MP3 or video file - it's going in and out of the app, and pretty much wastes memory when it's done so. It takes code changes, but posix_fadvise() takes a POSIX_FADV_NOREUSE which may be useful for mpg123 et al (if indeed they are actually using most of the data only once: I don't know enough about media formats to know if this is true). But mpg123 doesn't use it... and the kernel doesn't use it either. Oops.
(Log in to post comments)
Optimizing Linux with cheap flash drives Posted Feb 21, 2011 19:32 UTC (Mon) by Tara_Li (subscriber, #26706) [Link]
Yeah, oops.
I'm looking at the fact that by far, the thing slowing my system down the most is my hard drives. And looking at the specs of the newer hard drives, I'm not seeing any real advances in anything *BUT* capacity, and maybe in really large reads/writes.
But the myriad of little reads and writes from Gnome/KDE putting snippets of XML into individual files, the .cache directory, the web browser and it's plethora of relatively tiny files, the often-times widely scattered reads and writes of swap...
These are getting no loving what so ever. Meanwhile, everyone's saying just add RAM. Except my motherboard only goes up to 8GB of RAM, and I'm not so sure that's really going to improve things that much.
And I'm caching stuff that it doesn't seem to make sense to - multi-megabyte audio files, half-gigabyte and up video files - and once a program allocates itself some RAM, it *SURE* doesn't seem to want to give it up. (Of course, this is really hard to be sure of, because any long time linux user will tell you that the memory columns in ps don't really mean anything useful, and when you dig further - it's hard to be sure the *KERNEL* knows what memory belongs to who - although it does do a good job of cleaning up after the application crashes, so it knows *SOMETHING*.)
So why not a two-pronged approach. A file-system bit, like the sticky bit used to be to force caching of some kind (Linux nearly ignores it, apparently) - but to tell the kernel that said file is *NOT* to be cached. With /etc/ , /bin, etc all on a flash drive, there'd really be little reason to cache them, so you could mark them so - and you could also mark media files likewise, and not need to change the apps. There could be a flag for opening the file to force caching of it (though I'm not sure I can see why), allowing user space to handle things as it feels the need, but if it doesn't, the mostly right thing happens anyway. updatedb - would use the call to open *EVERYTHING* in non-caching mode, vlc wouldn't need to worry about it - could do its own limited caching it needs to operate - since it could assume media files wouldn't be cached if the user doesn't want them to be - while other programs could say cache it - even if it doesn't think it needs to be.
And for that matter - does the OOM killer create its own swap for some reason? I turned off swap a few minutes ago, after a reboot, opened some memory hogs, and when memory filled up (according to XOSview - I'm sure not the most accurate system meter, but it sure makes sense to me!), I got *MASSIVE* hard drive I/O, which pretty much tied the whole system into knots for well over 15 minutes (before I gave X it's version of the vulcan nerve pinch - took it another 5 to recognize *THAT*). What was the hard drive doing thrashing, since there was no designated swap for memory to be pushed off to temporarily?
I'm going to admit it here - I'm a Linux *USER*. I can - and have - done make clean ; make ; make install to build a kernel, or to install a program that didn't have a convenient package (and why can't I, as a user, install a package to *my* home directory, instead of having to go to root and install it system wide? I know, I wander off topic...) I just want a machine that runs nicely. The recent bit that does some kind of ulimiting per console (some kind of easy kernel patch, that someone else showed could be almost as easily done with a few lines in .bashrc ?) helped a fair bit - but still, every day I gnash my teeth as I wait for my hard drives to catch up with everything else.
Optimizing Linux with cheap flash drives Posted Feb 21, 2011 19:44 UTC (Mon) by nix (subscriber, #2304) [Link]
You won't be seeing slowdowns from writes unless a lot is being written or you are terribly short of memory, as they can always be cached and written back later. It's blocking for reads that's killing you.
The best way to speed up reads on current systems is probably to use RAID: add lots of disks and reads speed up enormously, given a fast enough bus (at a cost in write speed). e.g. my four-way RAID-5 here combines four fairly slow low-power disks to given an aggregate read speed between 190Mb/s and 250Mb/s. That knocks the socks off any single disk: even high-speed ones at the fast outer edge of the disk are slower than this RAID array is at the slow inside.
It might also be worth trying cachefs, caching onto a USB key (and not caching the filesystem on which your media files are located), but I'm not sure that this will actually gain you anything. (Worth trying though.)
If you turn off swap and the system is already short of memory performance will hit a wall, because all of a sudden rarely used but dirtied pages of non-file-backed memory *have* to be held in RAM, rather than being swapped to disk once and forgotten about. (glibc creates several hundred Kb of these for every program that might use locales, which pretty much means anything that calls printf(), i.e. everything). So all of a sudden your available memory plunges and lots of frequently-used but read-only pages of program text get forced out of memory, leading to major thrashing as they get pulled back off disk all the time. (So you will probably find iostat and/or blktrace reporting major *reads* in this situation, but not major *writes*.)
Optimizing Linux with cheap flash drives Posted Feb 21, 2011 21:58 UTC (Mon) by Tara_Li (subscriber, #26706) [Link]
How do you tell Linux not to cache a filesystem? I can't find it in mount options.
Optimizing Linux with cheap flash drives Posted Feb 22, 2011 0:24 UTC (Tue) by nix (subscriber, #2304) [Link]
This isn't about the page cache. I was talking about FS-Cache, which is a separate filesystem layer which can be used to cache other filesystems on local media (such as, well, flash drives, or, more often, hard drives).
-- unfortunately it needs specific support for each filesystem, and it doesn't look like this has been added for any non-networked filesystems. Curses.
(See Documentation/filesystems/caching/.)
Optimizing Linux with cheap flash drives Posted Feb 22, 2011 10:22 UTC (Tue) by cladisch (✭ supporter ✭, #50193) [Link]
> once a program allocates itself some RAM, it *SURE* doesn't seem to want to give it up.
A program's allocations tend to be too fragmented, so functions like free() usually do not even try to give the memory back to the OS.
> Of course, this is really hard to be sure of, because any long time linux user will tell you that the memory columns in ps don't really mean anything useful, and when you dig further - it's hard to be sure the *KERNEL* knows what memory belongs to who - although it does do a good job of cleaning up after the application crashes, so it knows *SOMETHING*.
The biggest problem are shared libraries; the kernel knows who uses them, but it's not clear how their memory should be counted. (Which process gets charged for that memory? And when it exits, does the memory usage of the other processes suddenly increase?)
> What was the hard drive doing thrashing, since there was no designated swap for memory to be pushed off to temporarily?
Normal data can get swapped out, if there is swap.
Optimizing Linux with cheap flash drives Posted Feb 25, 2011 21:39 UTC (Fri) by oak (subscriber, #2786) [Link]
> The biggest problem are shared libraries; the kernel knows who uses them, but it's not clear how their memory should be counted. (Which process gets charged for that memory?
Look at the PSS figures in /proc/PID/smaps file.
Nice tool for that could be e.g. smem: http://www.selenic.com/smem/
(You can just apt-get it and then do "smem --pie=name".)
> And when it exits, does the memory usage of the other processes suddenly increase?)
If you're looking at PSS figures, yes.
Installing to your home drive Posted Feb 22, 2011 13:46 UTC (Tue) by pflugstad (subscriber, #224) [Link] (and why can't I, as a user, install a package to *my* home directory, instead of having to go to root and install it system wide? I know, I wander off topic...) $ ./configure --prefix=/my/home/dir $ make $ make installBeyond this, you need to dig into the Makefile (or whatever the build utility is) to figure out how it's installed. Very often there is a install_prefix variable of some kind.
Installing to your home drive Posted Feb 22, 2011 18:15 UTC (Tue) by nye (guest, #51576) [Link]
The point was to install a *package*. To my knowledge no widely used package manager allows per-user package installation.
Installing to your home drive Posted Feb 23, 2011 5:45 UTC (Wed) by idupree (subscriber, #71169) [Link]
Most package managers offer binary (already-compiled) packages. Most Linux/Unix software can't have its prefix changed *after* compilation. Everyone's user directory has a different path (thus, prefix). Thus, problems.
Options I've heard of & played with: GoboLinux's "Rootless" project is a system for installing from source in your home directory (on any distro). ZeroInstall I believe does source and binary (not sure how it manages binary) without root privileges. Some NixOS research has looked into rewriting the paths in compiled packages (though not changing its total-number-of-characters length).
Installing to your home drive Posted Feb 23, 2011 9:34 UTC (Wed) by nix (subscriber, #2304) [Link]
Also, gnulib contains support code to make 'relocatable packages' work (by looking at argv[0], or, if this contains no path, by hunting along $PATH to find itself, then using relative paths everywhere in the knowledge of the location of the binary). GCC has worked this way for ever, but it's only fairly recently that relocatable support has started to find its way into other GNU programs. (Most non-GNU programs still don't care, but the GNU Project cares about keeping its stuff installable into people's home directories on random systems: that's how it started, after all.)
Installing to your home drive Posted Feb 23, 2011 19:08 UTC (Wed) by talex (subscriber, #19139) [Link] ZeroInstall I believe does source and binary (not sure how it manages binary) without root privileges. In my experience (0install developer), a surprising number of programs are relocatable:
Installing to your home drive Posted Feb 23, 2011 22:36 UTC (Wed) by nix (subscriber, #2304) [Link] Libraries always seem to be relocatable (not sure why this is; maybe to allow them to be bundled with other relocatable programs?).That is definitely not always true. The KDE3 libraries, for instance, were not relocatable: they had $datadir and $libdir/kde3 baked into them. (I think the same is true of glib and gtk as well.)
Optimizing Linux with cheap flash drives Posted Feb 25, 2011 19:22 UTC (Fri) by giraffedata (subscriber, #1954) [Link] How do you know you're caching these files? I don't know exactly what Linux's page replacement policy is this week, but I'd be surprised if it caches video and music files. It knows you're accessing these pages only once, so keeping them around in preference to something else would be a loss. It usually takes two accesses to a page to get it any significant priority for memory allocation. Linux also knows you're accessing the file sequentially, so it knows even sooner that it otherwise would that the pages won't be accessed a second time. Of course, if there's absolutely nothing else worth using memory for, Linux will just go ahead and fill it with this data, just in case. But that's not a problem.
|
|||||||||||||