Would it work to write such temp files to a tmpfs [LWN.net]

Would it work to write such temp files to a tmpfs

Posted Mar 11, 2014 15:01 UTC (Tue) by Wol (subscriber, #4433)
In reply to: A discussion between database and kernel developers by mel
Parent article: A discussion between database and kernel developers

But this is a fairly easy thing to solve today, if you can direct your temporary files.

For example, I run gentoo. With 16Gb ram, I have (iirc) two 32Gb swap partitions (overkill, I know :-). I also have /var/tmp/portage mounted as a 20Gb tmpfs. I doubt I ever spill into swap while "emerge"ing an update, but I can leave it all to the OS to handle without worrying about it.

With PostgreSQL, all a user would need to do would be to add more ram, and the same thing would apply - if it fits in ram it stays in ram, if it doesn't then swap gets involved - but it would have to get involved anyway under any relevant scenario.

Cheers,
Wol

Would it work to write such temp files to a tmpfs

Posted Mar 11, 2014 17:32 UTC (Tue) by dlang (guest, #313) [Link] (7 responses)

don't forget that paging in and out of swap tends to be significantly slower than to a simple disk file because the swap tends to be badly fragmented.

Would it work to write such temp files to a tmpfs

Posted Mar 11, 2014 17:46 UTC (Tue) by Wol (subscriber, #4433) [Link] (6 responses)

If you're going to use a lot of ram like this, just have several (hopefully very fast) disks and spread swap across them at equal priority.

Or if necessary (expensive but maybe worth it) just have swap on an ssd.

And tell the kernel developers you don't want the swap partition to fragment! :-)

Cheers,
Wol

Would it work to write such temp files to a tmpfs

Posted Mar 11, 2014 18:23 UTC (Tue) by dlang (guest, #313) [Link]

avoiding swap file fragementation requires the kernel either know ahead of time what's going to be swapped out in the future, or requiring that swap space be larger than virtual memory size so that there is a reserved spot for any page and it can be read in efficiently.

requiring a bunch of really fast disks to use swap when a medium speed disk doing a sequential write/read of the data will be just as fast is not a smart way to spend your money.

Would it work to write such temp files to a tmpfs

Posted Mar 12, 2014 16:00 UTC (Wed) by jeremiah (subscriber, #1221) [Link] (4 responses)

FWIW, I've found using an SSD for swap a great way to massively speed things up until the wear leveling really has to kick in. At which point in time everything goes downhill real fast. They seem to be okay when you have a little memory pressure, but when you hit a hard swap storm over a long period of time the drives have a hard time dealing with all of the overwriting of the same data repeatedly. This of course is all anecdotal and may have been caused by a particular version of Crucial/Micron's firmware in their m4 series, but I had ditch my swaps on those drives for fear of losing the whole drive.

Would it work to write such temp files to a tmpfs

Posted Mar 12, 2014 16:10 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Just do some TRIM-ming of empty space on these drives from time to time. Works wonders.

Would it work to write such temp files to a tmpfs

Posted Mar 12, 2014 20:44 UTC (Wed) by jeremiah (subscriber, #1221) [Link]

I'll give it a shot, thanks

Would it work to write such temp files to a tmpfs

Posted Mar 12, 2014 19:05 UTC (Wed) by parcs (guest, #71985) [Link] (1 responses)

Have you tried enabling the "discard" mount option on the swap partition (or using swapon -d)?

Would it work to write such temp files to a tmpfs

Posted Mar 12, 2014 20:44 UTC (Wed) by jeremiah (subscriber, #1221) [Link]

I have not, thanks.