| |||||||||||||
Temporary files: RAM or disk?Temporary files: RAM or disk?Posted Jun 1, 2012 3:58 UTC (Fri) by hpa (subscriber, #48575)Parent article: Temporary files: RAM or disk?
Calling tmpfs RAM is very misleading. tmpfs is swappable, and in conjunction with a larger swap partition can handle large files just fine. It still performs quite a bit better for many workloads, simply because it has no integrity constraints: if the system crashes, it doesn't have to be able to present a consistent state at all.
On some of the (pre-disaster) kernel.org servers we switched to /tmp on tmpfs when we found that it made some of the data generation scripts that were run on a regular basis run as much as 20 times faster.
(Log in to post comments)
Temporary files: RAM or disk? Posted Jun 1, 2012 5:21 UTC (Fri) by tomba (subscriber, #51091) [Link]
I wonder if a tmpfs-like filesystem which stores small files into ram, like it does now, and large files to a normal on-disk fs, would work. In a sense tmpfs does that already by using swap, but I guess having a normal on-disk fs would be faster. The on-disk fs could be a simple one, as it doesn't need to be retained over boot.
Temporary files: RAM or disk? Posted Jun 1, 2012 12:13 UTC (Fri) by sorpigal (subscriber, #36106) [Link] As I read the article I began thinking the same thing. A new tmpfs-like FS that stores all files in RAM at first, stores them on disk if they grow large and also on disk if they remain for a long time. The big question is how you define the on disk location to be used for backing this new tmp; swap seems like a crude solution as (1) I don't use it now and (2) I would prefer to be able have /tmp grow as big as needed without causing the e.g. firefox to get OOMK'd.
Temporary files: RAM or disk? Posted Jun 3, 2012 6:36 UTC (Sun) by quotemstr (subscriber, #45331) [Link]
> A new tmpfs-like FS that stores all files in RAM at first, stores them on disk if they grow large and also on disk if they remain for a long time.
You are aware that Linux, like all modern operating system kernels, has a unified caching subsystem, right? The VM subsystem _already does_ exactly what you want: it will page out tmpfs pages to backing storage (in this case, swap) just as it would page out file-backed pages to their backing stores; the exact identity of the backing store doesn't make a difference. The distinction between bytes in memory and bytes on disk isn't nearly as clear-cut as you think.
Temporary files: RAM or disk? Posted Jun 4, 2012 5:08 UTC (Mon) by raven667 (subscriber, #5198) [Link]
One of the few reasonable statements in this discussion... Using the robust features of the kernel to implement the most sensible policy, who would have thought.
Temporary files: RAM or disk? Posted Jun 1, 2012 5:34 UTC (Fri) by wahern (subscriber, #37304) [Link]
Some of us don't have swap. I've never needed it on any of my servers. If your resident set is too large for RAM, you'll have problems at the worst opportune time--under heavy load. If your virtual set is significantly larger than your resident set, then you have broken programs (probably descended from the mythical daemon which preallocates gigabytes of memory from malloc like candy, and necessitating the OOM killer). If a program wants to allocate a bunch of address space for object caching, or wants to slurp in a large data set into memory, there's mmap. Typical servers don't need swap; it's mostly brain dead desktop applications, and batch processing analytics software which copies huge datasets into malloc'd memory, and neither are heavy users of /tmp.
And I fail to see why tmpfs should be necessarily any better than a vanilla /tmp directory. Both primarily operate in RAM (tmpfs explicitly, /tmp through the buffer cache). Both spill to disk on memory pressure; probably the same disk, in fact. Instead of tmpfs, why not tweak the buffer cache? Are there any numbers comparing tmpfs with a /tmp on its own partition?
Temporary files: RAM or disk? Posted Jun 1, 2012 23:46 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]
it depends on if you allow overcommit.
If you disable overcommit and swap you run into fun where a large program that wants to execute a small one temporarily takes up twice it's large footprint as it forks to execute the small program.
Temporary files: RAM or disk? Posted Jun 3, 2012 0:19 UTC (Sun) by giraffedata (subscriber, #1954) [Link] The fact that some people don't have swap isn't a reason. They don't have swap because they don't need it. If tmpfs were otherwise the right choice, they could easily have swap space. I would add swap space to a system even if it couldn't benefit from it for swapping, just to back tmpfs. It's a more efficient way of storing temporary, expendable files than any disk-based filesystem I know.
Temporary files: RAM or disk? Posted Jun 3, 2012 0:29 UTC (Sun) by giraffedata (subscriber, #1954) [Link] I'm struggling to understand your comments that typical servers do not need swap space and (I think) that systems which swap are broken.The fundamental point of swap space, going back to its invention, is temporal locality of reference - the idea that in a given interval of time, certain memory is accessed far more frequently than other memory. Are you saying that isn't the case in typical servers? Or that it shouldn't be the case for typical servers? That typical servers should reference all memory uniformly over time and explicitly keep the less-accessed data in filesystems? (The latter, BTW, was the technology that swapping replaced 40 years ago).
Temporary files: RAM or disk? Posted Jun 3, 2012 1:55 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]
The problem is, swap allows to allocate more RAM than present, using disk as a backing storage. Usually it works just fine because you don't need to touch all of your RAM at the same time.
However, there are several pathological cases that can arise. One depressingly common case: a fairly inactive application (say, a Java webapp) with a large working set is slowly pushed into swap by other apps. Since application is inactive it lives just fine until something triggers a garbage collection. And then JVM has to walk through all the pages, tracing object references and that causes all of the working set to be brought into RAM.
And while app is swapping in, the system might appear to be locked - I have no idea why, in theory other apps should remain responsive.
As a bonus, in this scenario the swapin of a Java application might cause swap out of another application which might be active at that time, causing problems with response time.
And as additional bonus, there's an even simpler scenario - an application which constantly allocates RAM (perhaps, in an infinite loop of malloc). It reliably kills my machine for several minutes if I use swap.
Temporary files: RAM or disk? Posted Jun 4, 2012 13:09 UTC (Mon) by nix (subscriber, #2304) [Link] And while app is swapping in, the system might appear to be locked - I have no idea why, in theory other apps should remain responsive.Possibly part of the X server has been pushed out to swap, leaving it unable to dispatch events without swapping itself back in off the already-highly-contended disk.
Temporary files: RAM or disk? Posted Jun 5, 2012 14:29 UTC (Tue) by pboddie (subscriber, #50784) [Link]
And let us not forget that valuable property of the OOM killer who usually makes an entrance at this point: to kill all the desired applications, leaving stupidapp to finally emerge triumphant, only to exclaim, "Where did everybody go?! System is going down for reboot? What does that mean?!"
Temporary files: RAM or disk? Posted Jun 3, 2012 14:53 UTC (Sun) by bronson (subscriber, #4806) [Link]
Temporal locality of reference is the case for basically all programs (really oddball scientific ones being the only exception I can think of).
With most servers, though, it doesn't matter. The important bits (the server software) trivially fits into RAM, and the unimportant bits (the served content) comes off disk anyway. In this type of workload, swap is more of a liability than a benefit.
Another way of looking at it: best case it will swap a few unused nginx pages out to disk and bring 0.01% speed improvement, worst case it can fight the buffercache or even bring your server to its knees and light your pager up at 3am.
Temporary files: RAM or disk? Posted Jun 3, 2012 17:28 UTC (Sun) by giraffedata (subscriber, #1954) [Link] With most servers, though, it doesn't matter. The important bits (the server software) trivially fits into RAM, and the unimportant bits (the served content) comes off disk anyway. That's way too general a statement to make. There is a great diversity of kinds of servers - different ages, scales, applications, etc., and few of us have a broad enough view of them to say anything is true about the majority of them. Whether the important bits fit into RAM is the independent variable, not the dependent. The system designer chooses whether the important bits fit into RAM. So the only way the above makes sense is if you accept the oft-stated axiom that RAM is essentially free in 2012. I know there are servers where that is true, but there are plenty of servers where it's not. In one of those servers, if all the important bits fit into RAM, even though they're rarely accessed, that means someone screwed up and bought too much RAM. The fact that you say everything but the server software is "served content," already tells me you're limiting your view to web servers and things like them. Other servers have very important data that is neither the server software nor originally from disk. If you don't buy either RAM or swap space for it, you don't serve.
Temporary files: RAM or disk? Posted Jun 4, 2012 16:03 UTC (Mon) by bronson (subscriber, #4806) [Link]
Web servers, mail servers, file servers, directory servers, dns servers, Jabber servers, bittorrent servers, memcached, database servers, reverse proxy servers, load balancers, etc. etc. Almost all share a work profile that doesn't really benefit from swap.
You'll note that I did say *most*. Of course there exist servers that fall outside this but in my experience they're fairly rare.
So, I'm very curious, what is this somewhat common, swap-friendly type of server that I'm missing?
Temporary files: RAM or disk? Posted Jun 4, 2012 21:17 UTC (Mon) by giraffedata (subscriber, #1954) [Link] Web servers, mail servers, file servers, directory servers, dns servers, Jabber servers, bittorrent servers, memcached, database servers, reverse proxy servers, load balancers, etc. etc. Almost all share a work profile that doesn't really benefit from swap. You don't seem to be following the conversation. I said many servers have important data that is neither server software nor backed by a filesystem. I know I don't have to give you examples of those; many of your examples above use plenty of malloc memory. That was to cast doubt on the claim, which I said is way too general to make, that for most servers the important data is server software and other files. It still might be true, but I'm a long way from being convinced any of us has a wide enough purview of the computer industry to know that the servers with working data are in the minority.
Temporary files: RAM or disk? Posted Jun 5, 2012 5:59 UTC (Tue) by bronson (subscriber, #4806) [Link]
Obviously what I said was an oversimplification -- it's only 2 sentences. It still provides a decent mental model to answer the OP's question. If you'd like to improve on it, please do. There's plenty of room.
And, I have a reasonable view of the data centers that I've worked in... Most servers I've seen have avoided swapping. Some merely ignore it because it's redundant (Apache, nginx), and some go to unbelievable lengths to avoid it (Oracle). Very few actually embrace it (Varnish). That's just my experience. Again, if you've seen otherwise, please do share.
Temporary files: RAM or disk? Posted Jun 5, 2012 1:31 UTC (Tue) by vonbrand (subscriber, #4458) [Link] I believe you mean they require swap? At least a DNS resolver rapidly accumulates a huge database to keep in RAM, that is relatively rarely used.
Temporary files: RAM or disk? Posted Jun 5, 2012 6:25 UTC (Tue) by bronson (subscriber, #4806) [Link]
Very true, there have been caching DNS servers that malloc everything and let the VM handle the disk. BIND gained a reputation for absolutely shredding swap space, especially if you were running more than one instance on a box. Now that BIND has its sharable databse plus hooks to use mysql/postgres/ldap/etc, I don't think it works like that anymore...? (I haven't used BIND in quite a while, hallelujah).
Lots of other DNS servers use their own databases and handle caching themselves (tinydns, powerdns, djb). maradns is the one exception I know of, but I don't think it has seen much adoption...?
You make an excellent point, this is a great example of swap usage ont he server. Nevertheless, I'm still under the impression that my "most servers don't want swap" statement still holds.
Temporary files: RAM or disk? Posted Jun 1, 2012 8:42 UTC (Fri) by iq-0 (subscriber, #36655) [Link]
The only thing missing is an option that you could claim a swap space as designated for only allowing tmpfs spillover.
Most of our servers have small amounts of swap and aggressive overcommit_memory settings. The small amount of swap makes is enough for the incidental extra memory needs and can be helpfull on loaded servers for accumulating effectively wasted memory.
Tmpfs really is the most sane option for /tmp. Why bother doing cleanups at boot? Why bother guaranteeing on disk-consistency or doing disk flushes on sync/fsync/fdatasync. The only sane alternative would be a ext2 filesystem with all data guarantees turned off and recreating it every time you boot.
Temporary files: RAM or disk? Posted Jun 9, 2012 16:36 UTC (Sat) by Serge (guest, #84957) [Link]
> The only thing missing is an option that you could claim a swap space as designated for only allowing tmpfs spillover.
You have this option — use a separate ext3 partition. :)
> we do often generate lots of temporary files (for sorting or graphing) and some might actually be rather large.
Why do you use tmpfs then? You can just use regular disk. Have you noticed some speedup because of using tmpfs?
> Tmpfs really is the most sane option for /tmp. Why bother doing cleanups at boot?
Why bother about tmpfs size? Why bother about adding more swap. Why bother about system slowed down because of heavy swap usage? On-disk /tmp don't have these problems. And it's cleaned on boot automatically anyway, no need to bother.
> Why bother guaranteeing on disk-consistency or doing disk flushes on sync/fsync/fdatasync.
Nobody does fsync in /tmp, so nobody bothers. :)
> The only sane alternative would be a ext2 filesystem with all data guarantees turned off and recreating it every time you boot.
That's why ext3 is better. Replaying ext3 journal is faster than creating a new filesystem.
Temporary files: RAM or disk? Posted Jun 14, 2012 9:05 UTC (Thu) by iq-0 (subscriber, #36655) [Link]
> You have this option — use a separate ext3 partition. :)
The problem with disk based /tmp is not that tmpfs or ext3 is faster perse. It's about when they want to do I/O:
And you can tune a lot about ext3, but at it's core it wants to make sure an always valid structure on disk exists. You can't make it ignore any sync requests, you can't tell it to not bother updating the free-space (you want it to be empty each time it's mounted, so why bother keeping track of which blocks are free?).
> Why do you use tmpfs then? You can just use regular disk. Have you noticed some speedup because of using tmpfs?
I don't know about you, but disk I/O is one of the biggest bottlenecks on our systems. And trying to prevent any for of I/O unless really necessary really helps overall system performance. So yes, we do.
> Nobody does fsync in /tmp, so nobody bothers. :)
Oh? Most software is oblivious to where they write or how their writes might affect performance. And often software is rightfully written to be "correct" (power fail safe) but are in some cases used in a different capacity than was originally imagined. And some tools call 'sync' (like "dpkg" and probably "rpm" too), which sync *all* filesystems mounted.
> That's why ext3 is better. Replaying ext3 journal is faster than creating a new filesystem.
You know what is faster than replaying a journal and than performing (effectively) 'rm -rf' on it? Never creating a filesystem in the first place. Swap space never has to be recreated it's assumed to be unused on fresh boot.
Temporary files: RAM or disk? Posted Jun 1, 2012 11:35 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]
I don't have swap. I don't WANT swap because its implementation is a pile of excrement.
In case of problems it reliably slows my computer to a crawl so it's often easier just to reboot the computer than to wait several minutes to type htop and kill the offending process. I very much prefer OOM killer to come out quickly and kill something - that usually leaves me with a working system, at least.
Again, using swap to extend the size of a _filesystem_ seems a bit convoluted.
Temporary files: RAM or disk? Posted Jun 1, 2012 17:35 UTC (Fri) by gmaxwell (subscriber, #30048) [Link]
I haven't shared your negative expediences with swap— at least not at any time during the past 5 years or so. I run all my systems— servers, desktops, and laptops alike with /tmp on tmpfs, with large swaps. I frequently have tens of gigs of data in tmpfs on workhorse machines.
On my laptop, at least before I used a SSD, tmpfs made a visible increase in battery life because the drive was no longer being woken up by tmp activity.
Temporary files: RAM or disk? Posted Jun 1, 2012 18:12 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]
I've used laptop-mode on my laptop for ages. So I wasn't bother by disk IO from /tmp either.
|
|||||||||||||