User: Password:
|
|
Subscribe / Log in / New account

Temporary files: RAM or disk?

Temporary files: RAM or disk?

Posted Jun 1, 2012 5:34 UTC (Fri) by wahern (subscriber, #37304)
In reply to: Temporary files: RAM or disk? by hpa
Parent article: Temporary files: RAM or disk?

Some of us don't have swap. I've never needed it on any of my servers. If your resident set is too large for RAM, you'll have problems at the worst opportune time--under heavy load. If your virtual set is significantly larger than your resident set, then you have broken programs (probably descended from the mythical daemon which preallocates gigabytes of memory from malloc like candy, and necessitating the OOM killer). If a program wants to allocate a bunch of address space for object caching, or wants to slurp in a large data set into memory, there's mmap. Typical servers don't need swap; it's mostly brain dead desktop applications, and batch processing analytics software which copies huge datasets into malloc'd memory, and neither are heavy users of /tmp.

And I fail to see why tmpfs should be necessarily any better than a vanilla /tmp directory. Both primarily operate in RAM (tmpfs explicitly, /tmp through the buffer cache). Both spill to disk on memory pressure; probably the same disk, in fact. Instead of tmpfs, why not tweak the buffer cache? Are there any numbers comparing tmpfs with a /tmp on its own partition?


(Log in to post comments)

Temporary files: RAM or disk?

Posted Jun 1, 2012 23:46 UTC (Fri) by dlang (subscriber, #313) [Link]

it depends on if you allow overcommit.

If you disable overcommit and swap you run into fun where a large program that wants to execute a small one temporarily takes up twice it's large footprint as it forks to execute the small program.

Temporary files: RAM or disk?

Posted Jun 3, 2012 0:19 UTC (Sun) by giraffedata (subscriber, #1954) [Link]

The fact that some people don't have swap isn't a reason. They don't have swap because they don't need it. If tmpfs were otherwise the right choice, they could easily have swap space.

I would add swap space to a system even if it couldn't benefit from it for swapping, just to back tmpfs. It's a more efficient way of storing temporary, expendable files than any disk-based filesystem I know.

Temporary files: RAM or disk?

Posted Jun 3, 2012 0:29 UTC (Sun) by giraffedata (subscriber, #1954) [Link]

I'm struggling to understand your comments that typical servers do not need swap space and (I think) that systems which swap are broken.

The fundamental point of swap space, going back to its invention, is temporal locality of reference - the idea that in a given interval of time, certain memory is accessed far more frequently than other memory. Are you saying that isn't the case in typical servers? Or that it shouldn't be the case for typical servers? That typical servers should reference all memory uniformly over time and explicitly keep the less-accessed data in filesystems?

(The latter, BTW, was the technology that swapping replaced 40 years ago).

Temporary files: RAM or disk?

Posted Jun 3, 2012 1:55 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

The problem is, swap allows to allocate more RAM than present, using disk as a backing storage. Usually it works just fine because you don't need to touch all of your RAM at the same time.

However, there are several pathological cases that can arise. One depressingly common case: a fairly inactive application (say, a Java webapp) with a large working set is slowly pushed into swap by other apps. Since application is inactive it lives just fine until something triggers a garbage collection. And then JVM has to walk through all the pages, tracing object references and that causes all of the working set to be brought into RAM.

And while app is swapping in, the system might appear to be locked - I have no idea why, in theory other apps should remain responsive.

As a bonus, in this scenario the swapin of a Java application might cause swap out of another application which might be active at that time, causing problems with response time.

And as additional bonus, there's an even simpler scenario - an application which constantly allocates RAM (perhaps, in an infinite loop of malloc). It reliably kills my machine for several minutes if I use swap.

Temporary files: RAM or disk?

Posted Jun 4, 2012 13:09 UTC (Mon) by nix (subscriber, #2304) [Link]

And while app is swapping in, the system might appear to be locked - I have no idea why, in theory other apps should remain responsive.
Possibly part of the X server has been pushed out to swap, leaving it unable to dispatch events without swapping itself back in off the already-highly-contended disk.

Temporary files: RAM or disk?

Posted Jun 5, 2012 14:29 UTC (Tue) by pboddie (guest, #50784) [Link]

And let us not forget that valuable property of the OOM killer who usually makes an entrance at this point: to kill all the desired applications, leaving stupidapp to finally emerge triumphant, only to exclaim, "Where did everybody go?! System is going down for reboot? What does that mean?!"

Temporary files: RAM or disk?

Posted Jun 3, 2012 14:53 UTC (Sun) by bronson (subscriber, #4806) [Link]

Temporal locality of reference is the case for basically all programs (really oddball scientific ones being the only exception I can think of).

With most servers, though, it doesn't matter. The important bits (the server software) trivially fits into RAM, and the unimportant bits (the served content) comes off disk anyway. In this type of workload, swap is more of a liability than a benefit.

Another way of looking at it: best case it will swap a few unused nginx pages out to disk and bring 0.01% speed improvement, worst case it can fight the buffercache or even bring your server to its knees and light your pager up at 3am.

Temporary files: RAM or disk?

Posted Jun 3, 2012 17:28 UTC (Sun) by giraffedata (subscriber, #1954) [Link]

With most servers, though, it doesn't matter. The important bits (the server software) trivially fits into RAM, and the unimportant bits (the served content) comes off disk anyway.

That's way too general a statement to make. There is a great diversity of kinds of servers - different ages, scales, applications, etc., and few of us have a broad enough view of them to say anything is true about the majority of them.

Whether the important bits fit into RAM is the independent variable, not the dependent. The system designer chooses whether the important bits fit into RAM. So the only way the above makes sense is if you accept the oft-stated axiom that RAM is essentially free in 2012. I know there are servers where that is true, but there are plenty of servers where it's not. In one of those servers, if all the important bits fit into RAM, even though they're rarely accessed, that means someone screwed up and bought too much RAM.

The fact that you say everything but the server software is "served content," already tells me you're limiting your view to web servers and things like them. Other servers have very important data that is neither the server software nor originally from disk. If you don't buy either RAM or swap space for it, you don't serve.

Temporary files: RAM or disk?

Posted Jun 4, 2012 16:03 UTC (Mon) by bronson (subscriber, #4806) [Link]

Web servers, mail servers, file servers, directory servers, dns servers, Jabber servers, bittorrent servers, memcached, database servers, reverse proxy servers, load balancers, etc. etc. Almost all share a work profile that doesn't really benefit from swap.

You'll note that I did say *most*. Of course there exist servers that fall outside this but in my experience they're fairly rare.

So, I'm very curious, what is this somewhat common, swap-friendly type of server that I'm missing?

Temporary files: RAM or disk?

Posted Jun 4, 2012 21:17 UTC (Mon) by giraffedata (subscriber, #1954) [Link]

Web servers, mail servers, file servers, directory servers, dns servers, Jabber servers, bittorrent servers, memcached, database servers, reverse proxy servers, load balancers, etc. etc. Almost all share a work profile that doesn't really benefit from swap.

...

So, I'm very curious, what is this somewhat common, swap-friendly type of server that I'm missing?

You don't seem to be following the conversation. I said many servers have important data that is neither server software nor backed by a filesystem. I know I don't have to give you examples of those; many of your examples above use plenty of malloc memory. That was to cast doubt on the claim, which I said is way too general to make, that for most servers the important data is server software and other files.

It still might be true, but I'm a long way from being convinced any of us has a wide enough purview of the computer industry to know that the servers with working data are in the minority.

Temporary files: RAM or disk?

Posted Jun 5, 2012 5:59 UTC (Tue) by bronson (subscriber, #4806) [Link]

Obviously what I said was an oversimplification -- it's only 2 sentences. It still provides a decent mental model to answer the OP's question. If you'd like to improve on it, please do. There's plenty of room.

And, I have a reasonable view of the data centers that I've worked in... Most servers I've seen have avoided swapping. Some merely ignore it because it's redundant (Apache, nginx), and some go to unbelievable lengths to avoid it (Oracle). Very few actually embrace it (Varnish). That's just my experience. Again, if you've seen otherwise, please do share.

Temporary files: RAM or disk?

Posted Jun 5, 2012 1:31 UTC (Tue) by vonbrand (guest, #4458) [Link]

I believe you mean they require swap? At least a DNS resolver rapidly accumulates a huge database to keep in RAM, that is relatively rarely used.

Temporary files: RAM or disk?

Posted Jun 5, 2012 6:25 UTC (Tue) by bronson (subscriber, #4806) [Link]

Very true, there have been caching DNS servers that malloc everything and let the VM handle the disk. BIND gained a reputation for absolutely shredding swap space, especially if you were running more than one instance on a box. Now that BIND has its sharable databse plus hooks to use mysql/postgres/ldap/etc, I don't think it works like that anymore...? (I haven't used BIND in quite a while, hallelujah).

Lots of other DNS servers use their own databases and handle caching themselves (tinydns, powerdns, djb). maradns is the one exception I know of, but I don't think it has seen much adoption...?

You make an excellent point, this is a great example of swap usage ont he server. Nevertheless, I'm still under the impression that my "most servers don't want swap" statement still holds.


Copyright © 2017, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds