Storing Files/Directories In Memory With tmpfs [LWN.net]

HowtoForge takes a look at storing files in memory, instead of on a hard drive. "You probably know that reading from RAM is a lot of faster than reading files from the hard drive, and reduces your disk I/O. This article shows how you can store files and directories in memory instead of on the hard drive with the help of tmpfs (a file system for creating memory devices). This is ideal for file caches and other temporary data (such as PHP's session files if you are using session.save_handler = files) because the data is lost when you power down or reboot the system."

to post comments

Storing Files/Directories In Memory With tmpfs

Posted Dec 9, 2008 20:51 UTC (Tue) by BrucePerens (guest, #2510) [Link] (21 responses)

Your filesystem block cache is RAM, whatever filesystem you're using. Tmpfs files are going to end up in the swap space anyway - on the disk, if you don't reference them frequently, or they'll be using RAM you need elsewhere.

I'm dubious that tmpfs helps, but I'd be willing to look at benchmarks. Does anyone have some?

Storing Files/Directories In Memory With tmpfs

Posted Dec 9, 2008 21:50 UTC (Tue) by johnkarp (guest, #39285) [Link] (1 responses)

Any gains would probably be for writes. In a worst case scenario, you'll just be avoiding journalling overhead and possibly atime updates. In a best case scenario, you'll be cutting out all disk I/O completely.

Storing Files/Directories In Memory With tmpfs

Posted Dec 9, 2008 22:11 UTC (Tue) by quotemstr (subscriber, #45331) [Link]

tmpfs is also useful for forcing fsync() to be a no-op for overparanoid applications.

Storing Files/Directories In Memory With tmpfs

Posted Dec 9, 2008 22:35 UTC (Tue) by mikachu (guest, #5333) [Link] (6 responses)

using tmpfs is a _lot_ faster if you're writing a lot of small files, like say compiling code, since you don't need to write out inodes and all that, which most filesystems write out every 5 seconds or so. Checking out linux-2.6 from git (with the repo local) takes about 4 minutes on xfs and 4 seconds on tmpfs here.

Storing Files/Directories In Memory With tmpfs

Posted Dec 9, 2008 22:58 UTC (Tue) by BrucePerens (guest, #2510) [Link] (4 responses)

That's darned awful. But couldn't we say it's an XFS problem? XFS isn't handling transient files well. I wonder if it's as bad with other filsystems?

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 1:03 UTC (Wed) by sbergman27 (guest, #10767) [Link] (3 responses)

Well, it depends upon what you call "transient". Transient might be "created and then deleted 3 seconds later" or, in the case of the source code checkout example, "created and then deleted 30 minutes later". XFS might do quite well in the first case because the data may never have to actually be written out to disk. (I don't know for sure, but I suspect it would do pretty well.) But no "real" filesystem is going to do as well as tmpfs in the latter case, because "real" filesystems have to worry about keeping data safe. Which means syncing to disk with reasonable frequency. Actually, I suppose the best of both worlds would be to take a fast and light weight FS like ext2, and add a mount option to defer any syncs to disk unless memory pressure forces them. In my experience, access to data in the filesystem is far faster than access to pages in swap. So I suspect that when available memory becomes scarce, that solution would be better than letting tmpfs blocks get written to swap... and then read back out again.

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 15:20 UTC (Wed) by andrel (guest, #5166) [Link] (2 responses)

That's been my experience as well, and it puzzles me--why is accessing data on disk by way of the filesystem so much faster than accessing data on disk by way of swap?

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 15:37 UTC (Wed) by sbergman27 (guest, #10767) [Link]

Well, if you use sysstat to record the paging activity, it becomes apparent that when the swapping is bad enough to slow the system down, the actual number of MB/s being transferred to and from the swap is not very high in comparison to how fast those drives can stream serial reads and writes. And yet the drives seem quite busy. I'd say that swap access involves lots of random seeks.

Storing Files/Directories In Memory With tmpfs

Posted Dec 11, 2008 19:14 UTC (Thu) by lysse (guest, #3190) [Link]

That reminded me of something Butler Lampson wrote 25 years ago:

> The Alto operating system has an ordinary read/write-n-bytes interface to files, and was extended for Interlisp-D with an ordinary paging system that stores each virtual page on a dedicated disk page. Both have small implementations (about 900 lines of code for files, 500 for paging) and are fast (a page fault takes one disk access and has a constant computing cost that is a small fraction of the disk access time, and the client can fairly easily run the disk at full speed). The Pilot system which succeeded the Alto OS follows Multics and several other systems in allowing virtual pages to be mapped to file pages, thus subsuming file input/output within the virtual memory system. The implementation is much larger (about 11,000 lines of code) and slower (it often incurs two disk accesses to handle a page fault and cannot run the disk at full speed). The extra functionality is bought at a high price.

("Hints for computer system design." ACM Operating Systems Rev. 17.5 pp 33-48)

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 12:10 UTC (Wed) by adobriyan (subscriber, #30858) [Link]

> using tmpfs is a _lot_ faster if you're writing a lot of small files, like
> say compiling code, since you don't need to write out inodes and all that,
> which most filesystems write out every 5 seconds or so.

Here for CPU-bound compile jobs, difference between ext2 and tmpfs is
within statistical error.

The greatest slowdown in this case comes probably from compilation from
cold-cache source tree with small-enough parallelism (-j1 or -j2).

> Checking out linux-2.6 from git (with the repo local) takes about
> 4 minutes on xfs and 4 seconds on tmpfs here.

What exactly you did and in which order?

Storing Files/Directories In Memory With tmpfs

Posted Dec 9, 2008 22:48 UTC (Tue) by jengelh (subscriber, #33263) [Link] (9 responses)

>Your filesystem block cache is RAM, whatever filesystem you're using. Tmpfs files are going to end up in the swap space anyway

So use ramfs then, which is the unswappable variant.

Storing Files/Directories In Memory With tmpfs

Posted Dec 9, 2008 23:01 UTC (Tue) by BrucePerens (guest, #2510) [Link] (3 responses)

So use ramfs then, which is the unswappable variant.

No, thank you. I have better uses for that RAM. The point here is whether tmpfs gets you any advantage. What I am seeing so far is that it might be a band-aid for a bad filesystem property - that some of our filesystems really stink at handling transient files. See the comment about XFS above.

Storing Files/Directories In Memory With tmpfs

Posted Dec 9, 2008 23:17 UTC (Tue) by jengelh (subscriber, #33263) [Link]

All filesystems probably suck in that regard. (And especially since XFS is said to cache things rather aggressively.)

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 3:09 UTC (Wed) by rich0 (guest, #55509) [Link]

I'm not convinced that this is a problem with filesystems, but rather a
fundamental assumption.

Filesystems contain an underlying assumption that you care about the data
written to them - that you'd like to get it back some day. Data isn't
considered "safe" until written to storage (and journaled/etc). Almost
all filesystems flush write cache within a few seconds as a result.

A file created in tmpfs in the worst case gets swapped out, which is no
different from a filesystem in terms of performance. However, in the best
case it never gets written to disk at all. Also, the kernel can be more
liberal with swap than flushing write caches.

tmpfs should outperform a disk-based filesystem for temporary files in the
same way that udp outperforms tcp for streaming video. Why would you want
your bandwidth for the video showing NOW clogged up with retransmitted
packets for video that you already missed? There is nothing wrong with
tcp - it is just a matter of it having a different set of assumptions.

As far as "fixing" xfs/etc goes - please don't. I don't want to lose 30
minutes worth of work when my power supply fails or whatever... :)

Oh, for an example of real-world benefits - just talk to any gentoo user
(or anybody else who does a lot of compiling). A typical build on gentoo
goes many times faster on tmpfs than on a "real" filesystem. A build
might generate thousands of 10-100kb files in a minute, copy a handful to
disk, and then delete the whole directory tree it just created.

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 13:56 UTC (Wed) by elanthis (guest, #6227) [Link]

Meh. 4GB of RAM goes for $45 on Newegg right now. Not one of my desktops or servers ever seems to use more than about 50% of the RAM I put it in; usually 4GB for a desktop and 8GB for a server.

Whether or not tmpfs is a hack to work around poor disk-based filesystem storage, the simple truth is that right now it's a sometimes _useful_ hack sometimes, and those of us that have RAM to spare and could use the boost appreciate its availability.

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 9:19 UTC (Wed) by k3ninho (subscriber, #50375) [Link] (3 responses)

There's a middle ground between the two extremes. A tmpfs may swap, but its memory may be reused without a reboot. RAMFS won't swap, but once allocated in the kernel, you won't get the memory back without a reboot. Can't there be a mount option for the tmpfs which makes it unswappable?

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 10:24 UTC (Wed) by jengelh (subscriber, #33263) [Link] (2 responses)

Hm. Since rootfs is said to be also based on ramfs, and early userspace such as the launchers inside an initramfs images are advised to run `find / -xdev -print0 | xargs -0 rm -f` (or appropriate rm -Rf calls) before chrooting to the on-disk system, should not clearing the files just do the trick?

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 15:20 UTC (Wed) by k3ninho (subscriber, #50375) [Link] (1 responses)

It seems that the stuff I'd read on the web about general uses of ramfs vs tmpfs indicated that memory allocated to the RAM disk couldn't be freed. Googling just now changes that. I don't know whether to trust Documentation/ramdisk.txt because of its age, although it may not have changed because the Kernel's view on RAM disks hasn't.

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 15:22 UTC (Wed) by jengelh (subscriber, #33263) [Link]

ramdisk != ramfs

And yes, you can also free the blocks used by a ramdisk, using `blockdev --flushbufs /dev/rd0`.

Storing Files/Directories In Memory With tmpfs

Posted Dec 11, 2008 19:17 UTC (Thu) by lysse (guest, #3190) [Link]

My God, it's as though we're back in the land of DOS... :)

Utilization Dynamics

Posted Dec 10, 2008 4:55 UTC (Wed) by AnswerGuy (guest, #1256) [Link]

It's true that tmpfs may get pushed out to swap to relieve other memory pressure. This is absolutely the desired behavior.

The benefit of using tmpfs is for cases where one expects to usually be operating with a surplus of RAM ... so the temporary files on those mount points will *usually* not generate any disk/network I/O. Sometimes they may ... but the idea is to get typical case benefits with reasonable worst case handling.

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 22:45 UTC (Wed) by hmh (subscriber, #3838) [Link]

Yes, I got an improvement on amavisd-new work directories with tmpfs, it was consistently faster than xfs and ext3 (sorry, I didn't keep the test run data).

This is a workload where you create many files (write sequencially once each file), scan them over a small number of times, then rm -rf the whole stuff, and start all over. This is done concurrently by many threads (we used 32 amavisd-new threads in the test, because that's what we use in the production machines).

My guess is that tmpfs is much faster for metadata handling (it doesn't have to do much of it at all!), especially with multiple writers and lots of file creation and unlinking going around.

I didn't think of trying ext2, though.

Maybe a suitably prepared bonie++ run comparing xfs, ext3, ext2, tmpfs and whatever other fs one might care for (using the most data-can-go-to-hell modes of they allow for, such as barriers off, no data ordering nor data journaling) could give us better numbers. One might have to hack bonie++ to make sure it never syncs or fsyncs.

Storing Files/Directories In Memory With tmpfs

Posted Dec 9, 2008 23:27 UTC (Tue) by lmb (subscriber, #39048) [Link]

Why filesystems suck? Because they honor fsync() et al, which means "data must hit media" and thus can't remain in-cache; not to mention seek times and so on. This should come as no surprise.

In the long-term, tmpfs++ (or probably one of its successors) will be an interesting option for non-volatile memory.

One use case for tmpfs

Posted Dec 10, 2008 0:04 UTC (Wed) by NightMonkey (subscriber, #23051) [Link] (5 responses)

Nice article, but it only hints at the power that using a ram disk can offer. A web company with tons of traffic I used to work for did the following magic with tmpfs:

* PXEbooted kernel and initrd on diskless webservers, then untarred the / filesystem onto tmpfs, and pivotroot into the result. Mounted log partitions over NFS (which was the biggest headache of this comapny's scenario). Power on to service time: about 45-120 seconds (depending on competing network traffic).
* For DB servers with replaceable data, had MySQL DBs (yeah, I know) on tmpfs (wow, it really screams).
* No swap at all on these systems, so there were several more ways a box could die, but... just reboot on panic, and have a remote power box to flip the power if needed. The system comes back so fast that the reboot ceases to matter in terms of overall service availability.
* Much easier to just reboot your way out of problems, and upgrades (to kernel, software, new NFS or local disk mountpoints, etc.) were centralized and could be performed on blocks of servers at once. With PXE, dhcpd and the NIC MAC addresses, custom kernels and userland could be tailored to hardware and functional purpose.

The boxes were fit with dual quad-cores and as much RAM as they could handle.

Now, this scenario would NOT be good if any data was not replaceable, and they had disk-based systems for that (actually, those were systems that booted into the same type of environment, but with data-only physical disks, with the exception of the PXEboot server, which, obviously, needs to boot from disk). Also, any log files were sent over NFS mounts to NAS systems.

It was, all in all, a pretty neat setup. I had run systems in the past where a set of binaries (like, say, Apache and its subsystems) was run over a read-only NFS mount, and another read-only mount with the docroot, and a local disk for the OS, but this opened my eyes to the power of this almost all-RAM setup to scale up systems quickly, and centralize configurations. No need for Cfengine or Puppet, or worrying about disk failures (except on the central, critical systems, and the NASes). One of the downsides of this was that the strategy could mask some performance issues, and even some gross misconfigurations, since plodding disk latency was gone from the equation.

What I'm looking for in the future is a version of tmpfs that can be resized downward as well as up. I'm not sure if newer kernels allow this.

Cheers!

One use case for tmpfs

Posted Dec 10, 2008 2:42 UTC (Wed) by k8to (guest, #15413) [Link]

In practice I haven't found that tmpfs beats ext3 by much in any real scenarios. But it is more legwork to set up, and you do have to be sure the stuff you're storing there will never be useful after a crash, and ...

Been there seen it, not worth the effort.

Another use case for tmpfs

Posted Dec 10, 2008 4:26 UTC (Wed) by mcisely (guest, #2860) [Link] (2 responses)

How about diskless mythtv front end machines.

I use a read-only NFS root FS mount for most of the files - since it's read only, this allows the same file system to be shared by multiple front ends. And it's read only because the parts that must still be writable get shunted into a local tmpfs mount (mostly) via symbolic links... I've been running this way for almost a year now. Though tmpfs plays only a small part in this picture, it is a critical part:

http://www.isely.net/mythtv_setup.html

Another use case for tmpfs

Posted Dec 10, 2008 16:38 UTC (Wed) by jengelh (subscriber, #33263) [Link] (1 responses)

>And it's read only because the parts that must still be writable get shunted into a local tmpfs mount (mostly) via symbolic links...

Why bother with getting symlinks right if you can just use something like unionfs.

But of course, using a real NFS over untarring the base system into tmpfs means you have (much) more free space for COW files.

Another use case for tmpfs

Posted Dec 10, 2008 19:59 UTC (Wed) by mcisely (guest, #2860) [Link]

When I started with the sym-link approach I was also concerned that it would be messy and unmaintainable. But it took me roughly an evening to get it set up correctly, and I haven't had to tweak anything since then. Also, even if I had used unionfs there were still a few cases where symbolic links would have been needed for host-specific stuff that I couldn't set up via a DHCP attribute (maybe 2 cases).

I did however look at unionfs, and the deal-breaker in the end was that it wasn't in mainline and I didn't want to get into the business of having to patch it in each time.

I also looked at just downloading and untar'ing the whole FS into tmpfs, but that does chew up a lot of RAM and the massive download lengthens the boot time. The vast majority of the file system isn't touched during most operations, so it made natural sense just to use NFS instead to load on demand whatever is needed.

-Mike

One use case for tmpfs

Posted Dec 10, 2008 7:55 UTC (Wed) by NAR (subscriber, #1313) [Link]

I've seen similar setup in telephony exchange server (not in the part that does the accounting :-). It also didn't have data to keep, the binaries were read from NFS to tmpfs at boot, logging were done via NFS.

Storing Files/Directories In Memory With tmpfs

Posted Dec 10, 2008 8:40 UTC (Wed) by Marco (guest, #50397) [Link]

Have you ever heard about PRAMFS? Try look at it....

Execute In Place (XIP) with tmpfs

Posted Dec 10, 2008 9:35 UTC (Wed) by meuh (guest, #22042) [Link] (4 responses)

Running programs from a mounted tmpfs filesystem provide some kind of Execute In Place (XIP) feature. It's better not to have the program's segments twice in memory.
See https://kerneltrap.org/mailarchive/linux-kernel/2007/5/2/...

Execute In Place (XIP) with tmpfs

Posted Dec 10, 2008 10:56 UTC (Wed) by ebiederm (subscriber, #35028) [Link] (2 responses)

All linux filesystems that use the page cache have this property.
ext[234],reiserfs,xfs,etc

With the linux page cache the data is stored in the format applications
are going to use it, and is directly mappable to user space. Where they differ the linux page cache caches the file contents not the contents disk blocks.

Execute In Place (XIP) with tmpfs

Posted Dec 10, 2008 17:25 UTC (Wed) by drag (guest, #31333) [Link] (1 responses)

Well my understanding is that TMPFS is actually just page cache.

It's using the same code that provides the cache for file systems in Linux. So effectively TMPFS is disk cache without the disk.

This is a improvement over previous ram-based drives because you'd store the files in the ramddrive then they were cache'd again by the file system. Now with tmpfs being it's own file system it's all just cache.

So fundamentally once you get files out of cache on Ext3 or whatever you are actually using tmpfs, or at least the same concepts.

Execute In Place (XIP) with tmpfs

Posted Dec 10, 2008 21:44 UTC (Wed) by nix (subscriber, #2304) [Link]

tmpfs is swap-backed page cache. Normal filesystems are
block-device-backed page cache with a more complex mapping.

Execute In Place (XIP) with tmpfs

Posted May 25, 2016 1:19 UTC (Wed) by xiaokaoy (guest, #108949) [Link]

Sounds great. But in order to be able to run a program from a mounted tmpfs filesystem without having to load it into a separate RAM region, the executable file must already be page-aligned in tmpfs, I think. What guarantees this when copying an executable into tmpfs?

Storing Files/Directories In Memory With tmpfs

Posted Dec 12, 2008 18:57 UTC (Fri) by jrigg (guest, #30848) [Link]

Real time audio apps like JACK use tmpfs to reduce the frequency of non-audio related writes to disk on journalling file systems. If you're trying to record lots of channels of high resolution audio this makes the difference between unusable and usable. I suspect this would apply to video editing systems too.