Debian's /tmpest in a teapot
Debian had a major discussion
about mounting /tmp as a RAM-based tmpfs in 2012 but inertia
won out in the end. Debian systems have continued to
store temporary files on disk by default. Until now. A mere
12 years later, the project will be switching to a RAM-based /tmp in the Debian
13 ("Trixie") release. Additionally, starting with Trixie, the
default will be to periodically clean up temporary files automatically in
Join the party
By now, using tmpfs for the /tmp directory is a road well-traveled. Many Linux distributions have already switched to mounting /tmp as a tmpfs. Arch Linux, Fedora, openSUSE Tumbleweed, and many others have all made the switch. Red Hat Enterprise Linux (RHEL) and its clones, as well as SUSE Linux Enterprise Server (SLES) and openSUSE Leap, still default to /tmp on disk. Ubuntu, following Debian, also continues to have a disk-based /tmp rather than using tmpfs.
The knobs to control how /tmp is mounted, and the handling of temporary files, are part of systemd. (On systems where systemd is used, of course.) The upstream defaults for systemd are to mount /tmp as a tmpfs and delete files that have not been read or changed after ten days in /tmp and 30 days for those stored in /var/tmp. Debian Developer and systemd contributor Luca Boccassi recently decided it was time to revisit the topic of /tmp as a tmpfs, and start deleting temporary files in /tmp and /var/tmp.
On May 6 Boccassi resurrected a discussion from Debian's bug tracker that started in 2020 and that had trailed off in July 2022 without resolution. The bug was submitted by Eric Desrochers, who complained that Debian's systemd implementation did not clean /var/tmp by default. Michael Biebl wrote that this was a deliberate choice to match Debian's pre-systemd behavior. After a long period of inactivity, Biebl suggested in October 2021 (and again in July 2022) that Desrochers raise the topic on the Debian devel mailing list. That never happened.
In reviving the topic, Boccassi declared that it was time to bring Debian's defaults in line with upstream and other Linux distributions by making /tmp a tmpfs, and cleaning up /tmp and /var/tmp on a timer. He planned to apply those changes in the next systemd upload to Debian unstable within a week, with instructions in the NEWS file on how to override the changes for any users who wanted to keep the current behavior. That, in turn, sparked quite a discussion.
The case for and against
Biebl worried
about the effect of cleaning up temporary files by default. Currently,
he said, Debian only cleans /tmp on boot (which is
unnecessary if /tmp is a RAM-based tmpfs) and to never clean
up files in /var/tmp. Boccassi answered:
"Defaults are defaults, they are trivially and fully overridable
where needed if needed.
" Biebl replied
that there is value in aligning defaults across Linux distributions,
but argued that Debian has "a larger scope
" than a desktop
distribution like Fedora. He suggested that it was necessary to "gather feedback from all affected parties
to make an informed decision
". Boccassi responded
that it was impossible to have defaults that would make everyone
happy, and he was not looking for hypothetical arguments or
philosophical discussions, he wanted facts: "what would
break where, and how to fix it?
"
Sam Hartman noted that ssh-agent created its socket under /tmp, but it would be better if it respected the $XDG_RUNTIME_DIR setting and created its socket under /run/user. Boccassi agreed and said that he had filed a bug for ssh-agent. Richard Lewis pointed out that tmux stores its sockets in /tmp/tmux-$UID, and deleting those files might mean users could not reattach to a tmux session that had been idle a long time. Boccassi suggested that using flock() would be the right solution to stop the deletion, and said he had filed a bug on that as well.
Lewis questioned
whether files downloaded with apt source might be
considered "old" when they were extracted under /tmp and
immediately flagged to be deleted. Russ Allbery said
that systemd-tmpfiles would respect the access timestamp (atime)
and changed timestamp (ctime), not just the modified timestamp
(mtime), "so I think this would only be a problem on file systems that didn't support
those attributes
". Allbery said he believed tmpfs supports all three.
Some users store information in /var/tmp for long-running
jobs, because they want it preserved without being backed up, said
Barak A. Pearlmutter. Boccassi dismissed
that, saying that those users "assuming they actually exist
",
can customize the system defaults according to their needs.
Those users do exist, wrote
Jonathan Dowland, "[I've] been one of them, I've worked with many of
them, it's an incredibly common pattern in academic
computing
". Making a change to how these files are handled,
"should be a very carefully explored decision
". He also
asked for arguments in favor of the change, and to hold off the change
to allow the discussion to continue.
Hartman wished that it was possible to specify that directories under /var/tmp should be deleted entirely or left alone. Boccassi said that was a reasonable enhancement request, and that it had already been filed upstream for systemd.
Allbery eventually admitted
surprise at the number of people who reported using
/var/tmp "for things that are clearly not temporary
files in the traditional UNIX sense
". Periodically deleting files
in /var/tmp, he said, has been common UNIX practice for at
least 30 years:
Whatever we do with /var/tmp retention, I beg people to stop using /var/tmp for data you're keeping for longer than a few days and care about losing. That's not what it's for, and you *will* be bitten by this someday, somewhere, because even with existing Debian configuration many people run tmpreaper or similar programs. If you are running a long-running task that produces data that you care about, make a directory for it to use, whether in your home directory, /opt, /srv, whatever.
Hakan Bayındır said
that he did not use /var/tmp, but applications that
other people use do. He cited a high-end scientific application that was
not under his or other users' control. Allbery argued that this
was "not a good design decision
" but conceded that may not be
helpful if a user or admin does not have control over the
application's design. However, he said, it was "playing with fire
"
to use /var/tmp that way "and it's
going to result in someone getting their data deleted at some point,
regardless of what Debian does
".
Lewis said
he still did not understand the rationale for the change. Was there,
he asked, "perhaps something more compelling than 'other
distributions and upstream already do this'?
" That sounded like
the original rationale, Allbery
said, but he added that moving /tmp to tmpfs should make
applications run faster, and reaping files under /var/tmp
could help to avoid filling up a partition. Allowing the partition to
fill up, he noted, could
cause bounced mail, unstable services, and other problems. It may not
be a problem for desktop systems, which tend to have enough disk space
not to be affected, but for virtual machines that have /var/tmp
contained within a small root partition, it is still a concern.
Rolling out changes
In the end, none of the arguments for maintaining Debian's status quo managed to persuade Boccassi to stay his hand. On May 28, Boccassi announced the changes that he had made and uploaded to unstable. As expected, /tmp will become a tmpfs by default, for both new and existing installations of Debian unstable. New installs will use the systemd default behavior. The openssh and tmux packages, he said, had been fixed to provide exceptions to retain their temporary files. A description has been added to the Debian installer to inform users that /tmp is a tmpfs by default, and the changes have been noted in the NEWS file. He also offered to review and merge any changes to the Debian installer that would let users customize those options during installation. Users who want /tmp to remain on disk can override the upstream default with systemctl mask tmp.mount. To stop periodic cleanups of /tmp and /var/tmp users can run touch /etc/tmpfiles.d/tmp.conf.
None of the changes, it should be noted, will affect Debian versions prior to Trixie. Users of Debian stable and oldstable will only encounter these changes on upgrade.
As noted, many distributions have already made these changes without catastrophe. Debian, and its users, should be able to adapt to the new defaults or override them if they are unsuitable for the workloads to be run. At worst, it promises to be a temporary inconvenience.
Posted Jun 3, 2024 16:31 UTC (Mon)
by iustin (subscriber, #102433)
[Link] (2 responses)
Posted Jun 3, 2024 16:38 UTC (Mon)
by bluca (subscriber, #118303)
[Link] (1 responses)
Posted Jun 3, 2024 16:40 UTC (Mon)
by iustin (subscriber, #102433)
[Link]
Posted Jun 3, 2024 17:25 UTC (Mon)
by mb (subscriber, #50428)
[Link]
If cleaning is enabled, it should have a size threshold to not clean files that are less than a couple of kiB in size. Removing these files doesn't really help anybody in todays world with dozens of gigabytes of memory.
Posted Jun 3, 2024 17:46 UTC (Mon)
by ju3Ceemi (subscriber, #102464)
[Link] (38 responses)
One usage I've met many times is using /tmp to store temporary files that are too big to be kept in memory : rootfs are likely bigger than 10GB, while memory is more scarse (and while we can keep such files on disk for some time, it is expensive to use memory than way)
Nothing too difficult to handle, I think those users will be able to deal with the change accordingly
Posted Jun 3, 2024 18:09 UTC (Mon)
by smurf (subscriber, #17840)
[Link] (33 responses)
Presumably swapping has a bit more overhead than writing directly to disk, but that should be more than offset by all the small files you no longer write to disk.
Posted Jun 3, 2024 18:11 UTC (Mon)
by ju3Ceemi (subscriber, #102464)
[Link] (24 responses)
Posted Jun 3, 2024 18:32 UTC (Mon)
by atnot (subscriber, #124910)
[Link] (2 responses)
Posted Jun 6, 2024 7:45 UTC (Thu)
by taladar (subscriber, #68407)
[Link] (1 responses)
Posted Jul 17, 2024 0:54 UTC (Wed)
by immibis (guest, #105511)
[Link]
Posted Jun 3, 2024 18:33 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Posted Jun 4, 2024 15:01 UTC (Tue)
by gray_-_wolf (subscriber, #131074)
[Link]
Posted Jun 5, 2024 17:32 UTC (Wed)
by piexil (guest, #145099)
[Link]
Posted Jun 3, 2024 18:37 UTC (Mon)
by Heretic_Blacksheep (guest, #169992)
[Link] (1 responses)
I can't remember if the Linux kernel will dump its core to /var/crash or if it uses the swap space area for that by default. It's been so long since I saw the kernel completely crash I've forgotten where it puts things that went Boom.
Posted Jun 3, 2024 18:41 UTC (Mon)
by ju3Ceemi (subscriber, #102464)
[Link]
Posted Jun 3, 2024 18:57 UTC (Mon)
by mezcalero (subscriber, #45103)
[Link] (7 responses)
Hence, add swap. It makes things faster in most cases.
Posted Jun 3, 2024 23:02 UTC (Mon)
by josh (subscriber, #17465)
[Link] (6 responses)
Posted Jun 3, 2024 23:52 UTC (Mon)
by atnot (subscriber, #124910)
[Link] (2 responses)
Posted Jun 4, 2024 0:02 UTC (Tue)
by atnot (subscriber, #124910)
[Link]
Posted Jun 5, 2024 23:53 UTC (Wed)
by ringerc (subscriber, #3071)
[Link]
Posted Jun 4, 2024 6:39 UTC (Tue)
by ibukanov (subscriber, #3942)
[Link] (1 responses)
If SSD is older, the system uses a magnetic disk or SSD is modern but has to be encrypted with software encryption like LUKS, then use compressed memory like zram. It will also provide the benefits of gradual performance degradation when the system runs out of RAM.
Posted Jun 4, 2024 17:15 UTC (Tue)
by patrakov (subscriber, #97174)
[Link]
Posted Jun 4, 2024 9:22 UTC (Tue)
by farnz (subscriber, #17727)
[Link]
I get that result with a combination of 1 GiB swap and systemd-oomd, on a system with 64 GiB RAM. That's just enough swap that under memory pressure, I don't immediately start swapping executable pages in preference to less-frequently used data, but not so much swap that I thrash. And this gives systemd-oomd just enough headroom to kill off the actual memory hog, rather than an unfairly maligned victim.
Posted Jun 3, 2024 20:23 UTC (Mon)
by NAR (subscriber, #1313)
[Link] (2 responses)
Posted Jun 3, 2024 20:39 UTC (Mon)
by mb (subscriber, #50428)
[Link] (1 responses)
However, swap is also used to free (likely) completely unused pages from memory and use the free memory for something else that improves performance (e.g. disk cache).
Or it can be used to get rid of tmpfs files from RAM that are never used.
That is all happening in the background and has nothing to do with a runaway or overload scenario.
At the moment I have ~1% of my RAM freed for useful tasks due to swap. That's not much, but it's better than not having it.
Posted Jun 4, 2024 10:25 UTC (Tue)
by jhe (subscriber, #164815)
[Link]
Posted Jun 3, 2024 20:41 UTC (Mon)
by flussence (guest, #85566)
[Link]
Disabling the swap subsystem will not make things faster in the general case - you just lose a "L5" cache tier and the kernel has less verbs to work with for managing memory pressure. (Incidentally, the whole point of DAMON/DAMOS is to let the kernel use those verbs more effectively. It's intended for NUMA hot/cold balancing, but the distinction between that and swapping is becoming largely academic and I wouldn't be surprised to see it in use on desktop distros in a few years.)
Posted Jun 4, 2024 3:14 UTC (Tue)
by Paf (subscriber, #91811)
[Link]
Posted Jun 4, 2024 3:48 UTC (Tue)
by ringerc (subscriber, #3071)
[Link] (2 responses)
Posted Jun 4, 2024 5:42 UTC (Tue)
by zdzichu (subscriber, #17118)
[Link] (1 responses)
Posted Jun 6, 2024 0:06 UTC (Thu)
by ringerc (subscriber, #3071)
[Link]
It has also taken quite a while to make its way through to the CSP-hosted k8s stacks I'm unfortunately stuck with, and then through upgrade cycles.
Posted Jun 3, 2024 18:32 UTC (Mon)
by mgedmin (subscriber, #34497)
[Link] (5 responses)
There's a slight risk that programs who try to download large files into /tmp might end up filling it up completely and failing.
Posted Jun 4, 2024 6:04 UTC (Tue)
by smurf (subscriber, #17840)
[Link] (4 responses)
Large temporary stuff which you *know* is too large for RAM+swap belongs in /var/tmp IMHO.
Posted Jun 4, 2024 11:25 UTC (Tue)
by Wol (subscriber, #4433)
[Link] (3 responses)
That's not what /var/tmp is for ...
Just redefine /tmp as being larger. Running gentoo, I moved a lot of my compilation temporary dirs into /tmp, and seeing as I always run with masses of swap I think I had /tmp defined as maybe three or four times ram? On the basis that nearly all this stuff was pretty much WORN, and if compiling Firefox, or loffice, or gcc, etc etc pushed my machine into swapping, quelle surprise.
But as part of that, I found the definition of /tmp and /var/tmp in the FHS, and discovered that me making /var/tmp a tmpfs too was a big mistake ...
/tmp is defined as "anything that could disappear at any time", so not surviving a reboot was completely within spec.
/var/tmp on the other hand, is stuff "that isn't needed for long, but that must survive a reboot or crash". Don't remember the definition exactly, but it's clearly for recovery information, or working sets that should be deleted by the app but often aren't thanks to a crash, yadda yadda. Stuff that is likely to get orphaned and you don't want cluttering up permanent storage.
Cheers,
Posted Jun 4, 2024 11:31 UTC (Tue)
by adobriyan (subscriber, #30858)
[Link] (1 responses)
For new gentooers finding this post: /var/tmp/portage is Gentoo's graveyard of failed builds.
Set up some kind of autocleaner against the directory.
Posted Jun 6, 2024 7:50 UTC (Thu)
by taladar (subscriber, #68407)
[Link]
Posted Jun 5, 2024 5:03 UTC (Wed)
by wtarreau (subscriber, #51152)
[Link]
When I want something in RAM, I'm using /dev/shm which has been available as a tmpfs everywhere for well over a decade.
While I can understand the rationale for mounting /tmp as a tmpfs, I think that some users will complain that their system is super slow, because while swap does help freeing cold pages, when a system starts to swap out due to filling a tmpfs with lots of stuff, all caches are purged first, and even task switching can be slow because previously inactive processes have been swapped out. These are the situations where you can see processes lag a lot, especially with graphical environments often composed of lots of processes.
Another annoying case is with low-memory systems. I don't know how much RAM debian wants these days, but having only a few gigs in /tmp can definitely be a limiting factor for some users.
I think another approach could be to have /tmp as disk storage and /tmp/ram be ram-based. But there's no solution that would satisfy everyone anyway, so probably that commenting/uncommenting a line in fstab remains the best option. After all, if users are no longer able to add a hash at the beginning of a line in fstab, we probably have much more serious problems to solve!
Posted Jun 3, 2024 21:49 UTC (Mon)
by LtWorf (subscriber, #124958)
[Link] (1 responses)
I'm sure there's a lot of situations where that backfires.
Posted Jun 4, 2024 6:08 UTC (Tue)
by smurf (subscriber, #17840)
[Link]
Posted Jun 3, 2024 18:33 UTC (Mon)
by bluca (subscriber, #118303)
[Link] (2 responses)
That was very much intentional, because that sort of rat-holing is why this failed to happen before. For each and every anecdote about why it's not an issue, there will be an opposite example from somebody else. This is a default that can be trivially customized, and that includes both completely disabling it or setting a max filesystem limit, and a myriad of other combinations. If it works out of the box, great - if not, just customize it as needed.
Posted Jun 3, 2024 19:09 UTC (Mon)
by Heretic_Blacksheep (guest, #169992)
[Link] (1 responses)
Posted Jun 3, 2024 19:20 UTC (Mon)
by bluca (subscriber, #118303)
[Link]
Posted Jun 4, 2024 10:28 UTC (Tue)
by abo (subscriber, #77288)
[Link]
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15....
Posted Jun 3, 2024 18:29 UTC (Mon)
by mgedmin (subscriber, #34497)
[Link] (2 responses)
In the morning I discovered a collection of interesting facts: (1) wget sets the mtime according to the Last-Modified header, (2) my system had a daily cron script that deleted old files in /tmp, (3) as a result of (1) and (2) my webcomic archives only had a few of the latest files left.
A Lesson Was Learned that day. (Also, I somehow managed to re-download the files I needed and burn the CD-ROM before my flight.)
Posted Jun 4, 2024 5:55 UTC (Tue)
by smurf (subscriber, #17840)
[Link]
Posted Jun 7, 2024 16:20 UTC (Fri)
by rkraats (subscriber, #45998)
[Link]
Posted Jun 3, 2024 20:01 UTC (Mon)
by rknight (subscriber, #26792)
[Link] (7 responses)
Posted Jun 3, 2024 20:15 UTC (Mon)
by bluca (subscriber, #118303)
[Link] (2 responses)
Posted Jun 3, 2024 21:52 UTC (Mon)
by python (guest, #171317)
[Link] (1 responses)
Surely apt would just crash?
Posted Jun 3, 2024 22:24 UTC (Mon)
by bluca (subscriber, #118303)
[Link]
Posted Jun 3, 2024 21:18 UTC (Mon)
by Klaasjan (subscriber, #4951)
[Link] (3 responses)
Posted Jun 4, 2024 13:41 UTC (Tue)
by rknight (subscriber, #26792)
[Link] (2 responses)
Posted Jun 4, 2024 19:39 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Posted Jun 6, 2024 10:46 UTC (Thu)
by aragilar (subscriber, #122569)
[Link]
Posted Jun 3, 2024 21:44 UTC (Mon)
by LtWorf (subscriber, #124958)
[Link] (13 responses)
So of course I had a compilation fail because all the 7GB of space on my now very tiny /tmp got occupied by dependencies.
It seems one of these things that you have to do when you install a Debian machine. Like disable the mouse in vim.
Posted Jun 4, 2024 1:40 UTC (Tue)
by mathstuf (subscriber, #69389)
[Link] (12 responses)
Posted Jun 4, 2024 8:59 UTC (Tue)
by dsommers (subscriber, #55274)
[Link] (11 responses)
Posted Jun 4, 2024 9:22 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (10 responses)
Posted Jun 4, 2024 10:19 UTC (Tue)
by LtWorf (subscriber, #124958)
[Link] (9 responses)
[Citation needed]
As I said in another comment, downloading an .iso file was enough to kill my entire session.
I don't think that "downloading a file to then dd on a USB device and never use it again" is a rare thing among linux users.
Posted Jun 4, 2024 10:28 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (8 responses)
Posted Jun 4, 2024 10:49 UTC (Tue)
by LtWorf (subscriber, #124958)
[Link] (7 responses)
Which is not automatically cleaned, which is why I didn't use it in that case.
> It's completely absurd to demand that defaults should adapt to your specific use case of downloading ISOs to /tmp.
It seems to me that the use case you have in mind is "do not write anything at all ever in /tmp". Do you have any source that it is the normal use case and I am indeed the only one person in the whole world that uses it?
Why not remove /tmp completely then?
Posted Jun 4, 2024 10:51 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (6 responses)
You can trivially set up a tmpfiles.d in your home that cleans it, if you want that behaviour. Or you can use /var/tmp. The possibilities are endless.
> It seems to me that the use case you have in mind is "do not write anything at all ever in /tmp".
No, this is a strawman that you have just made up.
Posted Jun 4, 2024 15:34 UTC (Tue)
by Wol (subscriber, #4433)
[Link] (2 responses)
And so is the (mis)information on the net.
What you're forgetting, is that a lot of people just want to USE their computers. And trying to find information about eg tmpfiles.d is just a nightmare of ill-informed blogs and crap documentation.
I know pretty much the minimum I need to do to administer my personal home server. One of the reasons it runs gentoo is it was a personal choice that means I need to know more than most. But finding the information I need is usually search terms that don't find what I'm looking for, search engines that assume they know better than I do what I'm looking for so they ignore critical terms, as I said, ill-informed blogs, answers that didn't read the question, etc etc.
Just because the possibilities are endless doesn't mean they are within reach of mere mortals - all too often they've disappeared up the wazoo of the infinite impossibility search engine ...
Cheers,
Posted Jun 4, 2024 15:56 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (1 responses)
Posted Jun 4, 2024 17:28 UTC (Tue)
by Wol (subscriber, #4433)
[Link]
As someone else said, things like attachments in emails are generally stored in temp (be it c:\temp or \tmp or yada yada) and the grief I get with "I edited an attachment and saved it, where have my changes gone?"
Some people learn, unfortunately some people CAN'T learn. What's that saying? "Every time man invents something idiot-proof, nature invents a better idiot".
I get people are upset that Debian is being dragged kicking and screaming into the 21st century, but it doesn't mean the 21st is any better than the 20th ...
Cheers,
Posted Jun 5, 2024 5:15 UTC (Wed)
by wtarreau (subscriber, #51152)
[Link] (2 responses)
> No, this is a strawman that you have just made up.
No need to be aggressive and condescending again Bluca.
I'm in the same situation with my home over NFS, I use /tmp *a lot*, for the same reasons. Where do I download LLMs ? Into /tmp. Where do I compile large stuff ? In /tmp as well, because it's fast (especially with SSDs these days). There are plenty of small use cases like this. I'm totally fine with uncommenting a line in /etc/fstab to preserve the old behavior though. But I think that during installation it would be reasonable to check the available RAM size and consider a size below which the user is asked whether they want /tmp as tmpfs or not, because running OOM on older systems that were once considered as "still good enough to run debian" will likely cast a bad image of that distro.
Posted Jun 5, 2024 9:16 UTC (Wed)
by bluca (subscriber, #118303)
[Link] (1 responses)
There is also no need to invent arguments that nobody made, and yet here we are
> I'm in the same situation with my home over NFS, I use /tmp *a lot*, for the same reasons.
NFS is not enabled by default either, so as part of your actions to setup NFS, you can also configure your tmp according to your needs
Posted Jun 5, 2024 13:37 UTC (Wed)
by edgewood (subscriber, #1123)
[Link]
Yes. But that's what he said 4 sentences after the one you quoted.
Posted Jun 3, 2024 21:53 UTC (Mon)
by Thalience (subscriber, #4217)
[Link] (18 responses)
Posted Jun 4, 2024 1:32 UTC (Tue)
by raven667 (subscriber, #5198)
[Link] (3 responses)
Posted Jun 4, 2024 2:52 UTC (Tue)
by ringerc (subscriber, #3071)
[Link]
Their rationale? That's where they put stuff they're done with but might need later. They'd empty it at the end of the month.
Obviously absurd given the real world analogy, you say? But no. The cleaners had strict instructions not to empty this person's actual rubbish bin either. For the same reason.
The network backups excluded the "recycle bin". Items had been moved there long enough ago that they'd aged out of the backup retention window. Someone came in to stand in for them at their job and emptied it. That was fun.
Posted Jun 4, 2024 6:08 UTC (Tue)
by epa (subscriber, #39769)
[Link] (1 responses)
Posted Jun 4, 2024 6:55 UTC (Tue)
by magfr (subscriber, #16052)
[Link]
Posted Jun 4, 2024 7:58 UTC (Tue)
by LtWorf (subscriber, #124958)
[Link] (13 responses)
My whole session got killed because I dared to use /tmp. It wasn't even full.
At this point, if we don't want to have /tmp, we could just not have a /tmp at all.
Posted Jun 4, 2024 8:43 UTC (Tue)
by smurf (subscriber, #17840)
[Link] (12 responses)
Long term, /tmp-on-disk won't save you here.
Posted Jun 4, 2024 9:07 UTC (Tue)
by LtWorf (subscriber, #124958)
[Link] (11 responses)
A fresh install did that. So it was just the system defaults.
Anyway let's not forget that the advice online has been "do not create swap" for decades. So there's probably thousands of installations out there without swap (not mine) that are ready to start crashing once users start to receive this on their systems.
Posted Jun 4, 2024 9:28 UTC (Tue)
by smurf (subscriber, #17840)
[Link] (3 responses)
In any case, on a laptop you also need the swap space for hibernation (suspend to disk).
Posted Jun 4, 2024 10:54 UTC (Tue)
by LtWorf (subscriber, #124958)
[Link] (1 responses)
Suspend to disk is slow, suspend to RAM is fast… Also if your swap is full of the contents of your /tmp you can't hibernate anyway.
Posted Jun 4, 2024 13:02 UTC (Tue)
by smurf (subscriber, #17840)
[Link]
So don't fill the whole swapspace with your /tmp. It defaults to being limited to half of RAM for a reason. This is mostly orthogonal to /tmp-in-RAM, you have the same problem if (when …) your browser has a memory leak and ate your whole RAM+Swap when you want to hibernate.
Posted Jun 4, 2024 18:39 UTC (Tue)
by MarcB (subscriber, #101804)
[Link]
Heavy swapping can easily exceed those values. If the swapping happens on many VMs at once - maybe because they run the same buggy software or run into the same problematic data - the whole cluster might get into real trouble.
But in all scenarios where your storage gives sufficient guarantees, you really should use swap.
Posted Jun 4, 2024 11:27 UTC (Tue)
by intelfx (subscriber, #130118)
[Link] (3 responses)
That's just crappy advice.
It happens in all parts of life. Why should the world bend over to wrong things someone wrote and perpetuated on the Internet? You don't see vaccine centers shutting down because of anti-vax bullshit :-)
Posted Jun 4, 2024 12:04 UTC (Tue)
by malmedal (subscriber, #56172)
[Link] (2 responses)
> That's just crappy advice.
It's obsolete advice, a 7200 RPM desktop drive can do something like 72 seeks per second. I had plenty of workloads that would work fine without swap(or at least be OOM-killed), but completely hang the machine when swap was enabled.
With an SSD-drive the problem went away so it's reasonable to enable swap again.
Posted Jun 4, 2024 12:06 UTC (Tue)
by intelfx (subscriber, #130118)
[Link] (1 responses)
I'd hazard a guess and say this just meant `vm.swappiness` had been set to a wrong value.
Posted Jun 4, 2024 15:23 UTC (Tue)
by malmedal (subscriber, #56172)
[Link]
The disk at the time could read 150M/sec sequentially, but only 288K/sec with random access 4K blocks. Typical for a 7200RPM desktop-drive.
The performance cliff when swap was just a little bit too large was so tall that I just turned it off.
Posted Jun 4, 2024 11:39 UTC (Tue)
by Wol (subscriber, #4433)
[Link] (2 responses)
Don't forget that, round about 2000, linux 2.4, don't remember the details, Linus decided to force a rewrite of the memory management / swap code. At which point, people who ran vanilla kernels and didn't read the release notes suddenly found out the HARD way that the old wives tale of "swap should be twice ram" really was true. Any system that didn't have that much swap just crashed the instant the swap code was invoked. The guidance in the notes said "no swap, or at least twice ram. NOTHING INBETWEEN".
And there were plenty of people who'd been relying on the fact that it was an old wives' tale for years even back then.
Which is why my machines always have masses of swap. May be totally unnecessary today, but nobody's ever come back to me and said "yes the rewrite really did remove that requirement" - they just assume that because everybody says "twice swap" is an old wives' tale, then it must be, despite that being proved resoundingly wrong this century-ish.
Cheers,
Posted Jun 7, 2024 5:35 UTC (Fri)
by porridge (subscriber, #15054)
[Link] (1 responses)
Suddenly you start running out of storage space for your files :-(
Posted Jun 7, 2024 7:06 UTC (Fri)
by Wol (subscriber, #4433)
[Link]
Cheers,
Posted Jun 4, 2024 5:43 UTC (Tue)
by stop50 (subscriber, #154894)
[Link]
Personally i would go for /var/lib, but you must clean it up yourself.
Posted Jun 4, 2024 6:57 UTC (Tue)
by magfr (subscriber, #16052)
[Link] (37 responses)
Posted Jun 4, 2024 8:16 UTC (Tue)
by epa (subscriber, #39769)
[Link] (36 responses)
Posted Jun 4, 2024 8:56 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (35 responses)
Posted Jun 4, 2024 9:01 UTC (Tue)
by epa (subscriber, #39769)
[Link] (34 responses)
Posted Jun 4, 2024 9:18 UTC (Tue)
by smurf (subscriber, #17840)
[Link] (32 responses)
The correct place for transient stuff that shouldn't be cleaned (and also doesn't need to survive a reboot), these days, is /run/user/‹uid›/.
Posted Jun 4, 2024 9:30 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (28 responses)
Posted Jun 4, 2024 11:36 UTC (Tue)
by epa (subscriber, #39769)
[Link] (27 responses)
Posted Jun 4, 2024 11:41 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (26 responses)
Posted Jun 4, 2024 12:24 UTC (Tue)
by epa (subscriber, #39769)
[Link] (3 responses)
This is speculation, of course, but a generation ago the idea of deliberate hash collisions would have been seen as a straw man, or many similar “don’t do that” or “will never happen” scenarios.
Since the reaping period is defined to be at least 10 days (and not purely site-dependent) that takes away a lot of the worry in practice.
Posted Jun 4, 2024 14:34 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (2 responses)
All these tortured strawmen are exactly what I meant when I said "I am not really looking for philosophical discussions or lists of personal preferences or hypotheticals". If the new defaults do not work for you, just do the trivial one-line configuration change. Arguing about why your preferred configuration is the one true default and everything else is wrong is a waste of time and is not going to achieve anything at all.
Posted Jun 4, 2024 14:51 UTC (Tue)
by epa (subscriber, #39769)
[Link] (1 responses)
If you're saying that is never safe, not even when done with great care and exclusive mode opening and all the rest, then our positions are not so far apart. The safe uses of /tmp are a lot fewer than is common practice.
Posted Jun 4, 2024 14:58 UTC (Tue)
by bluca (subscriber, #118303)
[Link]
For sockets - just don't. It is honestly baffling that there are still programs doing that, like tmux.
Posted Jun 4, 2024 15:51 UTC (Tue)
by mb (subscriber, #50428)
[Link] (21 responses)
No, it's not broken. It worked just fine until you started deleting files.
> It can be fixed in a number of ways
Yeah. But you are breaking existing userspace here.
I'm really annoyed by this.
If we are talking about megabytes or gigabytes. Yeah. Fine. Establish a cleanup strategy, because it actually does help.
Posted Jun 4, 2024 16:10 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (20 responses)
Yes, it is, and no, it didn't, as the article says at the very top this has been the norm on other distros for a decade.
> But you are breaking existing userspace here.
Nothing is broken, this is made-up nonsense. Just stop wasting time and write the one line config file to customize your machine as you want it. Complaining is not going to change anything.
> But deleting small files and sockets? Just don't. There is no way this is going to help anybody, but plenty of room for breakage.
Size limit is not the only problem, especially on a tmpfs there is also an inode limit, and saturating a tmpfs with zero size files is very much possible and DOSes other uses of /tmp. So it's just as important to remove unused small files as it is large files to ensure the shared resource remains usable.
Posted Jun 4, 2024 16:31 UTC (Tue)
by mb (subscriber, #50428)
[Link] (19 responses)
Ok. Can you give us some numbers and show how that is not "made-up nonsense" (quoting yourself), please?
How many files must be created to exhaust the available space?
>write the one line config file to customize your machine as you want it
My machine _is_ configured the way I want and you are changing it. And I didn't read a plausible explanation for why this is needed, yet.
You are changing user interfaces. You are doing the very same thing that you constantly complain about.
Posted Jun 4, 2024 16:51 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (18 responses)
Depends on the system, just check it out - df -hi /tmp
> But there surely have been decades of Linux that don't delete files from /tmp at random times.
Sure there have - tmpfiles.d has been deleting files in /tmp, in Debian too, for the past ~15 years, and before that there were things like tmpreaper. Other OSes like Solaris also delete stuff there by default. You are pretending this is some sort of new, last minute development - it isn't, it's ancient history.
> You are changing user interfaces. You are doing the very same thing that you constantly complain about.
Nonsense. Major version of distributions change things all the time, that's why we publish this thing called "release notes" where changes are noted. If you don't want changes, stay on the same stable LTS version forever. Distributions, including Debian, never, ever said "we won't break compatibility across major version upgrades", quite the opposite, every new release changes something - that's the entire point of doing a new Debian release. The parallel you are trying to draw makes no sense, as the kernel _does_ say " we do not break compatibility". That makes all the difference, and pretending it doesn't is just silly.
Posted Jun 4, 2024 17:28 UTC (Tue)
by mb (subscriber, #50428)
[Link] (17 responses)
Ok.
>$ df -hi /tmp
>$ for i in $(seq 0 100000); do touch /tmp/x/$i; done
>$ df -hi /tmp
So it takes ~4.5 million empty files to fill up the available inode space.
It doesn't convince me that this limit can be used to argue that possibly used files must be deleted after 10 days.
Is that limit hit more frequently than applications breaking due to deleted files?
Posted Jun 4, 2024 17:38 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (16 responses)
There must be some misunderstanding here - I am not arguing for anything, not trying to convince you of anything. I'm simply explaining how things work and why. If you don't like how they work, that's too bad - feel free to configure your machine differently as already explained, you are free to configure your laptop or your desktop or your server as you see fit, it really doesn't make any difference to me, these settings are all configurable by design
Posted Jun 4, 2024 18:37 UTC (Tue)
by mb (subscriber, #50428)
[Link] (15 responses)
Well, but you should. You are changing the Debian world here.
>feel free to configure your machine differently
Keep in mind that you are changing my machine behavior here. You change how the Debian system works and I have to revert back to the old behavior.
It's the responsibility of you as a maintainer to have actual good reasons for a change. And you should be able to explain them. Yet, I did not see more than basically that you'd like it better if the files were deleted and "somebody else does it, too".
There is a huge responsibility with being a maintainer.
Posted Jun 4, 2024 18:43 UTC (Tue)
by pizza (subscriber, #46)
[Link] (6 responses)
No, he (nor the other systemd developers) is doing nothing of the sort. *DEBIAN* is changing its own world.
Posted Jun 4, 2024 19:28 UTC (Tue)
by mb (subscriber, #50428)
[Link] (5 responses)
Posted Jun 4, 2024 20:46 UTC (Tue)
by pizza (subscriber, #46)
[Link] (4 responses)
Sounds like you need to file a release blocker bug with Debian, but since this is fundamentally about overruling a Debian packager's decision, you'll probably need to escalate this to the TC or to a full project-wide vote.
Either way, you'll need a better technical argument than "I don't like it and I can't be arsed to make a one-liner configuration change"
Posted Jun 4, 2024 20:50 UTC (Tue)
by bluca (subscriber, #118303)
[Link]
Posted Jun 4, 2024 21:25 UTC (Tue)
by mb (subscriber, #50428)
[Link] (2 responses)
Yes. Changes don't require technical argument. "Only one reason is needed: I have decided as such." (bluca). And the users require technical argument against it. I got it and I will respect it.
I will certainly remember that rule.
Posted Jun 4, 2024 21:57 UTC (Tue)
by pizza (subscriber, #46)
[Link] (1 responses)
Except for the numerous times the technical merit of the "new" default has been explained, even in this thread. And then there's also the little detail where where the rest of the Linux world has been doing it this "new" way for about a decade. Those discussions (including technical arguments) are all part of their public record.
> "Only one reason is needed: I have decided as such."
Hate to break this to you, but the overwhelming majority of Debian packages also fall under this category - Both the packager and the upstream authors make countless such decisions every single day.
And again, if you (or whomever) doesn't like these decisions, feel free to invoke the Debian process for overruling a packager and/or upstream.
Posted Jun 4, 2024 21:59 UTC (Tue)
by corbet (editor, #1)
[Link]
Posted Jun 4, 2024 19:11 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (7 responses)
No. The next Debian release, due next year, will change your machine's behaviour in many ways, as it will be documented in the (lengthy) release notes. If you like it - good. If you don't - too bad, either reconfigure appropriately or stay on the current version until it goes EOL, and then start paying somebody to give you support beyond that.
> There should be a *good* reason for these changes.
Only one reason is needed: I have decided as such. Don't like it? Don't worry, you can have your money back, give me an address and I'll post you a cheque for £0.
Posted Jun 4, 2024 19:13 UTC (Tue)
by bluca (subscriber, #118303)
[Link]
Posted Jun 4, 2024 19:26 UTC (Tue)
by mb (subscriber, #50428)
[Link] (1 responses)
Ok. That's the rule and I will also apply it to you.
Posted Jun 4, 2024 19:33 UTC (Tue)
by corbet (editor, #1)
[Link]
Posted Jun 6, 2024 1:49 UTC (Thu)
by jccleaver (guest, #127418)
[Link] (3 responses)
Yep, that's bluca's and systemd's philosophy in a nutshell, right there.
Posted Jun 6, 2024 9:13 UTC (Thu)
by bluca (subscriber, #118303)
[Link] (1 responses)
Posted Jun 6, 2024 12:16 UTC (Thu)
by Wol (subscriber, #4433)
[Link]
If it's just Debian deciding to adopt systemd's defaults, that's not down to systemd.
bluca can be a pain in the arse - can't we all - but unless he's responsible for Debian policy I fail to see how this has anything to do with him.
(Or is this the transition where now-stable is SysVInit, and next-stable is systemd, but even there I fail to see how this is down to systemd at all ...)
Cheers,
Posted Jun 6, 2024 10:40 UTC (Thu)
by smurf (subscriber, #17840)
[Link]
It's a freakin' default. You don't like it, change it. Other systems have been using the same default settings for literally decades.
If your argument boils down to "it's a change and I don't like it", well, open your window, there's way worse changes than putting /tmp into RAM looming on the horizon. Doing something about these would be a more productive use of everybody's time.
Posted Jun 4, 2024 11:34 UTC (Tue)
by epa (subscriber, #39769)
[Link] (2 responses)
Posted Jun 4, 2024 11:39 UTC (Tue)
by bluca (subscriber, #118303)
[Link] (1 responses)
Posted Jun 4, 2024 12:02 UTC (Tue)
by epa (subscriber, #39769)
[Link]
While in general I dislike timing-dependent effects, such as timeouts after inactivity on a socket connection, they are a fact of life.
Posted Jun 4, 2024 9:33 UTC (Tue)
by bluca (subscriber, #118303)
[Link]
Posted Jun 4, 2024 12:28 UTC (Tue)
by smoogen (subscriber, #97)
[Link] (3 responses)
Add in the fact that Debian is the NetBSD of Linux (and I mean this as a compliment). It is ported and maintained on hardware systems which have not been produced in decades, and meant to work on systems with megabytes of memory while also working on terabytes of memory. This means that general solutions which might work in 90% of a different distro cause larger complaints due to expectations that someone's SPARC10(?) or M68000 system will still install cleanly in 2024.
I would expect that in the older Debian installers this would be a question to answer.. do you want /tmp to be tmpfs, dedicated partition, or on / so that the various solutions could be implemented. I don't know if that is at all possible with the newer installers.
Posted Jun 4, 2024 18:27 UTC (Tue)
by mathstuf (subscriber, #69389)
[Link]
- unaccounted to projects
Posted Jun 5, 2024 5:39 UTC (Wed)
by wtarreau (subscriber, #51152)
[Link]
Posted Jun 5, 2024 7:21 UTC (Wed)
by nim-nim (subscriber, #34454)
[Link]
Posted Jun 4, 2024 16:07 UTC (Tue)
by gray_-_wolf (subscriber, #131074)
[Link] (8 responses)
Maybe the /tmp should be (by default) of reasonable size (say 50 GB) with large enough swap created by default?
Posted Jun 4, 2024 16:16 UTC (Tue)
by smurf (subscriber, #17840)
[Link] (6 responses)
Or guix could do that for you, out of the box.
Posted Jun 4, 2024 16:27 UTC (Tue)
by gray_-_wolf (subscriber, #131074)
[Link] (5 responses)
My point being that there are programs that legitimately use many gigabytes in /tmp, and "tmpfs with max size of 1/2 of RAM" will just not work for them. I took guix package of an example that is packaged by debian, and therefore it seems reasonable to have it work out of the box.
Posted Jun 4, 2024 16:31 UTC (Tue)
by bluca (subscriber, #118303)
[Link]
Posted Jun 5, 2024 1:56 UTC (Wed)
by nakedhitman (subscriber, #90828)
[Link] (3 responses)
Depending on what you're compiling, each compilation thread will use up to a certain amount of temp space. That bit me a few times, but once I knew what it was, I was easily able to work around it.
> My point being that there are programs that legitimately use many gigabytes in /tmp, and "tmpfs with max size of 1/2 of RAM" will just not work for them.
tmpfs will work just fine for use cases that exceed the size of RAM. Just add swap (even better with zswap) and size accordingly. That was the solution in my case, and my system has been better for it overall. Systems really should be using swap anyway, and tmpfs should default to using a size that matches either swap or 50% RAM, whichever is greater.
Posted Jun 5, 2024 20:20 UTC (Wed)
by mbunkus (subscriber, #87248)
[Link] (2 responses)
Out of curiosity: which build systems place significant amounts of data in /tmp, even temporarily? And for what? With "significant" I mean more than a handful of KB per compilation process so that the total amount can become significant enough.
Posted Jun 6, 2024 8:17 UTC (Thu)
by smurf (subscriber, #17840)
[Link] (1 responses)
The result of preprocessing a ten-line C or C++ file can and will easily span multiple megabytes.
Same for the intermediate assembly output, esp. when you add debugging.
Posted Jun 6, 2024 15:25 UTC (Thu)
by mbunkus (subscriber, #87248)
[Link]
Posted Jun 5, 2024 7:53 UTC (Wed)
by intelfx (subscriber, #130118)
[Link]
Maybe by making Guix NOT do that? :-)
I do not use Guix, but I do have a home-grown package builder tool with lots of bells and whistles for painless automation (because I fork or patch so many packages). One of the first things I taught this tool is a dedicated "this package is large" annotation, which I manually use to mark packages like web browsers or toolchains or Linux kernel. If this annotation is not present, the build happens in /tmp, otherwise it happens in /var/tmp.
If Guix desires maximum efficiency in building packages, surely it must not be hard to implement something like that.
Posted Jun 4, 2024 19:37 UTC (Tue)
by dw (subscriber, #12017)
[Link] (32 responses)
To then come along and say "surprise, we moved your filesystem back into RAM, but it's okay, because if you run out of RAM, we'll just page out half your system (including the function that draws your mouse pointer) back to disk, so now you must enable swap too" frankly just angers me.
I have no idea why anyone is still looking at this part of the stack or who is paying for that work, it's just creating dumb breakage for absolutely stupid gains. Not bothering with the list thread, but presumably these fuzzy feelings weren't backed up by any meaningful metrics?
Posted Jun 5, 2024 1:58 UTC (Wed)
by nakedhitman (subscriber, #90828)
[Link]
Right back at ya.
Posted Jun 5, 2024 8:57 UTC (Wed)
by koh (subscriber, #101482)
[Link] (30 responses)
I can add one data point here:
This my personal laptop used every day, also for work, without periodic /tmp cleaning. What I do is removing specific files in /tmp from programs I know at specific points in time when I know those programs aren't running and might potentially use those files. Knowledge of either cannot reliably be obtained by any automated solution.
Posted Jun 5, 2024 9:24 UTC (Wed)
by bluca (subscriber, #118303)
[Link] (29 responses)
That's correct, and it's not going to be, as it would be completely pointless: for something like this one can produce millions of manufactured anecdotes that go one way or the other. The previous defaults also weren't established following the rigorous analysis of any large dataset, but simply through inertia. However, given you like those old defaults and they happen to fit your use case, you don't seem to have any problem with the lack of data backing them up. Funny how that works, eh?
> This my personal laptop used every day, also for work, without periodic /tmp cleaning.
That's great - then you can use one of the several available configuration options to fine-tune it to your specific needs, which is of course fully supported and well documented.
Posted Jun 5, 2024 10:58 UTC (Wed)
by koh (subscriber, #101482)
[Link] (28 responses)
To be honest, I don't care either way. What I care about is changing defaults in spite of existing technical criticism. Changing defaults is a decision affecting the technical level, so this criticism should be taken seriously. In my opinion, a less condescending attitude towards it would go a long way to increase acceptance.
Just to drive this point home: my Laptop does not use Debian and has a tmpfs based /tmp and /var/tmp. The change won't affect it. But it will affect some of the servers I manage - for no specific or specified reasons. That would be my complaint. I understand that change in general is necessary, and I'm not opposed. But change for the sake of change is counter-productive. What's the reason for it to have to take place in this fashion of automated 10 days / 30 days cleanups? Is it possible to name programs cluttering those directories which need cleaning up after? Are users just too lazy and keep filling up those directories with junk? Is a time limit really the best way to decide what's junk and what isn't? Is "10 days" being determined for a constantly changing system clock (meaning: ntp)?
Posted Jun 5, 2024 11:47 UTC (Wed)
by bluca (subscriber, #118303)
[Link] (27 responses)
There is zero "technical criticism" (left - there were some valid concerns and some bugs about a handful packages, and I've fixed them all already). There are a bunch of personal preferences and anecdotes and strongly held personal opinions, but I don't care about any of that, sorry, because for each one of those preferences and anecdotes and opinions, you can find its exact opposite too. If the new settings don't fit with your personal use cases, you are fully empowered to customize them as needed, and it's beyond trivial to do so. Complaining about how they don't work for your very special and super-duper important use case is not going to achieve anything useful.
Posted Jun 5, 2024 12:04 UTC (Wed)
by koh (subscriber, #101482)
[Link] (22 responses)
I see, so all those comments about actual programs expecting /tmp not be cleaned up is no criticism about changing the defaults, got it.
> Complaining about how they don't work for your very special and super-duper important use case is not going to achieve anything useful.
a) Here is a prime example of the condescending attitude mentioned earlier.
Posted Jun 5, 2024 12:14 UTC (Wed)
by bluca (subscriber, #118303)
[Link]
Correct - they are personal anecdotes, or in a lot of cases, even just made up hypotheticals (but what about if...)
Posted Jun 5, 2024 12:24 UTC (Wed)
by smurf (subscriber, #17840)
[Link] (5 responses)
Correct – they are not. They report a bug in $PROGRAM (and/or its packaging) which should be taken upstream and/or fixed.
/tmp is not and never was persistent. /tmp-in-RAM or not doesn't matter.
Posted Jun 5, 2024 12:38 UTC (Wed)
by koh (subscriber, #101482)
[Link] (4 responses)
For instance, I work on a program that verifies semantics of other C programs. This program does use /tmp to store a bunch of C headers and other files to give the Clang frontend the illusion of working on a particular sysroot. How long the verification (and thus potential usage of those /tmp files) lasts is unclear: As we all know, in general it's not a computable problem, and even for a specific subset where it is, it might take a long time. 10 days is just arbitrary at this point. I can't change defaults on rented servers not under my control. So there's that problem with defaults. Changing them makes this problem of a technical nature. Note: I'm not claiming that my use case is "super-duper important" here. Just pointing out a technical issue that makes Debian for those servers less interesting for me.
I would argue that usage of /tmp to populate a temporary sysroot for a single verification task is well within the scope of /tmp, thus there's no bug in my program. Persistence of /tmp is not relevant for this point.
Posted Jun 5, 2024 12:56 UTC (Wed)
by bluca (subscriber, #118303)
[Link] (3 responses)
Fortunately for your use case, as per documentation, your program can just take an flock on the file(s) it is using, and they will be skipped over, no need to change configuration. Also, it's not just ctime, but mtime and atime too that are taken into account for the aging algorithm, again as documented. So it doesn't matter when the file was created, if it was just accessed it will be left where it is.
Posted Jun 5, 2024 13:02 UTC (Wed)
by koh (subscriber, #101482)
[Link] (1 responses)
Posted Jun 5, 2024 13:22 UTC (Wed)
by daroc (editor, #160859)
[Link]
B) Jon asked for a very similar thread (back and forth without anyone changing their position, covering many of the same points, including flock coming up) to stop yesterday. We're all smart people here; I am certain that we can all generalize. I don't think anything new is being said, which means we can stop here.
Posted Jun 5, 2024 14:10 UTC (Wed)
by smurf (subscriber, #17840)
[Link]
Thus /tmp-on-RAM should arguably work *better* than /tmp-on-root-with-no-or-rel-atime-option, which has been the default for ages.
Posted Jun 5, 2024 12:53 UTC (Wed)
by pizza (subscriber, #46)
[Link] (14 responses)
See, I don't get this. Debian's default behaviour *today* is to periodically reap /tmp.
Meanwhile, my very first Linux installation (27 years ago!) came with a nightly /tmp cleaner-upper cron job. And the multi-user Solaris systems I had accounts on also (fairly aggressively) reaped /tmp.
Also, let me quote the Linux FHS, section 3.18:
"Programs must not assume that any files or directories in /tmp are preserved between invocations of the program."
And section 3.15:
"The /var/tmp directory is made available for programs that require temporary files or directories that are preserved between system reboots. Therefore, data stored in /var/tmp is more persistent than data in /tmp."
(Note that this text is from the FHS3.0 release of 2015, but the same language can be found at least as far as v2.2 from 2002)
Going back even further to the pre-FHS FSSTND v1.0 spec (from Early 1994), one will find this:
/tmp is used for temporary files, preferably on a fast device (a memory based filesystem, for instance).
The "persistence" of the data that is stored in /tmp is different from that of data which is stored in /var/tmp. /tmp may be cleaned out at boot time or at relatively frequent intervals. Therefore, data stored in /tmp should not be expected to remain for any long period.
Programs should use /tmp or /var/tmp (which was originally /usr/tmp) according to the expected requirements of the data, but should not rely on any particular persistence for any temporary storage directories.
/tmp may be on a RAM disk. /var/tmp should never be located on a RAM device.
tl;dr: Putting /tmp into a ramdisk, frequent reaping, and/or clearing it on every boot, has been part of its very definition for at least 30 years on Linux, and quite likely longer than that on "real" UNIXen.
And any programs expecting files to persistent in /tmp, much less do so forever [1], have *always been broken* according to the documented standards of the system they were intended to run on.
[1] Without out-of-band system administrator intervention.
Posted Jun 5, 2024 13:38 UTC (Wed)
by paulj (subscriber, #341)
[Link]
Posted Jun 5, 2024 13:47 UTC (Wed)
by koh (subscriber, #101482)
[Link] (10 responses)
My Debian 10 based server doesn't seem to be doing that, without me having changed the settings. Which program/daemon is doing that?
> Also, let me quote the Linux FHS, section 3.18:
OK, I'm fine with that. What happens to programs that run longer than 10 days? I've described an example in a comment above. From other comments on this article I gathered that sockets and the like for daemons should be moved to /run, so at this point the question becomes: what's the difference between a long-running process (which uses, say, 100% CPU continuously) and a long-running daemon that should put its stuff under /run?
> /tmp may be cleaned out at boot time or at relatively frequent intervals. Therefore, data stored in /tmp should not be expected to remain for any long period.
That's just an unusable rule. Basically makes /tmp worthless as there are no longer any guarantees whatsoever. Rightly so it's no longer been specified in this crude way - at least I hope. Well, up to now, I guess.
Posted Jun 5, 2024 14:12 UTC (Wed)
by farnz (subscriber, #17727)
[Link] (9 responses)
The big use case for /tmp that's always been OK is "write out a small file that another program will immediately take in as input". If the dirent gets deleted not long after creation, that's A-OK, because the reader has already opened it and has a lock on the inode as a result (the file contents are therefore safe until the file is closed).
Basically for the case where you can't pipe output to the input of the other program, because it needs to seek or otherwise do file-like things on it, not pipe-like things.
Posted Jun 5, 2024 14:48 UTC (Wed)
by koh (subscriber, #101482)
[Link] (8 responses)
I suppose I should be glad that we finally have the following sequence atomic and uninterruptible:
1. spawn child process
All this time I was under the impression that it's not, but now appearently we can be sure that noone presses ctrl-z or puts the computer to sleep while compiling something...
Posted Jun 5, 2024 15:29 UTC (Wed)
by farnz (subscriber, #17727)
[Link] (7 responses)
We don't have that sequence; but you need some form of error handling for that case, anyway, since it's possible that you write out the file to a drive that physically fails between write and read - and there's never been a guarantee that the failure of the drive that contains /tmp will take down the system as a whole.
Posted Jun 5, 2024 15:37 UTC (Wed)
by mb (subscriber, #50428)
[Link] (6 responses)
Drive failure: The error reaction is: Throw an error and notify the user. Change the driver and the next run will succeed.
A "cleanup" daemon deleted my files: A sane error handling strategy does not exist. Next time the "cleanup" will delete my files again. The "cleanup" is the root cause of failure.
Posted Jun 5, 2024 16:48 UTC (Wed)
by farnz (subscriber, #17727)
[Link] (5 responses)
In the case where the user paused your program for multiple days, then got surprised, it's "re-run without stopping at a critical moment, and the next run will succeed". The cleanup won't delete your files again unless you have that long pause, which in the case I was responding to was introduced with Ctrl-Z or system suspend.
The underlying issue is the expectation that data in a temporary location is safe for an arbitrary amount of time; I could have replaced "Ctrl-Z" and wait with "Ctrl-Z, then rm -fr /tmp/*" for the same effect. You should already be handling "the system-wide shared temporary location might get erased while in use", because there's already lots of ways in which it could get erased behind your back; making it a tmpfs, or adding a clean-up daemon, is just one more way on the existing list of ways you could lose data in /tmp.
Posted Jun 5, 2024 21:00 UTC (Wed)
by koh (subscriber, #101482)
[Link] (4 responses)
The FHS-3.0 section 3.18 just says this about /tmp:
There is no wording indicating that processes can't rely, during their execution, on the integrity of the files they themselves have written (modulo system bugs and hardware errors, as usual). That indeed would be contradictory to the file system's only stated purpose: storing temporary files *required* by programs.
If my understanding is correct, periodic "cleaning" reduces reliability on purpose. tmpfs is fine, though, as long as resuming a suspend-to-disk doesn't touch it.
Posted Jun 5, 2024 21:14 UTC (Wed)
by bluca (subscriber, #118303)
[Link] (3 responses)
But for precious, precious data a state directory in /var/lib needs to be used instead. Using /tmp means tacit acknowledgement that "yes this might go away at any time, and it's fine, I'll just detect it and recreate it if needed". And that's not even getting into the issue of the various filename hijacking vulneratbilities, that are non-obvious to handle if one isn't aware of them.
Posted Jun 5, 2024 23:18 UTC (Wed)
by koh (subscriber, #101482)
[Link] (2 responses)
Citation needed.
> Using /tmp means tacit acknowledgement that "yes this might go away at any time, and it's fine, I'll just detect it and recreate it if needed".
A fractal of bad design: how can you "detect it" in particular when (rightly so) the advice has always been to use mkstemp(3) or similar?
Posted Jun 6, 2024 0:25 UTC (Thu)
by pizza (subscriber, #46)
[Link] (1 responses)
For Linux: Linux FHS 2.2, section 3.15 and 3.18, and before that, section 3.11 of the FSSTND, dating all the way back to Feburary 1994 [1], which was largely derived from standard Unix conventions.
Meanwhile, for Unix I can't be bothered to look it up, but every Unix system I had access to in the mid-late 90s [2] had an aggressively-reaped /tmp. Solaris and other Unix support forums show similar answers to [1].
[1] http://www.ibiblio.org/pub/Linux/docs/fsstnd/old/fsstnd-1.0/
Posted Jun 6, 2024 6:38 UTC (Thu)
by koh (subscriber, #101482)
[Link]
Thanks, I see.
Nitpick: Unlike [1], the FHS 2.2 and current 3.0 do no longer mention that dentrys in /tmp are unreliable by default. In fact, they state that programs may *require* temporary files stored therein. Appearently reliably only for WORN usage, since "detection" is racey with periodic cleanups.
Posted Jun 6, 2024 22:40 UTC (Thu)
by dskoll (subscriber, #1630)
[Link] (1 responses)
See, I don't get this. Debian's default behaviour *today* is to periodically reap /tmp. I don't believe that's the case. None of my Debian systems seem to do that. They clear /tmp on boot, but otherwise AFAIK the system doesn't do any regular cleaning of /tmp.
Posted Jun 8, 2024 0:23 UTC (Sat)
by pizza (subscriber, #46)
[Link]
You are correct -- It's been so long time since I did a new Debian installation that I forgot that tmpreaper isn't installed by default.
(But the fact that /tmp defaults to being cleaned at boot does obliterate arguments about how a ramdisk /tmp would behave any differently..)
Posted Jun 5, 2024 12:19 UTC (Wed)
by mb (subscriber, #50428)
[Link] (3 responses)
The biggest problem of all is how you communicate with people.
Posted Jun 5, 2024 12:52 UTC (Wed)
by bluca (subscriber, #118303)
[Link] (2 responses)
Posted Jun 5, 2024 16:21 UTC (Wed)
by rschroev (subscriber, #4164)
[Link] (1 responses)
What we have here is a case where one of alternatives has been in use for a very long time. At the same time seems that none of the alternatives is a clear winner (or at least that's the feeling I get from reading these comments). So why change? It leads to some people having to change their configuration, it leads to endless discussion, and as far as I can see has no real benefit.
From this discussion and the one about keepassxc, I get the feeling that Debian maintainers see every release as a new starting point, with not much consideration for keeping things the same between releases. What happened to "If it ain't broken, don't fix it" and the principle of least surprise?
> There is literally nothing of value to be gained from having an endless discussion about what's the best default for something that is entirely down to personal preferences - ask 10 people, you'll get 10 different answers.
So yes, choose a default, ignore the bike shedding. But once it's chosen (which in this case happened a long time ago), *stick with it* (until a clearly better alternative presents itself).
It could very well be that there actually is a good reason to change here, but honestly I haven't seen it anywhere in this whole discussion.
Posted Jun 5, 2024 16:24 UTC (Wed)
by bluca (subscriber, #118303)
[Link]
Nothing was chosen, there was just inertia. tmpfs as a filesystem wasn't even available at the very beginning.
Posted Jun 5, 2024 12:37 UTC (Wed)
by dskoll (subscriber, #1630)
[Link]
I hesitantly jump in to the fray here... I am not a fan of time-based deletion. Rather than making it the default, IMO it should be off by default with a very simple way to turn it on for those who want it. Otherwise it turns /tmp into a weirdly-broken disk drive.
I've been running with a ramdisk for /tmp for years, and current stats are as follows:
after an uptime of about 3 days, but this is typical even after much longer uptimes.
(Yes, I have 64GB of RAM in the machine.) I've never seen /tmp even come remotely close to filling up, so time-based deletion is completely unnecessary for me.
Posted Jun 5, 2024 18:17 UTC (Wed)
by wsy (subscriber, #121706)
[Link]
Posted Jun 9, 2024 22:12 UTC (Sun)
by anton (subscriber, #25547)
[Link]
And here's some data from a multi-user Debian 11 machine with 446 days of uptime:
Concerning using a RAMdisk and then using swap when memory runs out, with swap being supposedly usable again thanks to SSD speeds: Why do we want to use a RAMdisk in the first place if we have an SSDs and that is so fast? The multi-user machine shown above should do ok with /tmp on a RAMdisk (it has 128GB RAM, and 0 swap space), but OTOH the current arrangement (/tmp on ZFS) works fine.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
I don't auto-clean it, though. That is not worth the risk, IMO. tmpfs is swapped out to disk if needed.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Is swap really a good practice in high-performance computing environment ?
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
What you describe is extreme memory pressure because there's some runaway process or some process is completely overloading the available hardware resources.
You are right that swap hardly helps in these situations.
Unused pages in RAM just reduce the amount of usable RAM.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Wol
Debian's /tmpest in a teapot
> making /var/tmp a tmpfs too was a big mistake
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Wol
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Wol
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
User: What do you mean my trash automatically emptied itself? That's where I keep my important files!
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Wol
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Wol
Debian's /tmpest in a teapot
/var/lib/<something>: not cleaned up, usually no restrictions, useful for your service
/var/cache: like /tmp but for longer lived data, but you must expect them to be cleaned up.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
I'm not sure it actually exists though, it really sounds like a strawman.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Thanks, sorry I didn't realize it was you. I didn't intend to waylay your work with hypotheticals -- it looks like a good improvement and certainly the status quo ante wasn't perfect either.
Debian's /tmpest in a teapot
If "attacker takes over filename" is part of the threat model, _DO NOT USE /tmp_ for that data
Perhaps I am not understanding, but that seems to rule out /tmp for almost anything on a multiuser system. Many programs expect to create a file in /tmp and then reference it by name later. I know that the best way is to create the file in a single atomic action and get back the file *handle*, no longer caring about the name, but there are plenty of times when you want to run an external program and pass it the name of the temp file you created. For example running an external diff command to print the differences between two strings. I think that these cases outnumber those when you're content to use an anonymous file handle.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Is there a difference, if the kernel breaks userspace or if systemd breaks userspace?
Can you please give one valid reason, why you absolutely have to delete a couple of bytes from /tmp after it was sitting there without a problem for 10 days?
But deleting small files and sockets? Just don't. There is no way this is going to help anybody, but plenty of room for breakage.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Which real world application is affected?
Traditional disk based /tmp surely has the same problem, right? So there has never been a version of Linux without this "problem". But there surely have been decades of Linux that don't delete files from /tmp at random times.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
>Filesystem Inodes IUsed IFree IUse% Mounted on
>tmpfs 4.4M 37 4.4M 1% /tmp
>Filesystem Inodes IUsed IFree IUse% Mounted on
>tmpfs 4.4M 98K 4.3M 3% /tmp
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
If you make a change that requires millions of people to change just one configuration line to get the old behavior back, you are wasting millions of hours of work time.
There should be a *good* reason for these changes.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
I repeat: this has all been said, no more minds will be changed (if any ever were). I think we can stop this back-and-forth now.
Second request
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
So ... once again we would appear to have reached a point where everybody has said their piece — several times — and any minds that might conceivably be changed will have been changed. This seems like a good place to stop this back-and-forth, please.
A place to stop
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Wol
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
1. Long ago, quotas were put on various systems home directories. What didn't have a quota was /tmp as it was considered scratch space and might even have a dedicated disk to it. Cleanup usually happened once a year or so and you might even put a larger disk on it versus other places. While home quotas and such got rarer as 'shared' computers moved more to workstations and personal laptops, the trend to put large stuff in /tmp by default in everything was going strong even in 2009 when I left that space. Most of the time I have seen someone complain about Fedora using swap /tmp it has been in academic circles where something broke at the wrong time and data was lost.
2. There are different definitions of time for research and systems administration. A system admin might think something in /tmp living longer than a day or a week needs to be moved to something more permanent. A researcher may think that the 6 months it took to gather raw data or something is temporary and might even need a year. That data may be thrown away afterwards so is 'temporary'.
Debian's /tmpest in a teapot
- not subject to sharing restrictions
- has an expectation to persist for long periods of time
- critical to the mission of the machine owner
- under such lax operator scrutiny that a Debian default change like this comes in silently (though given the first two points, maybe they really are just far on a scale of negligence)
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
2. That is not limited to academic circles
3. The main reason for this reliance is beancounter and sysadmin avoidance : /tmp and /var/tmp are sort of a gray area that allows ignoring the limits set by the people charged with provisioning storage (people that know that storage and budgets are not infinite)
4. Users are usually right in pointing they need a large scratch storage area to work
5. Beancounters and sysadmins are usually right that infinite storage does not exist in the real world, storage gets filled and BAD THINGS happen
6. A good compromise is a large scratch space where things decay before they accumulate too much (relying on humans to housekeep a scratch area is hopeless)
7. Like all real-world compromises you will find borderline cases where the default limit is set to the wrong value
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
build systems & /tmp
build systems & /tmp
build systems & /tmp
Debian's /tmpest in a teapot
> I wonder how to tackle that.
Debian's /tmpest in a teapot
Debian's /tmpest in a teapot
Metrics missing
> $ uptime
> 10:46:36 up 35 days, 19:40, 1 user, load average: 1.01, 1.06, 0.98
> $ df -hi /tmp
> Filesystem Inodes IUsed IFree IUse% Mounted on
> none 7.4M 125 7.4M 1% /tmp
Metrics missing
Metrics missing
Metrics missing
Metrics missing
b) I have not, which makes this a nice strawman. Keep up the good work.
Metrics missing
Metrics missing
Metrics missing
Metrics missing
Metrics missing
Metrics missing
Metrics missing
Metrics missing
Metrics missing
Metrics missing
>
> "Programs must not assume that any files or directories in /tmp are preserved between invocations of the program."
Metrics missing
Metrics missing
2. wait until child is initialized and starts running
3. wait until child decides to open *the right file*
Uses for /tmp
Uses for /tmp
Uses for /tmp
Uses for /tmp
> The /tmp directory must be made available for programs that require temporary files.
> Programs must not assume that any files or directories in /tmp are preserved between invocations of the program.
Uses for /tmp
Uses for /tmp
Uses for /tmp
> Citation needed.
[2] Solaris, SunOS, and HP-UX.
Uses for /tmp
Metrics missing
Metrics missing
Metrics missing
Metrics missing
Metrics missing
Metrics missing
Time-based deletion
$ df -h /tmp/
Filesystem Size Used Avail Use% Mounted on
tmpfs 32G 2.3M 32G 1% /tmp
Not a huge deal
In the bad old days systems deleted old files from /tmp, and that resulted in occasional breakage. Nothing that could not be worked around, but not a desirable state of affairs, either. Then one day I expected breakage, but it did not happen. And I was pleasantly surprised that Debian does not delete from /tmp except on rebooting; the latter is perfectly fine, because a process does not continue across reboots anyway. But it can live for longer than 10 days, and for longer than 30 days. Going back to deleting files after a while would be a regression.
Planned Debian regression
# df -hi /tmp
Filesystem Inodes IUsed IFree IUse% Mounted on
rpool/tmp 77M 647 77M 1% /tmp
neue-a4:/tmp# df -h /tmp
Filesystem Size Used Avail Use% Mounted on
rpool/tmp 39G 14M 39G 1% /tmp
# ls -ltrc|awk '{print $3}'|sort -u|wc -l
9
The last line means that there are 6 regular users with files in /tmp, as well as root and nobody, plus an empty entry from a summary line. There are files there that were created over a year ago. In this data I see no reason to delete files from /tmp between reboots.