Does it actually work?

Posted Apr 9, 2025 2:12 UTC (Wed) by intelfx (subscriber, #130118)
In reply to: Does it actually work? by pm215
Parent article: Three ways to rework the swap subsystem

> Is it possible to avoid the machine getting into that thrashing state in the first place, though?

There are several solutions to this problem in Linux that fit various definitions/stages of "possible". We have cgroup memory limits (which can be set manually), PSI metrics and userspace OOM killers (that can be more aggressive and/or proactive than the in-kernel one, having a chance to terminate the offending processes before the system becomes unusable), and MGLRU working set protection (albeit it's not certain if it actually causes more good than harm, cf. https://github.com/zen-kernel/zen-kernel/commit/856c3874). Perhaps there are other things also.

However, none of it is related to *making swap faster*. The point I was making is that once you get to the state of thrashing, no amount of "making swap faster" is going to make a system usable while it's in that state.

to post comments

Does it actually work?

Posted Apr 10, 2025 12:50 UTC (Thu) by khim (subscriber, #9252) [Link] (23 responses)

> The point I was making is that once you get to the state of thrashing, no amount of "making swap faster" is going to make a system usable while it's in that state.

But why? What makes that impossible precisely and exactly on Linux while perfectly possible on other OSes?

I have one friend who is unfortunate enough to be developing Rust app on Macbook Air. And when rustc starts using 20GiB of RAM (two processes 10GiB each) on his 8GiB MBA… compilation takes ages. What takes 5 minutes for me takes 50 minutes for him. So… 10x slowdown, sure.

Yet… Mac remains perfectly responsive (if sluggish). Chrome works, VSCode works, you can run Word and edit something in it.

Try the same thing on Linux… and everything freezes entirely.

Do you seriously want to imply that rustc has that special code somewhere in it that makes it freeze Linux and specifically and exclusively Linux only? While all other OSes chug along just fine?

Does it actually work?

Posted Apr 10, 2025 13:39 UTC (Thu) by intelfx (subscriber, #130118) [Link] (20 responses)

> But why? What makes that impossible precisely and exactly on Linux while perfectly possible on other OSes?

Because you are (either accidentally or deliberately) conflating swap performance and memory management strategy.

> Do you seriously want to imply that rustc has that special code somewhere in it that makes it freeze Linux and specifically and exclusively Linux only?

Why do *you* want to imply that? I never said anything that could be interpreted as what you just said. Please stop putting words in my mouth.

Does it actually work?

Posted Apr 10, 2025 13:51 UTC (Thu) by khim (subscriber, #9252) [Link] (12 responses)

> Because you are (either accidentally or deliberately) conflating swap performance and memory management strategy.

I guess. For me “swap performance” is a simple ratio: without swap XXX takes N seconds, with a swap XXX takes K * N seconds. And K is swap performance. There the question about how to more precisely measure XXX, obviously, but K looked like a non-ambiguous measure…

Sure, when you want to improve that ratio you would need some kind of more nuanced measures and maybe even introduce many other, lower-level, more tractable and less integral, numbers… but if we talk about “swap performance”… what else can you ever mean?

Virtual memory is “poor man's imitation of real memory” and swap is supposed to make it behave like real one… what other kind of measure may we want to talk about in the absence of any clarifications?

I even every explicitly wrote that I want to know what other natural way to measure “swap performance” is – preferably with explanation about who may want to know these numbers and why.

Does it actually work?

Posted Apr 10, 2025 13:54 UTC (Thu) by pizza (subscriber, #46) [Link] (11 responses)

> I guess. For me “swap performance” is a simple ratio: without swap XXX takes N seconds, with a swap XXX takes K * N seconds.

You're missing the point where "without swap" something in the system (possibly even what you're using) will be force-killed due to insufficient memory.

If you have enough memory for your workload, nothing will ever get paged out.

Does it actually work?

Posted Apr 10, 2025 14:11 UTC (Thu) by khim (subscriber, #9252) [Link]

> If you have enough memory for your workload, nothing will ever get paged out.

Bingo! That means that swap is, essentially, a cheap replacement for an expensive extra RAM. And it has a theoretical, never achievable goal: be as fast as said extra RAM (it's not really possible to make it faster, I suspect).

> You're missing the point where "without swap" something in the system (possibly even what you're using) will be force-killed due to insufficient memory.

Not if you would buy extra RAM. Swap is, ultimately, a replacement for an extra RAM thus it's natural to compare it's speed to speed of a computer system with said extra RAM, isn't it?

After that you can look on list of workloads that you use, attach some $$ measure to the time that you are losing and then decide whether purchase of an extra RAM is worth it.

What other “swap speed” may you want to talk about and why? What would you do with these numbers and why?

Does it actually work?

Posted Apr 11, 2025 7:46 UTC (Fri) by taladar (subscriber, #68407) [Link] (9 responses)

I think you are missing the point that something being killed is fundamentally a lot better than the system becoming so slow that it will never finish its task before someone force reboots it to get a usable system back.

Does it actually work?

Posted Apr 11, 2025 8:05 UTC (Fri) by khim (subscriber, #9252) [Link] (8 responses)

In a cloud, for server applications it's often better: fault tolerance requirements means that you have some system to cope with death of any process (if only because you have to be ready to cope with sudden death of the hardware) – and restarting work on another computer is faster than using swap.

On a desktop becoming sluggish and 10x slow is much better then killing any processes, you only want to kill them when these are truly rogue processes that wouldn't ever produce any useful results.

Satisfying both requirements is hard but that doesn't give us any useful insights into what kind of “swap speed” one may need or want if the end result is that slowdown from the use of swap is so big that the only advice about tuning of swap that one may ever get is “ensure that swap is never used”.

Does it actually work?

Posted Apr 11, 2025 14:20 UTC (Fri) by kleptog (subscriber, #1183) [Link] (7 responses)

> On a desktop becoming sluggish and 10x slow is much better then killing any processes, you only want to kill them when these are truly rogue processes that wouldn't ever produce any useful results.

Only 10x would be ok. In practice it's something that's normally almost instant taking upwards of 20 minutes and your mouse pointer doesn't even move anymore. At that point killing something like firefox (which will restore from state anyway) is the right solution. Firefox is actually a good choice, because it has many subprocesses you can probably kill without the user even noticing.

I'd settle for: if moving the mouse pointer has no visual effect within 10 seconds, trigger OOM killer. I just have no idea how to go about implementing that. Actually, I'd like it to happen as soon as mouse movement becomes choppy, because at that point all is lost already. At that point I'm only desperately trying to get the mouse to the close icon in the top-right hoping clicking it gets through before the system hangs entirely.

Does it actually work?

Posted Apr 14, 2025 8:34 UTC (Mon) by taladar (subscriber, #68407) [Link] (6 responses)

Another good indicator might be if loading files during login takes more than a second or two. I am not talking about a whole desktop worth of applications starting up but just a terminal login. At that point I almost certainly want the slow application killed as my only purpose after completing the login.

Does it actually work?

Posted Apr 14, 2025 10:39 UTC (Mon) by khim (subscriber, #9252) [Link] (5 responses)

Login? What login? It wasn't uncommon for me to not be able to login on a text console because after I type my username and hit enter I never get to the Password prompt because getty kill that session for “inactivity” before that may happen.

I have no idea how much work /bin/login does, but couldn't imagine why that would take 5 minutes or more on any system where swap works properly.

This stopped, of course, after I gave up and switched to the use of Linux in container. With browser outside of it.

Does it actually work?

Posted Apr 14, 2025 22:14 UTC (Mon) by mbunkus (subscriber, #87248) [Link] (3 responses)

I can confirm khim's description. It's spot on & not hyperbole in the least. I've found myself in the exact same situation several times over the last several years. When this happens, no login method works at all (local console, ssh, display manager if it happens to be running on that server) due to how incredibly long each. and. every. key. stroke. takes. Debugging in such a situation is completely impossible, let alone fixing anything. Even if I manage to log in getting a process list takes multiple tens of minutes. The only sane thing to do is to hard reboot & eat the file system damage & loss of data.

And that sucks so much.

BTW, all of those cases were severs, not desktop systems.

Does it actually work?

Posted Apr 18, 2025 23:32 UTC (Fri) by Jandar (subscriber, #85683) [Link] (2 responses)

> The only sane thing to do is to hard reboot & eat the file system damage & loss of data.

I use ALT+SysReq+{s,u,s,b} in that order with a few seconds between. That's my way out of the semi-regular freezes apparently caused by swapping. Every time I have to do this I cross my fingers and pray to the filesystem-gods to spare my home directory ;-)

Does it actually work?

Posted Apr 24, 2025 16:00 UTC (Thu) by paulj (subscriber, #341) [Link] (1 responses)

SysRq is disabled by default now on some common distros. The point where you remember this is the point where you need it, but can no longer login to change this, of course.

Does it actually work?

Posted Apr 24, 2025 16:45 UTC (Thu) by farnz (subscriber, #17727) [Link]

Note, too, that SysRq's functions can be individually configured. On Fedora, the default setting is to allow sync, but nothing else; this is safe, since it's a function that the kernel will eventually do in the background, but not very powerful.

Does it actually work?

Posted Apr 24, 2025 16:08 UTC (Thu) by paulj (subscriber, #341) [Link]

Speaking of containers and browsers.... I used to run a system with a slow disk - plenty of RAM, but disk was slow enough that it was easy to trigger Linux's suicidal, "off a cliff edge", swap behaviour if I ignored the initial signs of slow-down from the bloated browser. I took to running the browser using 'systemd-run --user --slice -p MemoryMax=xG <browser>'. So it runs in a cgroup that will limit its memory use, and systemd-status --user can show you exactly how much RAM the entire group of browser processes is using, and it's also to kill the entire group - without needing to first run ps to find the process.

Does it actually work?

Posted Apr 11, 2025 10:16 UTC (Fri) by Wol (subscriber, #4433) [Link] (6 responses)

> > But why? What makes that impossible precisely and exactly on Linux while perfectly possible on other OSes?

> Because you are (either accidentally or deliberately) conflating swap performance and memory management strategy.

And I may well be out of date, but people seem to be conflating swapping and paging. Am I right, in that a *swap* system *swaps processes* in and out of memory. While a *paging* system *swaps pages* in and out of memory. Which is why systems that page don't become as slow as systems that swap.

So a swapping system, faced with a large executable, will pull in the entire executable in order to run it. While a paging system will map the executable, pull in the first page to execute it, and only fault in further pages as required.

Same with working memory - a paging system will file-back the memory, and presumably flush dirty pages, and flag hot pages. While a swapping system will dump the entire memory, and read it all back, as required.

So a swapping system will be (relatively) slow because it's forever flushing and retrieving entire processes. A paging system on the other hand, will be dropping clean cold pages if required, so any program that is actively running will normally keep its hot paths in RAM without being dropped.

Cheers,
Wol

Does it actually work?

Posted Apr 11, 2025 11:48 UTC (Fri) by daroc (editor, #160859) [Link] (1 responses)

I'm not intimately familiar with Linux's swap system, but I think the distinction you're making is not one the swap system makes. On my laptop right now I have ~500MB in swap, which appears to be small sections of memory from a variety of different processes.

Does it actually work?

Posted Apr 11, 2025 12:51 UTC (Fri) by khim (subscriber, #9252) [Link]

If I remember correctly μClinux can use “a swapping system” (just like ancient Unix versions like Xenix 8086 had to do because they worked on a system without any MMU, even strange and primitive ones, like 80286).

Mainstream Linux never had support for “a swapping system”, because it was never supported any hardware where that even makes any sense.

MacOS (Classic) and Windows developed very elaborate and clever system with handles and movable pages and also never did that.

That's why that distinction that Wol talks about is more-or-less forgotten in a world of modern software.

Does it actually work?

Posted Apr 11, 2025 12:44 UTC (Fri) by khim (subscriber, #9252) [Link] (3 responses)

> And I may well be out of date, but people seem to be conflating swapping and paging.

Well… there's a reason for that. Indeed, you are a bit out of date. About 30-40 years our of date, give or take.

> Am I right, in that a *swap* system *swaps processes* in and out of memory. While a *paging* system *swaps pages* in and out of memory. Which is why systems that page don't become as slow as systems that swap.

That's true, but the problem here is that the last system that was swapping out processes that someone may have had a chance to use was probably a DOS Shell that was included in MS DOS 4 (that's year 1988), moved to supelement disk in MS DOS 6 (that's year 1993) and excluded from Windows 95 (and later version). DR-DOS also had similar thing (it was born out of Concurrent_CP/M-86 that was used swapping, after all), but I don't remember the dates.

Linux, Windows and macOS never used this strategy and thus it's very hard for anyone to even recall that ever existed.

> So a swapping system will be (relatively) slow because it's forever flushing and retrieving entire processes. A paging system on the other hand, will be dropping clean cold pages if required, so any program that is actively running will normally keep its hot paths in RAM without being dropped.

You are absolutely correct, but in a world where “a swapping system” is “something that briefly existed very long time ago and for a relatively short time”… most people wouldn't even understand what you are talking about when you would start discussing that difference. Paging needs support in hardware to be efficient, but after Intel made said support mandatory with introduction of 80386 in 1985 no one was even thinking about using “a swapping systems” for anything anywhere.

From what I understand swapping was popular before introduction of IBM/370 in 1970, but I don't know the history of computers well enough to say how exactly “a swapping systems” and “a paging system” competed 50 years ago. “A swapping systems” were already a stuff that you read in a history book and not use for work, when Linus started work on Linux, thus, of course, in the context of Linux they don't matter at all.

Does it actually work?

Posted Apr 11, 2025 14:51 UTC (Fri) by Wol (subscriber, #4433) [Link] (2 responses)

> From what I understand swapping was popular before introduction of IBM/370 in 1970, but I don't know the history of computers well enough to say how exactly “a swapping systems” and “a paging system” competed 50 years ago. “A swapping systems” were already a stuff that you read in a history book and not use for work, when Linus started work on Linux, thus, of course, in the context of Linux they don't matter at all.

I think you might be surprised then!

The original linux "swapping system" is pretty much an exact copy of the original Unix swapping system, which probably does date from your aforementioned 1970!

Do you remember back when Linus made a rather controversial change to linux swap, 2.4.10 or 2.6.10 I think - I remember it was roughly a 10 - that gave people running Vanilla Linux a massive shock? Do you remember the old saw that "swap must be at least twice ram" which most people - even back then - thought was an old wive's tale?

Because Linus ripped out all the optimisation code, and that requirement came back! If you had a swap file, and the system tried to touch just ONE BYTE of swap, if it didn't have at least twice ram available, it locked up. HARD. (Most people didn't notice, because the distros put the code back ...)

But that was sparked by Andrea Arcangeli and ?Rick van Riel? squabbling over a new approach to memory management. I don't know the end of that story, other than the "swap system" was replaced with something much better and I don't know what the new system does.

I thought all this was documented by LWN, and I've tried to find it on several occasions without success. It could have been Zak's Kernel News, which was quite big back then ...

Cheers,
Wol

Does it actually work?

Posted Apr 11, 2025 15:27 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

> The original linux "swapping system" is pretty much an exact copy of the original Unix swapping system, which probably does date from your aforementioned 1970!

Maybe in some very early versions, I'm not familiar with what was there before Linux 1.2.something

> Do you remember back when Linus made a rather controversial change to linux swap, 2.4.10 or 2.6.10 I think - I remember it was roughly a 10 - that gave people running Vanilla Linux a massive shock?

Definitely not 2.4.10 and not 2.6.10. Maybe 1.1.10? Very unlikely even back then, actually.

I know that with Linux 2.2 it was already possible to build MySQL by adding temporary swap on my puny machine with 16MiB RAM. I.e. Linux behaved example like macOS behaves today and how tutorials sell “virtual memory” idea normally: if your system doesn't have enough RAM… just add more swap and get more virtual RAM!

That's fundamentally impossible with “a swapping system” approach: if you are swapping out the whole process and there are not enough RAM to even put a single copy in memory, then that's it, you couldn't continue…

I think what you remember is different quirk that early versions of Linux had: in early versions of Linux each page had a static place in swap (but could be loaded into memory in different places) and that meant that total amount of virtual memory was equal to the size of swap and not to the size of swap+RAM. That was somewhat irritating, sure, but I don't remember any version of Linux (except μClinux) that swaps out whole processes and not individual pages.

And yes, this is exactly what puzzles me, too: I know that it was possible to use Linux while using swap and a “poor man's RAM extension” quarter century ago. And it's essentially not possible to use it like that today. But what and when have changed? Was that change to Linux or to something in userspace? I actually suspect it's change to userspace: while old twm/fvwm/mwm setup designed to support dozen of hardware X terminals over slow network was very frugal in it's copying of data and thus worked even on Linux with pretty inefficient swap implementation “new way” with Compiz and its descendants now copies megabytes (or maybe gigabytes?) of data around every time it needs to redraw a single pixel… and without pile of hacks that macOS or Windows employ that means that use modern Linux UI without enough actual, real, physical RAM is, pretty much, impossible.

> I thought all this was documented by LWN, and I've tried to find it on several occasions without success. It could have been Zak's Kernel News, which was quite big back then ...

It would be great to find out. Because I don't remember of any version of Linux that had “a swapping system”, but I sure remember that time when Linux used swap inefficiently… yet it paging, just not a very efficient one… you were able to run app that needed more memory than system has physically available on all versions of Linux that I ever used.

Does it actually work?

Posted Apr 11, 2025 15:49 UTC (Fri) by corbet (editor, #1) [Link]

Wol is thinking about the big MM switch in 2.4.10 - see this page.

Linux has never had full-process swapping, though. What is called "swap" is paging for anonymous memory.

Does it actually work?

Posted Apr 14, 2025 10:31 UTC (Mon) by farnz (subscriber, #17727) [Link] (1 responses)

IME, Windows does better at not paging out part of a process's working set in order to page something else that the process needs in, and as a consequence is better at getting useful work from a process when it starts to thrash. It then combines this with longer timeslices, so that once you're doing useful work, you get a longer time to finish it before you're swapped out.

Does it actually work?

Posted Apr 14, 2025 10:52 UTC (Mon) by khim (subscriber, #9252) [Link]

It definitely plays tricks with pushing processed to swap and back.

One way to experience “a last straw breaks the camel's back” performance issue, that plagues Linux, on Windows – is to overfill both memory and swap, too (may only do that with “permanent”, fixed size swap… that's not a default Windows setup)

When Windows may no longer “push out” some “victim” processes to swap then it starts behaving like Linux and, eventually, freezes, too.

But I have found out that adding more swap to Linux doesn't help: it may take 20 minutes to switch to a text console and then you can not login… even if there are plenty of swap left unused.