LWN: Comments on "The search for the correct amount of split-lock misery"

The search for the correct amount of split-lock misery

Wol — Tue, 08 Nov 2022 13:10:48 +0000

The question is simple - if the majority of apps affected by this are games, then this is a user space regression. It needs to be reverted.

Seeing as it appears to be pretty much only gaming systems that are affected, then it most definitely is a user space regression. Especially as a lot of these applications cannot be fixed by the users. And driving gamers to Windows/Apple will be seriously damaging to linux.

And lastly, isn't the (mis)feature itself actually a HARDWARE bug? "Mitigating" the bug by rendering the computer pretty much unusable for its intended purpose isn't a good idea!

Cheers,
Wol

The search for the correct amount of split-lock misery

roblucid — Tue, 08 Nov 2022 08:39:57 +0000

No, a performance bug that prevents effective multi-core operation has been detected and should be fixed.
Gamers are concerned about core scaling, the application is sabotaging itself.

The search for the correct amount of split-lock misery

joib — Sat, 29 Oct 2022 21:11:13 +0000

Not sure cgroups are appropriate either, since the kernel cannot isolate the impact of the split-locking only to the members of the cgroup. I guess you could make some mechanism like "at most N split locks per second for this cgroup, after that split lock usage will be throttled", though I'm not sure if it would actually be useful for any practical situation. The sysctl knob sounds much simpler and should cover most usecases.

The search for the correct amount of split-lock misery

nybble41 — Sat, 29 Oct 2022 20:06:43 +0000

prctl() is for things which affect the operation of the process itself. This controls whether a program making use of split locks can negatively impact the performance of the entire *system*, which puts it in a category similar to real-time priorities. As such, a privileged sysctl knob is a reasonable solution. A cgroup option could also work, and would allow access to split locks to be more precisely targeted, but it would need to require some form of elevated privilege to enable it for a regular user account.

The search for the correct amount of split-lock misery

wtarreau — Sat, 29 Oct 2022 08:40:58 +0000

I was going to ask the same. prctl() is precisely used exactly for such things, you can decide of FPU emulation, alignment checks, speculation bypass etc. The sole reason people complain precisely is because there's no wrapper allowing the behavior to change for the life time of a program, which prctl would do just fine.

The search for the correct amount of split-lock misery

Vipketsh — Sat, 29 Oct 2022 02:30:37 +0000

> by stalling abusive applications this improves the present of other applications too

In theory I understand, but do you or anyone else have any numbers to the magnitude of this slowdown on, say, a typical 4/8 core laptop ? Theoretical problems should always be trumped by practical breakage, until proven otherwise.

This is like arguing that some application causing heavy contention on a kernel lock is a DoS attack or, if not that, the application is slowing down the system and thus the kernel should prevent it from running. Clearly this is madness. How are split locks any different ?

> also about performance under unintentional abuse too.

The malicious case is all that matters. For everything else, one of two possible outcomes is possible: (i) the performance issue is fixed and so is not a problem or (ii) the performance issue is not fixed in which case the choices are "slow(er) system" and "program does not work". A slower system is *always* preferred to a non working one and that is doubly true for machines of individuals.

Also let's not forget that this case is only relevant to x86 -- the risc machines (arm & riscv) not only don't support this feature but often don't support *any* unaligned access, atomic or otherwise. With their rise in popularity there will practically not be any cases of this in opensource and with propriety apps the question is that it works or not and working is always preferred.

The search for the correct amount of split-lock misery

marcH — Fri, 28 Oct 2022 22:20:40 +0000

> The hope was that the warn mode would be sufficient to alert users to the problem, and lead to software being fixed, while [...]. By the time the 5.19 development cycle came around earlier this year, though, it seemed that little progress toward the removal of split lock operations had been made, and the denial-of-service problem was as present as before.

Wait, how does anyone know the warning had no effect? When we spot a warning and try to address it we don't tell anyone and AFAIK the kernel has no telemetry feature yet.

The search for the correct amount of split-lock misery

marcH — Fri, 28 Oct 2022 22:18:16 +0000

> If you are only ever looking to make the future good, the present will *always* suck.

Actually, by stalling abusive applications this improves the present of other applications too. Single app gaming really does not feel like a top Linux use case today. Unless you're counting ChromeOS, Android and Steam deck but then you're not administering your own system and you can count on its admins to optimize all this for you.

> Is there any known exploit using this scenario to guard against ?

This is not just about intentional abuse but also about performance under unintentional abuse too.

> recent trend that the linux kernel is to do well out-of-the-box for cloud-based deployments, and only as a second concession allow individuals

As already explained elsewhere this is not just about multi-user but about multi process in general.

The search for the correct amount of split-lock misery

mrugiero — Fri, 28 Oct 2022 21:35:44 +0000

> * "Why shouldn't my game/database/whatever consume all available resources for maximum performance?"

To be fair, split locks hurt their own performance as well. Maybe you do want your game to consume all available resources for maximum performance. Then take care of it actually providing maximum performance instead of harming it :)

The search for the correct amount of split-lock misery

mrugiero — Fri, 28 Oct 2022 21:31:16 +0000

> I would go as far to argue that throttling an application due to split locks on a single user system is always wrong.

You could, probably accurately, argue that this is always wrong if forced on the user. I.e., you could argue it should be opt-in for single user systems. But some of use would rather want to be able to report these kind of issues. Let alone being the developer of one such game and wanting a fair warning that you may be causing problems. Do you code in a multi-user system most of the time? I don't.

The search for the correct amount of split-lock misery

Vipketsh — Fri, 28 Oct 2022 11:52:58 +0000

If you are only ever looking to make the future good, the present will *always* suck.

Is there any known exploit using this scenario to guard against ? I mean, users shouldn't go off and install random "break my machine" programs or if they do DoS-ing an individual's machine isn't exactly beneficial, nor is it the worst that an evil application can do.

I see this change and behaviour as more inline with the seemingly recent trend that the linux kernel is to do well out-of-the-box for cloud-based deployments, and only as a second concession allow individuals to tweak things so their single-user laptops and systems may work reasonably. A little sad and quite backwards since cloud operators have lots of talent on hand to tweak and analyse their systems (and they do anyway) with which individuals can't hope to compete.

The search for the correct amount of split-lock misery

geert — Tue, 25 Oct 2022 10:22:29 +0000

Didn't they just needed an OS to develop a document formatting system for the AT&T patents division? ;-)

https://www.gnu.org/software/groff/manual/html_node/Histo...

The search for the correct amount of split-lock misery

Cyberax — Tue, 25 Oct 2022 06:27:59 +0000

Can this be a prctl setting? Perhaps a cgroup one?

The search for the correct amount of split-lock misery

Wol — Mon, 24 Oct 2022 21:01:19 +0000

And wasn't Unix originally written to run games? Specifically Star Trek, iirc ... :-)

Cheers,
Wol

The search for the correct amount of split-lock misery

developer122 — Mon, 24 Oct 2022 15:08:07 +0000

*Due to unix's legacy as a multiuser system.* It dates back to the early days of unix that only root can decrease a nice value, giving more cpu time.

Does this privileged tunable make any sense on a singe user system? No it does not.
Worst case scenario, the user only DoSs themselves.

The search for the correct amount of split-lock misery

developer122 — Mon, 24 Oct 2022 15:01:39 +0000

All of the above is pretty much invalidated by "on windows."
We all know that user account permissions on windows are a total mess.

The point is that high permissions should *never* be required for installing and running a game, which does nothing but:
1) download some files
2) copy them into the home directory
3) launch an executable
4) access graphics APIs/sound APIs/input APIs

NONE of those things have ever warranted elevated privledges. Wine has continued doing it's thing for over a decade now without requiring sudo even once. The use of sudo wasn't needed when the ability to trap and redirect windows syscalls was added to the linux kernel, it sure as hell shouldn't be required because of a performance knob.

The search for the correct amount of split-lock misery

implr — Mon, 24 Oct 2022 13:35:46 +0000

>The entire steam linux runtime, and indeed every wine runtime MUST NOT require any more privileges than a regular user program. For installation or during runtime.
It already does, at least for VR. On first run SteamVR will ask for sudo, which it uses to setcap CAP_SYS_NICE on its various binaries. If it detects that it can already run at nice -10, it'll skip that step.

The search for the correct amount of split-lock misery

marcH — Mon, 24 Oct 2022 06:20:46 +0000

It improves the future for maintained and future apps. For everything except orphaned apps.

The search for the correct amount of split-lock misery

marcH — Sun, 23 Oct 2022 17:27:55 +0000

> The purpose of the kernel is to serve userspace, not to tell userspace what to do.

No, the purpose of the kernel is to protect userspace from each other (some single application systems don't even have a kernel)

> If userspace wants to hurt its own performance, that's the sysadmin's* problem.

Default settings matter A LOT and it's really good to see overbusy maintainers spending so much time discussing and getting them right.

> > Affected gamers will have to set the new knob appropriately, but knowing which sysctls to tweak could be said to be part of being a true God of War.

Happy ending:
- The "bug" will not go unnoticed and new applications will avoid it
- Old applications will run too after a few minutes searching the Internet.

Very delicate trade-off perfectly found.

The search for the correct amount of split-lock misery

marcH — Sun, 23 Oct 2022 17:22:12 +0000

> Considering the long history of silicon bugs around "transactional memory" which is basically atomic operations over a much larger amount of multiple cache lines

Silicon bugs asides, this "basically" sounds weird in a discussion about split lock performance. The concept of a transaction is exactly what provides atomicity over a larger set of data _without_ any extra performance cost. Transactions make performance independent from the data size.

The search for the correct amount of split-lock misery

NYKevin — Sun, 23 Oct 2022 01:11:59 +0000

It's not a DDoS, it's just a regular DoS. There's only one system involved.

More to the point, this whole idea is clownshoes anyway. The purpose of the kernel is to serve userspace, not to tell userspace what to do. If userspace wants to hurt its own performance, that's the sysadmin's* problem. For some configurations, it might make sense to allow the sysadmin to block or restrict split-lock operations, but it should function like an rlimit, not a system-wide "block first, ask questions later" flag.

* If there is no sysadmin, that means it's a single-user system and the "problem" is even more nonsensical.

The search for the correct amount of split-lock misery

NYKevin — Sun, 23 Oct 2022 00:26:57 +0000

> One philosophical reason: If it's emulating the windows ABI to run a program that doesn't requires privileges, then it must not as for any additional privileges of it's own.
>
> Privileges during installation have never been required and are not allowed. Privileges that are repeatedly granted through permanently-lodged a setuid program that runs every time the game launches? Unthinkable.

That is a somewhat odd position to take, because on Windows:

1. Steam installs a service, which runs as "Local System." This is entirely equivalent to a daemon running as root, it's just that Windows uses different terminology.
2. It is not uncommon for Steam games to display a UAC prompt on first run. This is usually to install MSVC++ redistributable or something of that nature, rather than doing some weird custom thing, but it still technically qualifies as elevated privileges.
3. The way that Steam manages to install games without requiring a UAC prompt for every installation is kind of... terrible. Basically, it sets the library as world-writable and world-executable. That means anyone can drop a DLL in the same directory as the game and silently hijack it, because Windows considers the directory permissions to constitute a security boundary (indeed, you could just as easily replace the whole exe file). Now, you may argue that this is irrelevant, since the game is running as an ordinary user rather than a superuser, but we have to consider the https://xkcd.com/1200/ factor.

Note: (3) is my understanding of how it works, but like a lot of gamers, I do something *even more terrible than the above* and install everything into C:\Games instead of Program Files, so I don't know for sure if (3) is the default behavior or just my own damn fault. Why do I do this, if I know it's terrible? Because life is too short, that's why. It is not uncommon for older games to randomly break if they don't have full write access to "everything," so deliberately subverting the OS's security is not uncommon for PC gamers. Even if Steam is doing everything right, that doesn't help when half the users are overriding the defaults and using a less-secure configuration.

The search for the correct amount of split-lock misery

developer122 — Sat, 22 Oct 2022 23:26:53 +0000

Not possible, see comment below. This approach was also suggested during the kernel syscall emulation saga and also shot down then too.

The search for the correct amount of split-lock misery

developer122 — Sat, 22 Oct 2022 23:25:51 +0000

This is not possible.

This same suggestion *also* came up during the kernel syscall emulation work. It was suggested that instead of having the kernel reroute attempts by a program to call a windows kernel syscall, wine could simply scan for and patch any such attempts.

This is not only hideously complicated and unreliable, it also doesn't work at all with exactly the class of programs we're interested with: Games.

They frequently include a wide variety of very gnarly anti-tampering features which cannot be automatically overcome. It is for this reason wine seeks to emulate the environment around the program, and never ever attempts to reach inside it.

The search for the correct amount of split-lock misery

developer122 — Sat, 22 Oct 2022 23:21:05 +0000

This is unfortunately completely unacceptable.

The entire steam linux runtime, and indeed every wine runtime MUST NOT require any more privileges than a regular user program. For installation or during runtime.

One philosophical reason: If it's emulating the windows ABI to run a program that doesn't requires privileges, then it must not as for any additional privileges of it's own.

Privileges during installation have never been required and are not allowed. Privileges that are repeatedly granted through permanently-lodged a setuid program that runs every time the game launches? Unthinkable.

The search for the correct amount of split-lock misery

bartoc — Fri, 21 Oct 2022 23:40:25 +0000

yeah ofc, but I mean _this specific problem_

The search for the correct amount of split-lock misery

mathstuf — Fri, 21 Oct 2022 15:00:29 +0000

Yeah, it's done those kinds of things: https://arstechnica.com/gadgets/2022/10/windows-95-went-t...

TheOldNewThing blog by Raymond Chen also has lots of tales to this effect: https://devblogs.microsoft.com/oldnewthing/

The search for the correct amount of split-lock misery

bartoc — Fri, 21 Oct 2022 12:29:16 +0000

Yeah. Though actually another option might be to just have wine know how to bump around the allocations in question to avoid the split-locks in the first place. It would have to be a game-specific hack probably.

I wonder if windows has compat shims for this kinda stuff.

The search for the correct amount of split-lock misery

mathstuf — Fri, 21 Oct 2022 12:19:43 +0000

Mmm, yes, the fun problem of figuring out "what did this program do to my machine?" when trying to uninstall it. Not sure I'd like to import that particular behavior from Windows.

Incidentally, more programs supporting `/etc/prog.d/*.conf`-style configuration would be greatly appreciated :) .

> restoring misery mode when the game has exited.

ABAB problems sound fun with this. Also sounds like a maze full of fun error code paths that will never be reliably tested.

The search for the correct amount of split-lock misery

bartoc — Fri, 21 Oct 2022 12:11:34 +0000

Presumably you would do something similar to how windows programs are installed on steam, upon first launch it (sometimes) requests elevation and installs "things", in this case one such thing could be a suid program that turns off misery mode and then launches the game, restoring misery mode when the game has exited.

The search for the correct amount of split-lock misery

scientes — Fri, 21 Oct 2022 10:04:45 +0000

Considering the long history of silicon bugs around "transactional memory" which is basically atomic operations over a much larger amount of multiple cache lines, this is clearly just the "exercise for the reader" warm-up to getting that stuff correct.

Ivan Goddard said that atomic operations rather than the cache-flushing approach is not really worth it, because it costs almost the same due to cache-coherency issues. I have no thought through that however (I can't even get my head around L2 cache-coherent Zync-9000 FPGA with AXI yet...)

The search for the correct amount of split-lock misery

ballombe — Fri, 21 Oct 2022 04:08:29 +0000

Well, there should exist a mitigation that still allow God of War to run with decent performance.
The current one was made painful by design, not by necessity. This is a false dilemma.

The search for the correct amount of split-lock misery

developer122 — Thu, 20 Oct 2022 20:32:34 +0000

These same arguments came up in system call emulation and it didn't fly there either.

The search for the correct amount of split-lock misery

excors — Thu, 20 Oct 2022 17:37:25 +0000

This article sounds a little suspicious of the merits of the gods and wars, but God of War is actually a heartwarming tale about a troubled man learning how to be a better father to his estranged son as they go on a journey together, hoping to achieve some closure by scattering his wife's ashes on the highest peak in the lands. It's surprisingly sophisticated and a lovely game.

Also you have a cool axe and kill a few gods and rip many thousands of monsters' heads off.

The search for the correct amount of split-lock misery

WolfWings — Thu, 20 Oct 2022 15:53:50 +0000

In this case the mitigation is preventing impact to the rest of the system, which indeed is what "1" does. It directly penalizes anything that triggers split-cacheline-atomic scenarios by forcing it to eat a 10ms pause in it's next scheduling.

"0" disables the mitigation, allowing any single application to hog the bus for it's own locks and arbitrarily DDoS the rest of the system as a result.

The search for the correct amount of split-lock misery

ericproberts — Thu, 20 Oct 2022 15:41:59 +0000

It certainly would make things more miserable.

The search for the correct amount of split-lock misery

jhoblitt — Thu, 20 Oct 2022 15:32:07 +0000

If a sysctl is being added, why not expose the ability to make split locks fatal via the knob (or even make that the default)?

The search for the correct amount of split-lock misery

JamesErik — Thu, 20 Oct 2022 14:12:21 +0000

Couldn't agree more. For me, the welcome levity was this:

'''...split-lock operations have long been frowned upon. Unfortunately, software that is malicious (or just poorly written) turns out to be remarkably indifferent to even the most severe of frowns.'''

Thank you, Jonathan. I very much admire your editorial style, your professionalism, and your consistently human touch to your work... including humor that hits the funnybone of geeks everywhere!

The search for the correct amount of split-lock misery

DouglasJM — Thu, 20 Oct 2022 13:13:41 +0000

"...proper god of war using just ordinary locks, so the game does a lot of split locking. Luck's patch had achieved its intended purpose; God of War players are now suitably miserable. "

This is the type of writing that makes reading LWN so enjoyable, tongue in cheek comments that make me smile or laugh out loud. Keep up the great work folks, I look forward to reading LWN!

The search for the correct amount of split-lock misery

tamara_schmitz — Thu, 20 Oct 2022 13:00:56 +0000

I think it's already as you described it:

0 Disables the misery mode - just warns the split lock on kernel log.
1 Enables the misery mode (this is the default) - penalizes the split lockers with intentional performance degradation.