LWN: Comments on "Developers split over split-lock detection"

Cloud providers -- lead the pack!

oldtomas — Fri, 20 Dec 2019 14:16:33 +0000

Since the most interested parties are the cloud providers... why not let them lead the pack and test this feature until BIOSes and other funny firmware stabilizes?

After that, it may be time to declare it the default!

IOW: why should everyone else play guinea pigs to the benefit of the cloud providers?

Developers split over split-lock detection

GoodMirek — Fri, 13 Dec 2019 08:47:07 +0000

Maybe, but the issue could be caused by a faulty software or a hacked VM. Contract termination seems as a last resort, e.g. in case the customer does not cooperate. However, in the meantime there can be a huge performance issue caused to many others.
E.g. if I run 100 faulty worker nodes, spread over 100 random hosts, each host having 48 CPU cores (96 vCPU's due to hyperthreading/SMT), that looks like a potentially significant issue.

Developers split over split-lock detection

bof — Fri, 13 Dec 2019 08:26:16 +0000

I'm sure if that ever becomes a thing, there's performance counters to read and recognize split lock membus locking, and automatic termination of contracts afterwards...

Developers split over split-lock detection

GoodMirek — Fri, 13 Dec 2019 06:51:09 +0000

It actually sounds like DoSing a cloud provider this way would be an easy task, requiring to continuously create split-locks just on a single thread of a hyperthreaded core. Would that affect just one NUMA node or all of them?

Developers split over split-lock detection

dps — Thu, 12 Dec 2019 16:03:33 +0000

A lot of commercial environments are agile but many of them build up vast piles of low priority bugs which never get fixed. In many agile environments is hard to fix code which sorts vast buffers with bubble sort because the code works. Customers can't see the source code and therefore won't be aware of this problem.

Tasks like fixing spelling mistakes, especially those in comments, become almost impossible.

The only way agile environments will be made to fix split locks is for them to cause something very bad to happen somewhere they care about. A demonstration that an allegedly safe interface to root allowed anybody to read /etc/shadow was required to get time to fix it.

As you might expect no customers where told anything and no fix issued. I will neither confirm nor deny any suggestions about the company or product involved.

Developers split over split-lock detection

corbet — Mon, 09 Dec 2019 23:50:17 +0000

The signal only affects the process creating the split lock. The slowdown affects the entire system...

Developers split over split-lock detection

nix — Mon, 09 Dec 2019 23:46:03 +0000

> Personally, I think this feature should be enabled by default, since it's a security fix against a local DoS issue, with a compatibility knob for those running systems with trusted but buggy software.

Isn't a SIGBUS also a local DoS? Rather more of one than a 1000-clock stall?

Developers split over split-lock detection

marcH — Mon, 09 Dec 2019 20:27:18 +0000

> The nuances are not lost on Zijlstra; [...] nobody's gonna fix anything until the plug's been pulled.

...

Developers split over split-lock detection

naptastic — Mon, 09 Dec 2019 16:54:43 +0000

> The Big Kernel Lock MUST be removed, otherwise everything will be/remain broken

The nuances are not lost on Zijlstra; they've been covered elsewhere in the thread. His point is correct though: split-locks cause huge latencies, which are only going to get worse, and nobody's gonna fix anything until the plug's been pulled.

(ATI's graphics drivers still relied on the BKL despite YEARS of advanced warning that it was going away. When the BKL was fully removed, I couldn't upgrade my kernel; I think I waited 3 kernel releases before saying "screw it" and selling all my ATI/AMD gear in favor of Nvidia. Pull the plug. Break the bad things. It's the only way to make progress happen.)

Developers split over split-lock detection

marcH — Mon, 09 Dec 2019 16:34:18 +0000

> The logic behind that statement is: (1) most people use the defaults

Why write "everyone" if/when you mean "most"? Considering the main question is "how many?", it gives the impression that one hasn't really thought about it.

There are cases where the difference between "all" and "most" doesn't matter. But here it does because it takes very few early adopters to find bugs. Even fewer if they're running a data center.

> (2) there's no pressure on fixing the bug if most people aren't affected, [...] not only will broken software not be fixed (there being no pressure to do so), but also there will be pressure to never enable this feature.

How's any of that worse than not merging the code at all?

Is blocking the code itself merely used as leverage in the discussion about the default setting?

Developers split over split-lock detection

smooth1x — Mon, 09 Dec 2019 12:07:54 +0000

It would have to be optional as not every environment will be capabale of being 100% clean.

I would expect distributions to either have this turned off on Desktop installs or have their installer have a blacklist for firmware versions.

However for commercial environments would they not test first?

Surely all commercial environments are agile pipeline driven environments who break early and can fix things instantly? :->>

Developers split over split-lock detection

cesarb — Mon, 09 Dec 2019 12:07:14 +0000

The logic behind that statement is: (1) most people use the defaults; (2) there's no pressure on fixing the bug if most people aren't affected, especially if those few people affected are using a non-default setting which can be turned off without visibly breaking anything else.

That is, in the scenario where this feature is default-enabled, broken software will end up being fixed; in the scenario where this feature is default-disabled, not only will broken software not be fixed (there being no pressure to do so), but also there will be pressure to never enable this feature.

Personally, I think this feature should be enabled by default, since it's a security fix against a local DoS issue, with a compatibility knob for those running systems with trusted but buggy software.

Developers split over split-lock detection

xophos — Mon, 09 Dec 2019 11:43:50 +0000

Maybe they have enough data from previous similar issues, to be certain.
Or this is a simple hyperbole to make a point.

Developers split over split-lock detection

pbonzini — Mon, 09 Dec 2019 11:37:17 +0000

Truly ancient stuff is not going to use locked instructions at all (except perhaps XCHG which got an automatic LOCK prefix with the 386, but even that is quite unlikely)

Developers split over split-lock detection

hmh — Mon, 09 Dec 2019 03:20:31 +0000

I foresee it getting reverted about one week after it gets exposed to a wide range of users, unless it is optional. It is almost certain that enough embedded crapware (aka firmware) and x86-only commercial applications out there will trigger split locking.

Not to mention truly ancient stuff that runs under DOSEMU and friends...

So, I bet one will be able to disable this "feature" in its final form during boot, if not at any time...

Developers split over split-lock detection

Tov — Mon, 09 Dec 2019 00:31:48 +0000

So it might be a nice feature to be able to enable specifically on cloud servers. However, I am sure all desktop/laptop users with less than optimal firmware implementations and little chance of getting their firmware fixed will be thrilled by the thought of kernel panics starting to appear out of nowhere.

Furthermore people will have little sympathy of their trusty old applications suddenly being killed with SIGBUS errors due to some new standards of performance and "correctness".

Hopefully I am misunderstanding how this is supposed to work...

Developers split over split-lock detection

tux3 — Sun, 08 Dec 2019 13:42:50 +0000

I think I read cloud providers are the main force pushing for this. If it only takes a single thread busy-looping on a split-lock to stall the other 0xXX cores of your expensive hardware, you can suddenly annoy a whole lot of people for not a whole lot of money. Talk about noisy neighbors.

Developers split over split-lock detection

flussence — Sun, 08 Dec 2019 05:20:25 +0000

This seems like an extreme overkill reaction to something that, as described, doesn't sound much worse than x86 BIOS/UEFI crapware stalling ring 0 and above at bad times. Which we've dealt with for a *long* time.

So maybe there's something else going on here that Intel's not telling us.

Developers split over split-lock detection

marcH — Sun, 08 Dec 2019 01:53:02 +0000

> This feature MUST be default enabled, otherwise everything will be/remain broken

There are two statements too simplistic to be true in this single sentence:

- "Everything" implies the subset of users who would explicitly turn the feature on is exactly zero.
- "Remain" implies the default setting can never be changed in the future after the feature is merged and - for instance - some transition period.

The ability to predict the future is impressive, the ability to predict it with so much precision is even more.

Nuances maybe?