LWN: Comments on "Detecting and handling split locks"

Detecting and handling split locks

wsy — Wed, 15 Nov 2023 04:25:25 +0000

https://lwn.net/Articles/911219/
https://elixir.bootlin.com/linux/latest/source/arch/x86/k...

Kernel will give a warning with IP attached.

Detecting and handling split locks

farnz — Tue, 14 Nov 2023 21:28:54 +0000

The easiest way to catch these is to boot with split_lock_detect=fatal; then the kernel will send your process a SIGBUS signal every time you get it wrong. You can then use catch signal SIGBUS or handle signal SIGBUS stop to get GDB to take control whenever you get a SIGBUS, and your normal tools to debug the program. Note that catch signal SIGBUS sets a catchpoint, which can have all your usual nice tools for working with breakpoints applied to it (e.g. automatic commands run when you hit the catchpoint, or conditional behaviour such as continuing anyway for a SIGBUS that isn't related to a split lock.

Detecting and handling split locks

d3x0r — Tue, 14 Nov 2023 20:10:21 +0000

Is there any support for GDB to be able to pause on these? I have some code apparently that needs alignment, but I don't know where; most structures are typically aligned so setting the break on the exchange will catch thousands of operations otherwise. Is there a signal generated? when I do a search for 'split lock detection gdb' I get articles about one or the other, but not both topics together.

Detecting and handling split locks

pabs — Sat, 28 May 2022 02:44:26 +0000

Is the 32-bit userland on 64-bit kernel issue something that could be handled by the kernel or a hardware limitation?

Detecting and handling split locks

plugwash — Sat, 28 May 2022 02:06:22 +0000

Unaligned access on arm is a horrible mess.

On modern arm32, regular loads and stores are unaligned safe, but many other instructions (notably ldrd and vldr) are not. 32-bit kernels will trap and emulate unaligned accesses by default, but 64-bit kernels running 32-bit applications will not.

Detecting and handling split locks

plugwash — Mon, 05 Jul 2021 02:00:22 +0000

It's not that simple.

arm can handle unaligned accesses on regular ldr and str instructions (and I think also ldrh and strh but i'm not 100% sure on that) since armv6. There are other instructions the hardware can't handle unaligned accesses on though (ldrd and vldr spring to mind as ones I've had trouble with in real code). Some of these will be emulated by 32-bit kernels but not by 64-bit kernels leading to programs crashing when run on 64-bit kernels.

While I haven't checked I certainly wouldn't expect fancy stuff like atomics to properly support unaligned access on arm.

Detecting and handling split locks

erkki — Wed, 22 Jan 2020 16:22:41 +0000

I benchmarked this and split locks cause a 200 times slow down for single threaded workloads: https://rigtorp.se/split-locks/

Improving portability advice

meuh — Fri, 14 Jun 2019 08:03:20 +0000

> This is actually bad advice, here’s my suggestion for improvement,
> and why this is important:
>
>> When writing code, assume the target architecture has natural alignment
>> requirements, but be prepared for less alignment.

The original advice should be interpreted as you have to explicitly pad your structure so that members are naturally aligned.

> [...] the proper fix is to make all struct padding explicit by adding explicitly unused members to the structures.

Yes, this is the rule you have to follow when designing a data structure to be exchanged between kernel and userspace (or between kernel and firmware/hardware) so that the structure as a fixed size regarding the architecture (and padding can be initialized and/or inspected for unknown bit set).

Improving portability advice

fotoba — Tue, 11 Jun 2019 15:02:02 +0000

If pointers will be huge or far
not near or not specified
https://www.geeksforgeeks.org/what-are-near-far-and-huge-...

This problem will not happend as I read well waht happend

Improving portability advice

k8to — Tue, 11 Jun 2019 13:26:57 +0000

The typical userland allocators on Solaris & Windows also behave this way. It's not very surprising.

On Linux my memory is a bit muddied between the default glibc allocator, tcmalloc, jemalloc etc. so I'm less confident, but I would expect the same.

Detecting and handling split locks

flussence — Mon, 10 Jun 2019 20:26:48 +0000

If x86-64 was to support a full x86-32 feature set anyway (which it had to, to avoid the fate of Itanic), it would probably be more costly to selectively gate off all those features in long mode than to let them through.

Improving portability advice

bcopeland — Mon, 10 Jun 2019 13:17:46 +0000

If the arrays are dynamically allocated, slab allocator (for example) might do this. I once looked at a case that was allocating a bunch of individual pointers that all got rounded up from 8 to 32 bytes, meaning 75% of each allocation was unused. It makes sense that the allocators work that way, but if you don't know that and do this sort of thing hundreds of times, and especially if the memory being allocated happens to be something like 2**n + small_amount, you can accidentally waste a lot of memory.

Detecting and handling split locks

dmiller — Mon, 10 Jun 2019 10:03:06 +0000

You could get a deadlock pretty easily this way, and that's one thing that hardware engineers are going to be very careful to try and avoid. Imagine processor 1 has virtual address 0xA0000 mapped to physical address 0x1000 and 0xA1000 mapped to 0x2000, while processor 2 has 0xA0000 mapped to 0x2000 and 0xA1000 mapped to 0x1000. Now say that both do a 4-byte atomic operation to 0xA0FFF at about the same time. This is equivalent to the "Dining Philosophers Problem" for n=2 which requires a good amount of complexity to do safely.

Detecting and handling split locks

camhusmj38 — Mon, 10 Jun 2019 07:10:38 +0000

Lots of things should have been tidied up at that time. Presumably, AMD thought that might make porting more difficult.

Detecting and handling split locks

epa — Mon, 10 Jun 2019 06:20:55 +0000

That doesn’t explain why it wasn’t quietly dropped with the move to x86-64.

Detecting and handling split locks

jcm — Mon, 10 Jun 2019 02:38:26 +0000

Indeed. This is literally what the LOCK instruction prefix byte means in x86.

Detecting and handling split locks

khim — Sun, 09 Jun 2019 22:53:31 +0000

TSX is very recent invention and it's implementation is very tricky and buggy. First generations of CPUs where it was implemented had it disable on almost all models in the end.

I'm not surprised they haven't used it to implement something they needed 20 years ago.

Improving portability advice

itvirta — Sun, 09 Jun 2019 18:44:26 +0000

What exactly pads your structures to powers of two in arrays?

Improving portability advice

mirabilos — Sun, 09 Jun 2019 11:53:28 +0000

Oh, so ESR wrote a detailled article about it. I only knew from OpenBSD commits and figuring out the rest myself.

Yes! I’ve significantly reduced the in-memory size of several structures (which are then used in arrays, which pads them to powers of 2 at the end too, so going down from 36 to 32 actually reduces from 64 to 32) in some codebasēs.

Improving portability advice

marcH — Sun, 09 Jun 2019 04:29:29 +0000

I liked this BTW, do you approve? http://www.catb.org/esr/structure-packing/

indeed

gus3 — Sun, 09 Jun 2019 01:19:11 +0000

I agree with you wholeheartedly. In fact, you inspired me to renew my subscription, which was set to end next month.

Detecting and handling split locks

luto — Sat, 08 Jun 2019 23:43:50 +0000

Intel *has* that feature: TSX. I’m a bit surprised that microcode doesn’t emulate split locks using TSX.

Improving portability advice

scientes — Sat, 08 Jun 2019 23:14:58 +0000

I did that for SocketCAN frames. a2f11835994ed5bcd6d66c7205947cc482231b08

Detecting and handling split locks

pbonzini — Sat, 08 Jun 2019 22:32:49 +0000

Because back in the 386 (and earlier) days there was no cache and atomic operations were done by literally locking the whole bus, so for the processor it was simpler not to do any check on the alignment after all it did not do any check on non-locked accesses). And when a cache was added, backwards compatibility meant you couldn't drop the feature.

Improving portability advice

mirabilos — Sat, 08 Jun 2019 20:54:05 +0000

The kernel documentation currently states:

> When writing code, assume the target architecture has natural alignment
> requirements.

This is actually bad advice, here’s my suggestion for improvement,
and why this is important:

> When writing code, assume the target architecture has natural alignment
> requirements, but be prepared for less alignment.

On m68k, even dword (32-bit) quantities are only aligned for 16 bits,
and there has been software that implicitly relies on the implicit
padding added by natural-alignment architectures if you have a
struct { short; int; }; which is not added on m68k.

We’ve been fixing some of them by changing some assertions (against
accidental struct growth) from sizeof(struct) == value to <= value,
but the proper fix is to make all struct padding explicit by adding
explicitly unused members to the structures. This also helps visua‐
lising the problems of uninitialised padding when comparing them.

I’d appreciate if someone with experuence in the LKML community
could pick this up and discuss and fix it there.

Detecting and handling split locks

mokki — Sat, 08 Jun 2019 19:39:38 +0000

Most likely the easiest way to avoid deadlocks for this rare case is to take bus lock

Detecting and handling split locks

lkundrak — Sat, 08 Jun 2019 17:57:41 +0000

Thanks for this article. It did an excellent job at being an easy read even for someone who happens to be rather ignorant about the multi-processor synchronization or cache architectures. Quality writing of this sort makes it easy to recommend subscribing to LWN to friends.

Detecting and handling split locks

andresfreund — Sat, 08 Jun 2019 17:55:04 +0000

> The Intel architecture allows misaligned memory access in situations where other architectures (such as ARM or RISC-V) do not. One such situation is atomic operations on memory that is split across two cache lines. This feature is largely unknown, but its impact is even less so.

Does anybody have any insight why support for such "split" atomic accesses was added? Seems like it's an obvious source of complexity - without IMO having a lot of practical benefit? Allowing un-aligned memory accesses is certainly useful for some work, but I don't quite see that being necessary for atomic operations?

Detecting and handling split locks

marcH — Sat, 08 Jun 2019 05:14:32 +0000

In software you can implement large number of overly complex features for a fixed and low cost and deliver bug fixes later (or... never).

Hardware is a bit different.

Detecting and handling split locks

khim — Sat, 08 Jun 2019 03:11:38 +0000

Because it would need to introduce the whole new protocol for that. Basically adding *lots* of complexity to the common and very time-critical pass for the sake of something which, in property designed system, doesn't happen at all.

Detecting and handling split locks

quotemstr — Sat, 08 Jun 2019 01:12:00 +0000

Why doesn't the processor just lock both cache lines in the case that an atomic operation spans a cache line? I don't see why it would have to lock the whole bus.

Worst case scenario ?

mpr22 — Fri, 07 Jun 2019 23:41:32 +0000

I wasn't even thinking that far, merely as far as "if your software system does this, then something is wrong with it and you should do something about that".

Detecting and handling split locks

daney — Fri, 07 Jun 2019 22:58:14 +0000

They are not talking about "normal" memory accesses. The fun seems to happen on unaligned "atomic" accesses. IIRC, ARM does not allow for cache line splitting on atomic operations nor on load-exclusive/store-exclusive.

Detecting and handling split locks

scientes — Fri, 07 Jun 2019 22:06:40 +0000

ARM can handle unaligned memory accesses since ARMv6.

Worst case scenario ?

JoeBuck — Fri, 07 Jun 2019 21:24:49 +0000

"Identify the saboteur" ... if this approach can be used deliberately as an attack that could allow unprivileged code in a container to bring the host to its knees.

Worst case scenario ?

mpr22 — Fri, 07 Jun 2019 20:58:06 +0000

At that point I submit that you need to either recruit better programmers or identify the saboteur :)

Worst case scenario ?

meuh — Fri, 07 Jun 2019 20:50:57 +0000

What about an unaligned atomic operation across two consecutive unmapped pages, which will be mapped from physical memory belonging to two different NUMA nodes ?

Detecting and handling split locks

jcm — Fri, 07 Jun 2019 19:02:01 +0000

Intel calls these "data chunk split misaligned" loads and stores. When they occur, the (e.g.) load buffer will hold them until the ROB indicates that the execution results would be committable to architectural state. There's a couple of fun patents if you search "data chunk split misaligned" in your favorite search engine.