PLT Rewriting [LWN.net]

PLT Rewriting

Posted Feb 6, 2024 23:03 UTC (Tue) by jrtc27 (subscriber, #107748)
Parent article: GNU C Library version 2.39

Firstly, one key piece of information is missing from the PLT explanation (for most architectures in common use): the PLT entry isn't rewritten, the PLT *GOT* entry is. The PLT entry (code) loads the corresponding pointer from the PLT GOT (data) and performs an indirect jump to it (which, for x86, is one instruction, an indirect jump taking a memory operand). When using lazy binding, the run-time linker sets things up such that this indirect jump eventually ends up inside the run-time linker itself, which looks up the function that the binary was trying to call and replaces the PLT GOT entry with its address such that future calls of the PLT don't need to go into the run-time linker, and finally itself jumps to the function in question for this call. Exceptions to this include SPARC and some PowerPC ABIs, but x86 follows this standard model.

Now, as far as I'm aware, PLT rewriting isn't really a security feature, it's a performance one. Lazy binding can already be disabled in two ways:

1. Set the LD_BIND_NOW environment variable
2. Link with -z now

Both methods instruct the run-time linker to fully initialise the PLT GOT when loading the binary rather than deferring most of until the first call. However, option 2 is the better one as, when combined with RELRO, it will make the PLT GOT read-only once the run-time linker has loaded the binary. Option 1, to the best of my knowledge, will incur the startup cost of the initialisation (but gain more deterministic function calls rather than penalising first use), without being able to make the PLT GOT read-only, because the region of memory in the program that can be marked as read-only once loaded is dictated by the static linker, though one could imagine an implementation that adds the PLT GOT to that list when LD_BIND_NOW is set, with the caveat that it likely won't be page sized and aligned, so the ragged ends would remain read-write.

What PLT rewriting aims to do is reduce the cost of using the PLT by getting rid of the load and indirect branch. With eager binding enabled you still have an indirect branch, but never exploit its dynamic potential, since the pointer loaded from the PLT GOT is initialised before the code is run and never changes. PLT rewriting replaces that load and indirect branch with a direct branch to the actual function. Removing the load reduces the memory footprint of the binary. Replacing the indirect branch also removes a hard-to-predict instruction, and one that is dependent on the load to have executed. This comes at the cost of not being able to share the PLT code pages of the executable or library in question with other processes that have the same object loaded, though, and does not work on systems that forbid mapping code as writable. Depending on the ISA it may also be difficult, or impossible, to do this in the general case, due to the range of operands available for direct branches. On x86 it's easy as you can have the full 64-bit absolute address in the instruction, but most architectures can't do that. You might still want to rewrite it to an indirect branch where the address is constructed directly by an instruction sequence rather than loaded from memory, which avoids the PLT GOT use, but that's rather more limited in benefit and likely quickly stops being an optimisation.

to post comments

PLT Rewriting

Posted Feb 7, 2024 0:33 UTC (Wed) by pbonzini (subscriber, #60935) [Link] (2 responses)

Wasn't the 64-bit absolute jump just announced as part of APX? I think glibc must rely on the PLT being within +/-2GiB of the GOT for the direct jump to work?

PLT Rewriting

Posted Feb 7, 2024 9:01 UTC (Wed) by maskray (subscriber, #112608) [Link] (1 responses)

I have some notes at https://maskray.me/blog/2021-09-19-all-about-procedure-li... (I frequently update articles after I've learned new things).

> otherwise, when GLIBC_TUNABLES=glibc.cpu.plt_rewrite=2 is specified on APX processors, the indirect jump instruction is rewritten to jmpabs $target (64-bit absolute jump).

PLT Rewriting

Posted Feb 7, 2024 18:21 UTC (Wed) by jrtc27 (subscriber, #107748) [Link]

Ah, I had missed that detail and made an assumption there (knowing that MOV r64, imm64 exists, but forgetting that it's a special case). Thanks both for the correction and added information.

PLT Rewriting

Posted Feb 11, 2024 17:45 UTC (Sun) by anton (subscriber, #25547) [Link] (6 responses)

Replacing the indirect branch also removes a hard-to-predict instruction, and one that is dependent on the load to have executed.

Actually an indirect branch that always branches to the same target is easy to predict perfectly: predict that it jumps to the same target as the last time. Even the 1993-vintage Pentium has a branch target buffer that is up to that task. On a core with out-of-order execution (common on smartphones, universal on laptops, PCs and servers), a correctly predicted branch means that the load is just needed for verification and does not add to the latency of other operations.

I have seen slow indirect branches on the PPC970 and Itanium 2 (both from 2002), but I doubt that many users will use glibc-2.39 and PLT rewriting on these CPUs. And I think that these CPUs have no direct jumps that have enough reach for rewriting the PLT jumps as direct jumps, so even if PLT rewriting is used for these CPUs, the slow indirect branch will still strike.

PLT Rewriting

Posted Feb 12, 2024 12:50 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (3 responses)

Doesn't it take up space in the branch predictor table though that might be better spent on other branches? (Not saying that it isn't worth it, just trying to understand the costs better.)

PLT Rewriting

Posted Feb 12, 2024 13:54 UTC (Mon) by anton (subscriber, #25547) [Link] (2 responses)

I have measured indirect (and direct) branch/call speed, so what I wrote earlier is based on direct evidence. I have not made measurements on branch predictor capacity, so what I write below is just based on what I have read:

The hardware not only predicts indirect branches, it predicts unconditional direct branches as well. That's because, even with an uop cache that caches decoded instructions, you don't want to wait until the direct branch is fetched from the uop cache and processed by the branch unit in order to decide what instructions or uops to fetch next.

Instead, the hardware puts the direct branch into the BTB, too, where it will be predicted based on the address of the instruction, not the contents. So at least as far as the BTB is concerned, there is no difference between a direct and an indirect branch.

Modern cores also use an additional indirect branch predictor that keeps more history. It's possible that a perfectly predictable indirect branch occupies capacity there, or the hardware people may have optimized this relatively common case (PLTs, practically monomorphic method calls) for better utilization of the sophisticated branch predictor (some of the articles I read on chipsncheese gave me that idea).

In any case, if I got a penny for every unsubstantiated claim of a performance advantage from a compiler maintainer or compiler supremacy advocate, ... And even when the original claim is debunked, often someone feels the urge to bring up another possible, but equally unsubstantiated performance advantage. This is not about you or jrtc27 in particular, I just have been in too many of these kinds of discussions.

PLT Rewriting

Posted Feb 12, 2024 22:19 UTC (Mon) by andresfreund (subscriber, #69562) [Link] (1 responses)

> I have measured indirect (and direct) branch/call speed, so what I wrote earlier is based on direct evidence. I have not made measurements on branch predictor capacity, so what I write below is just based on what I have read:

It's pretty easy to end up being bottlenecked by something other than what you're trying to measure if you're not careful.

> In any case, if I got a penny for every unsubstantiated claim of a performance advantage from a compiler maintainer or compiler supremacy advocate, ... And even when the original claim is debunked, often someone feels the urge to bring up another possible, but equally unsubstantiated performance advantage. This is not about you or jrtc27 in particular, I just have been in too many of these kinds of discussions.

FWIW, I've seen pretty clear production performance benefits from using -fno-plt, even though that "just" eliminates a single direct call by moving the load from the GOT inline into the callers.

PLT Rewriting

Posted Feb 14, 2024 14:06 UTC (Wed) by anton (subscriber, #25547) [Link]

Concerning the supposedly hard-to-predict and therefore slow indirect jump that always jumps to the same place, I would like to see how "bottlenecked by something other" explains the original threaded-code microbenchmark results on CPUs with BTBs (e.g., the Pentium, compare with the BTB-less 486) or the v2 results on recent CPUs, where, e.g., direct-threaded code achieves 1 indirect branch per cycle on Tiger Lake (and that despite the fact that only half of the executed indirect branches always jump to the same target in V2, while the other half varies between 5 targets; modern history-based indirect branch predictors predict that perfectly, too).

My feeling is that -fno-plt has better performance benefits than PLT rewriting, because the latter just replaces a perfectly predictable indirect jump with a direct jump (which is often equally expensive). But of course my feelings are just in the "unsubstantiated claims" department, and we need proper and reproducible measurements to get something better.

PLT Rewriting

Posted Feb 12, 2024 22:09 UTC (Mon) by andresfreund (subscriber, #69562) [Link] (1 responses)

> On a core with out-of-order execution (common on smartphones, universal on laptops, PCs and servers), a correctly predicted branch means that the load is just needed for verification and does not add to the latency of other operations.

The branch predictor has limited size, the cold cache case is still important. And even if perfectly predicted, a indirect call still requires more uops, which can be a limiting factor (due to limit of the total number of uops being tracked and due to execution ports being busy).

According to https://uops.info/table.html on Alder-Lake-P a 32bit offset JMP has a throughput of ~2, whereas a jmp to a memory location has a throughput of ~1. And the indirect jump needs 2 execution units. It's similar on other architectures.

Would be interesting to measure the performance benefits of PLT rewriting (1 direct call, 1 direct jmp) vs the performance benefits of -fno-plt (1 indirect jump).

PLT Rewriting

Posted Feb 14, 2024 13:11 UTC (Wed) by anton (subscriber, #25547) [Link]

Is the cold case important? If it is important, why is it cold? I doubt that the cold case is important. Do you have any data to support your claim of the importance of cold cases?

Anyway, in the cold case, both variants will see two branch mispredictions (one for the cold call to the PLT, one for the cold jump in the PLT), but indeed, the memory-indirect jump will also have to load the memory value (probably incurring a D-cache miss) in order to be able to know where the target address is and the misprediction penalty will be larger. Both variants probably also incur I-cache misses for both the PLT cache line and for the target. The -fno-plt variant will have only one misprediction (with the likely D-cache miss and longer misprediction penalty) and only one I-cache miss. I would not be surprised if the -fno-plt variant performed better.

Concerning the throughput of JMP (Rel32) on Alder-Lake-P, the TP column actually contains the inverse throughput, so the 2.08 says that they measured 1 JMP every 2.08 cycles (compare with, e.g., ADD_01 (R64,R64) with an inverse throughput of 0.2, i.e., 5 register-register adds every cycle). The JMP number may be due to the way they measured it (only a limited number of taken jumps per I-cache or uop-cache line have BTB predictions and are fast, see the TP number of JMP Rel8, or compare the loop and unrolled inverse throughputs of JMP M64). In any case most microarchitectures support at most one taken branch/jump per cycle.

Concerning the additional load uop, please present a real-world example where this makes a difference.