Placement of saved registers in a stack frame

Posted Jan 16, 2023 22:39 UTC (Mon) by jreiser (subscriber, #11027)
In reply to: Fedora's tempest in a stack frame by mcatanzaro
Parent article: Fedora's tempest in a stack frame

One of the issues identified is (in many cases) the use of +d8(%rsp) addressing without frame pointer, versus -d32(%rbp) addressing with a frame pointer. +d8(%rsp) costs one byte for the 8-bit displacement plus one byte for s-i-b addressing mode to allow the stack pointer %rsp as a base register. -d32(%rbp) costs 4 bytes for the 32-bit displacement.

Why would using a frame pointer cause so many more 32-bit displacements? Because of the placement of saved registers in the stack frame. On x86_64, traditional entry to a subroutine which saves all 6 saved registers (with a frame pointer) looks like

push %rbp; mov %rsp,%rbp
push %r15; push %r14; push %r13; push %r12; push %rbx

in which 40 bytes of the 128-byte range for 8-bit displacement beneath the frame pointer %rbp are consumed by saved registers, leaving only 88 bytes for programmer-defined values. This contrasts to the 128 bytes available for eight-bit positive displacements from %rsp.

Changing the entry sequence to

push %r15; push %r14; push %r13; push %r12; push %rbx
push %rbp; mov %rsp,%rbp

would move the saved registers to the other side of the frame pointer %rbp, which would recoup the 40 bytes as long as there were at most 80 bytes of incoming on-stack actual arguments (10 or fewer pointers, etc.) The return address has a different position relative to the frame pointer, but it can be found by disassembling the entry code, looking for push %rbp.

As observed in the blog, nothing except intellectual inertia by the compiler prevents the use +d8(%rsp) with a frame pointer, too.

Placement of saved registers in a stack frame

Posted Jan 18, 2023 10:22 UTC (Wed) by jengelh (guest, #33263) [Link] (2 responses)

>nothing except intellectual inertia by the compiler prevents

Have you considered ABI? This looks like a change of calling convention.

Placement of saved registers in a stack frame

Posted Jan 18, 2023 13:54 UTC (Wed) by dezgeg (subscriber, #92243) [Link]

Changing location of rbp on the stack would be an ABI break (for anyone trying to unwind the stack), yes. But the last sentence is (if I understood right) an independent optimization of having the compiler still generate local variable accesses via rsp-relative offsets when beneficial, instead of always through the frame pointer (when frame pointers are enabled, of course).

Placement of saved registers in a stack frame

Posted Jan 19, 2023 1:59 UTC (Thu) by ncm (guest, #165) [Link]

There is no suggestion to change the calling convention.

The compiler is free to push the frame pointer and then ignore it in subsequent stack accesses and use the same addressing mode, relative to the stack pointer, as it does now with no frame pointer. It also does not need to pop the value it pushed, but just increment the stack pointer past it.