Fedora's tempest in a stack frame
A stack frame contains information relevant to a function call in a running program; this includes the return address, local variables, and saved registers. A frame pointer is a CPU register pointing to the base of the current stack frame; it can be useful for properly clearing the stack frame when returning from a function. Compilers, though, are well aware of the space they allocate on the stack and do not actually need a frame pointer to manage stack frames properly. It is, thus, common to build programs without the use of frame pointers.
Other code, though, lacks insights into the compiler's internal state and may struggle to interpret a stack's contents properly. As a result, code built without frame pointers can be harder to profile or to obtain useful crash dumps from. Both debugging and performance-optimization work are made much easier if frame pointers are present.
Back in June 2022, a Fedora system-wide change proposal, then aimed at the Fedora 37 release, called for the enabling of frame pointers for all binaries built for the distribution. While developers can build a specific program with frame pointers relatively easily when the need arises, the proposal stated, it is often necessary to rebuild a long list of libraries as well; that makes the job rather more difficult. Some types of profiling need to be done on a system-wide basis to be useful; that can only be done if the whole system has frame pointers enabled. Simply building the distribution that way to begin with would make life easier for developers and, it was argued, set the stage for many performance improvements in the future.
There is, of course, a cost to enabling frame pointers. Each function call must save the current frame pointer to the stack, slightly increasing the cost of that call and the size of the code. The frame pointer also occupies a general-purpose register, increasing register spills and slowing down code that might put the register to better use. Avoiding these costs is the main reasons why distributions are built without frame pointers in the first place.
The proposal resulted in an extensive discussion on both the mailing list and the associated Fedora Engineering Steering Council (FESCo) ticket. As would be expected, the primary objection was the performance cost, some of which was benchmarked on the Fedora wiki. Compiling the kernel turned out to be 2.4% slower, and a Blender test case regressed by 2%. The worst case appears to be Python programs, which can see as much as a 10% performance hit. To many, these costs were seen as unacceptable.
The immediate reaction was enough to cause the proposed changed to be deferred to Fedora 38. But the discussion went on. The proponents of the change were undeterred by any potential performance loss; for example, Andrii Nakryiko argued:
Even if we lose 1-2% of benchmark performance, what we gain instead is lots of systems enthusiasts that now can do ad-hoc profiling and investigations, without the need to recompile the system or application in special configuration. It's extremely underappreciated how big of a barrier it is for people contribution towards performance and efficiency, if even trying to do anything useful in this space takes tons of effort. If we care about the community to contribute, we need to make it simple for that community to observe applications.
He added that Meta builds its internal applications with frame pointers
enabled because the cost as seen as being more than justified by the
benefits. Brendan Gregg described the
benefits seen from frame pointers at Netflix, and Ian Rogers told a similar
story about the experience at Google. On the other hand, the
developers in Red Hat's platform tools team, represented by
Florian Weimer, remained steadfastly opposed to enabling frame
pointers. Neal Gompa, instead, supported the
change but worried that Fedora would be "roasted
" on certain
benchmark-oriented web sites for reducing performance across the entire
distribution.
The change was discussed at the
November 15 FESCo meeting (the IRC log is
available) and the proposal was ultimately rejected. That led to some
unhappiness among proponents of the change, who were unwilling to let the
idea go, despite Kevin Kofler's admonition
that "the toolchain people are the most qualified experts on the
topic
" and that it was time to move on. Michael Catanzaro complained
that he could "no longer trust the toolchain developers to make rational
decisions regarding real-world performance impact due to their handling of
this issue
". But even Catanzaro said that it was time to move on.
But that is not what happened. On January 3, FESCo held another meeting in which an entirely new ticket calling for a "revote" on the frame-pointer proposal was discussed; this was the first that most people had heard that the idea was back. The new ticket had been opened six days prior — on December 28 — by Gompa; it was approved by a vote of six to one (with one abstention). So, as of this writing, the plan is to enable frame pointers for the Fedora 38 release, which is currently scheduled for a late-April release.
There appear to be a few factors that brought about FESCo's change of heart, starting with the ongoing requests from the proposal's proponents. While this whole discussion was going on, FESCo approved another build-option change (setting _FORTIFY_SOURCE=3 for increased security). That change also has a performance cost (though how much is not clear); the fact that it won approval while frame pointers did not was seen by some as the result of a double standard. The proposal was also modified to exempt Python — which is where the worse performance costs were seen — from the use of frame pointers. All of that, it seems, was enough to convince most FESCo members to support the idea.
As might be imagined; not all participants in the discussion saw things the
same way. There were complaints about the short notice for the new ticket,
which was also opened in the middle of the holiday break, and that
participants in the discussion on the older ticket were not notified of the
new one. Vitaly Zaitsev said
that the proposal came back "because big corporations weren't
happy with the results
" and called it a bad precedent; Kofler called the process
"deliberately rigged
". Fedora leader Matthew Miller disputed that
claim, but did acknowledge that things could have been done better:
I agree with your earlier post that this did not have enough visibility, enough notice, or enough time. I was certainly taken by surprise, and I was trying to keep an eye on this one in particular. [...] BUT, I do not think it was done with malice, as "deliberately rigged" implies. I don't see that at all -- I see excitement and interest in moving forward on something that already has taken a long time, and looming practical deadlines.
The rushed timing for the second vote, it seems, was done so that a result could be had in time for the imminent mass rebuild. It obviously makes sense to make a change like that before rebuilding the entire distribution from source rather than after. But even some of the participants in the wider discussion who understand that point felt that the process had not worked well.
There is still time for FESCo to revisit (again) the decision, should that
seem warranted, but that seems unlikely. As FESCo member Zbigniew
Jędrzejewski-Szmek pointed out,
much of the discussion has already moved on to the technical details of how
to manage the change. Thus, Fedora 38 will probably be a little
slower than its predecessors, but hopefully the performance improvements
that will follow from this change in
future releases will more than make up for that cost.