|
|
Log in / Subscribe / Register

Debugging information for inlined functions

By Daroc Alden
July 29, 2026

LSFMM+BPF

BPF programs use BPF type format (BTF) debugging information in order to determine how to interact with functions in the kernel. Specifically, tracing a kernel function involves finding its address in the kernel's BTF section — but that doesn't work for functions that have been inlined, and therefore don't have a single, specific address. Alan Maguire wants to add information about inlined functions to BTF in order to allow them to be traced, and led a session on that topic at the 2026 Linux Storage, Filesystem, Memory-Management, and BPF Summit.

There are more than 100,000 inlined functions in the kernel, Maguire said, spread across five times as many locations. Worse, some of them are partially inlined: called normally in some places and inlined in others. That can lead to cases today where it appears that a function was traced successfully, but some invocations were not seen.

The good news is that the rest of the infrastructure for tracing inlined functions is already in place to enable kprobes, which can be attached to arbitrary locations. It is just a matter of getting the data about where functions have been inlined into a usable format, Maguire stated. "The story is actually pretty complete."

So, what is needed to store this information in BTF? The DWARF debugging format already has a way to indicate inlining information, but DWARF is also difficult to work with, and doesn't have a simple way to represent the common cases. A solution for BTF should be compact, and permit deduplication, to keep the memory overhead low, Maguire said. Ideally, inlining information could be stored in a separate section of the kernel binary, or even be distributed as a separate kernel module, so that it is not loaded until it is needed.

Concretely, Maguire proposed three new pieces of information be added to BTF. The first was inline-site-specific information about which function was inlined at each call site and how it would have been called if it had not been inlined, called the "location section". That data can't be easily deduplicated because it's specific to a given call site, so as much as possible the information should consist of pointers to data that can be deduplicated.

The second and third pieces of information he wants to add would be the pointed-to data: a "location prototype" and "location parameter". The location prototype specifies how the inlined function's arguments are represented at the call site, as a list of pointers to location parameters, which each store how to access a single function argument. In theory, the compiler could store a given function parameter at a different location for every inlined call site (of which there are 538,090); in practice, there are a limited number of ways that the compiler will transform parameters, and many functions have compatible signatures that result in the compiler making the same choices. In the current kernel, deduplicating location prototypes results in just 57,141 distinct entries referencing only 17,535 location parameter entries in total.

This means that the location sections take up most of the added data. Overall, the additions to BTF that Maguire proposes would come to approximately 11MB of additional data, or about 21 bytes per inlined call site. When pulled out into a separate kernel module and compressed, the total amount of data goes down to 3.5MB.

Maguire then went through an example of how information about a specific function would be stored. Consider this function:

    int foo(int a, void *b, bool c);

If the compiler chose to inline foo() in a way that eliminates a as unused, promotes b to be passed in a register, and determines that c is a constant, the BTF representation would be a single location section entry storing the BTF type ID of foo(), the offset of the call site from the kernel's base address in memory, and a pointer to the location-prototype entry. That entry becomes a length-tagged array of references to location-parameter entries. The first entry is null, indicating that a cannot be recovered. The second points to a location parameter entry with a flag that indicates the value is contained in a register, and then the specific register number of b. The last location-parameter entry has a different flag specifying that it is a constant, and then the value of the constant.

Alexei Starovoitov asked what order the location-section entries would be stored in, since the kernel will have to search through them to find the right entry when setting up tracing. At first, Maguire thought that sorting by the address of the call site would be best, but after looking at how the data would be used, he decided to sort the entries by function name, instead. That way, tracers that are looking for information on a particular function can find it quickly with a binary search.

There were two prerequisite problems with adding inlining information to BTF that Maguire covered. For one thing, the number of location-section entries required increasing the size of the length field of the containing structure to 24 bits. More seriously, some of the existing tooling for processing BTF did not cope well with the presence of new tags that it doesn't know how to read. That is a problem that comes up any time BTF is extended with new kinds of information, and part of how DWARF ended up becoming so brittle. To address that, Maguire added information on how to parse BTF into BTF itself. Now, any tool that can read that meta-information can correctly skip past any new tags that it does not understand.

Maguire also had to update the poke-a-hole (pahole) utility to handle properly sorting the new kinds of BTF tags. That ended up being a bit complicated, but it should be working for normal kernel builds. Andrii Nakryiko asked about how it would handle out-of-tree modules. Those modules would need resilient module BTF that could be explicitly relocated when loaded into a running kernel, Maguire explained. His design allows that, but it complicates the build somewhat. There was a bit of back-and-forth about the changes that would be needed to support out-of-tree modules elegantly, but ultimately no changes were agreed upon.

At the time of writing, Maguire's patch set has not been merged. It is clearly useful to be able to trace inlined and partially inlined functions; the question is whether it is worth the complexity and memory overhead. Time will tell.


Index entries for this article
KernelBPF
ConferenceStorage, Filesystem, Memory-Management and BPF Summit/2026


to post comments

There should be a name for this syndrome

Posted Jul 29, 2026 19:59 UTC (Wed) by roc (subscriber, #30627) [Link] (4 responses)

> That is a problem that comes up any time BTF is extended with new kinds of information, and part of how DWARF ended up becoming so brittle.

Feels like another case where the kernel developers look at the existing solution, decide it's overkill for their needs, implement something new that addresses just the use-cases they care about, then gradually over time that set of use-cases grows and they incrementally extend their solution, reintroducing many of the problems that caused them to reject that existing solution in the first place, until they reach a point where they have their own alternative solution which is better than the original solution in some ways but worse in others. And everyone who's writing tools or dealing with the ecosystem has to understand and handle both of the solutions and maybe convert between them.

See also: BPF.

There should be a name for this syndrome

Posted Jul 29, 2026 20:14 UTC (Wed) by daroc (editor, #160859) [Link] (1 responses)

I don't necessarily disagree. But that's practically traditional for the entire open-source ecosystem, really. I mean, this is all being done in the context of the Linux kernel, which also followed that trajectory — starting as a small, scratch-my-own-itch alternative to existing systems, and then growing well beyond that, gaining users, and spawning its own alternatives.

There should be a name for this syndrome

Posted Jul 31, 2026 7:08 UTC (Fri) by quotemstr (subscriber, #45331) [Link]

That's not traditional for the open source ecosystem in general. That's why DWARF is universal, except, for the reasons OP describes, in the Linux kernel.

There should be a name for this syndrome

Posted Jul 30, 2026 7:22 UTC (Thu) by taladar (subscriber, #68407) [Link]

It is more or less just Greenspun's Tenth Rule in action.

> Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of
Common Lisp.

Obviously with some modern technologies instead of the ones back then but same basic principle.

There should be a name for this syndrome

Posted Aug 3, 2026 2:47 UTC (Mon) by kkdwvd (subscriber, #179603) [Link]

If I understood you correctly, your position is that implementing a full blown DWARF parser in the kernel would have been a better outcome?

Please add support for location expressions describing the *return values* of inline functions

Posted Jul 29, 2026 20:00 UTC (Wed) by roc (subscriber, #30627) [Link] (2 responses)

DWARF doesn't support this, which is a huge missing feature. Tracing features will want to make use of this.

Please add support for location expressions describing the *return values* of inline functions

Posted Jul 30, 2026 7:05 UTC (Thu) by wahern (subscriber, #37304) [Link] (1 responses)

AFAIU, DWARF does support this, and supporting parameter value extraction for inlined calls is one reason for DWARF's state machine design. See this related article on BTF and inlining from last year: https://lwn.net/Articles/1018475/

Please add support for location expressions describing the *return values* of inline functions

Posted Jul 30, 2026 9:04 UTC (Thu) by roc (subscriber, #30627) [Link]

I don't think DWARF supports this, because this extension proposal is still open: https://dwarfstd.org/issues/221105.1-r4.html


Copyright © 2026, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds