Compact formats for debugging—and more

At the 2025 Linux Plumbers Conference in Tokyo, Stephen Brennan gave a presentation on the debuginfo format, which contains the symbols and other information needed for debugging, along with some alternatives. Debuginfo files are large and, he believes, are a bit scary to customers because of the "debug" in their name. By rethinking debuginfo and the tools that use it, he hopes that free-software developers "can add new, interesting capabilities to tools that we are already using or build new interesting tools".

He works on the sustaining-engineering team at Oracle, which means that, unlike many in the room, he is mainly concerned with "fixing bugs in old released products" rather than adding new features to the latest kernel. Fixing bugs in customers' production kernels has "its own set of challenges". It has given him some insight into the needs of enterprise-kernel users, as well, which is what led him to conclude that debuginfo is not well-liked in that world.

Debuginfo

He introduced debuginfo with a few examples of using GDB on C-language "hello world" binaries built in different ways. Using the strip utility on a binary produced by GCC results in something that is not really debuggable—it lacks symbols and other information so that breakpoints cannot be set, for example. That is kind of self-inflicted; skipping the strip produces a binary with some debugging information, so breakpoints can be set, but it lacks line numbers and other data that would allow single-stepping. Normally, GDB would step by setting a breakpoint at the start of the next line of code, but it lacks the information needed to do so.

As most people already know, he said, using the "-g" option for GCC will add DWARF debugging information to the binary, which will allow "the full fat GDB debugging experience". For example, setting a breakpoint on a function will show the source file and line number rather than just an address; hitting the breakpoint will show the line of code from the source as well. In addition, arguments are shown by name with their values. GDB can also interpret various complex types, such as structures and unions.

While none of that is surprising to most, it demonstrates what he sees as the classical approach to debugging: "you get nothing until you use -g and then you get everything". Meanwhile, distributions build their packages with DWARF information but most distributions provide them as separate "debuginfo" packages because "DWARF is really big". In practice, that means regular binaries on Linux systems will have minimal debugging information, similar to his second example.

When users encounter a crash, the typical, though perhaps a bit dated, suggestion is to install a debuginfo package. Then they can run a debugger, generate a report, or send a full core dump to a support person for diagnosis. There are now some better tools, including debuginfod and various helpful crash-reporting and handling tools that he encourages people to look into. But in Brennan's experience, it often comes down to convincing customers to install debuginfo packages—something they are allergic to, at least in the enterprise-kernel world.

But he has a gripe with the name "debuginfo" for two reasons. First, it is misleading because that information can be used for more than just standard debugging with GDB. An application may have a need to unwind its own stack or examine its types at run time, for example. The term is also not specific about what kind of information it provides; it encompasses many different kinds of information about the program and its types, variables, source code, and even macro definitions. He is not proposing that some kind of alternative term be adopted, but noted that, in practice, it is simply a shorthand for DWARF information.

Introspection

There are facilities for run-time introspection of code in many high-level languages. He noted that Java has ways to inspect running code and that Python "would be a hilarious example of just how much you can do" with the ability to look at dictionaries of global and local variables, inspect everything in a class, unwind the stack, and more. Those facilities effectively use "debuginfo", but they do not call it that. C has only limited inspection options, such as backtrace(), and compilers can do some introspection for array-bounds checking and other things, but that ability "completely disappears after compile-time".

The Linux kernel is an unusual C application because it has quite a bit of introspection support. It has stack-unwinding metadata built in; it can also look up its symbols using kallsyms. Beyond that, it has a type system, BPF Type Format (BTF), available as well.

There is a spectrum of things that he considers to be debuginfo, ranging from the standard debugging information, such as DWARF, "to maybe weirder things to consider debuginfo, but that kind of fit the bill". After DWARF comes Compact Type Format (CTF) and BTF, which provide type information. SFrame and ORC are next; both are aimed at stack unwinding, but ORC is only available for x86-64. ELF symbol tables round out the standard formats.

Moving into the weirder end is kallsyms, which is used by various tools. Something that the Fedora project does, which he really likes, is to create an ELF section (.gnu_debugdata) with a compressed set of debugging symbols that can be used with GDB, the Python-based drgn debugger, and others. Two other oddball sources of debuginfo would be the last branch record (LBR) hardware feature and frame pointers.

He put up a slide (slides), shown above, that summarized the kinds of information that are contained in the different formats. Obviously, if DWARF is available, it covers pretty much everything, he said, but it is not available in some environments. For those, "you can kind of pick and choose a few of these other things on the right and piece together something that might be useful for you".

There are some "warning" signs in the slide, which he briefly touched on. For example, macro definitions are only available in DWARF if extra flags (-gdwarf -g3) are passed to GCC and BTF only has information on functions and per-CPU variables, not all variables. The latter is something he plans to work on changing.

Case studies

He then moved on to describe "a few case studies, historically, of how compact formats are useful in different Linux applications and tools" with an eye toward future ideas for using those formats. He started with the venerable ps utility, which at one time worked by reading /dev/kmem (literally, kernel memory as a file), as he learned from an LWN article about the removal of that interface. It would root around in the task structures in memory to pull out the things that it wanted to report, "which is, honestly, pretty smart, pretty cool, [and] a little bit dangerous". It required that ps have setuid-root privileges and it might need to be rebuilt any time the kernel's data structures changed. Now ps just reads information out of the /proc filesystem, which is far superior.

While it makes sense to have a dedicated interface for ps, there is other information locked inside the kernel where adding a user-space interface is not really called for, he said, which is where something like BTF or CTF could be used. The kernel's BPF developers had a problem similar to that of the older ps, in that BPF code needed to be rebuilt for the target kernel because the details of a data structure may have changed, but they solved it another way. In order to support compile once, run everywhere (CO-RE) for BPF, BTF was used to provide structure offsets to adjust the BPF for the target kernel, which eliminates the need for a compiler on the target and BPF binaries can be run on multiple kernel versions.

Another interesting user of compact debuginfo is the drgn programmable debugger, which has a focus on the kernel. It normally uses DWARF, but work has been done to enable "DWARFless debugging" with drgn. For example, kallsyms support was added in December 2024 and stack unwinding using ORC from x86-64 kernel core images (/proc/vmcore) was added in April 2025. Using CTF (which is available in Oracle kernels) is under review and Brennan is working on BTF support; he is hopeful that CTF and BTF can converge since they are already quite similar.

VMCOREINFO is a 4KB ELF section that contains only the limited amount of information about the kernel needed to construct a smaller dump file. It was not one of the entries on his list, but he thinks that VMCOREINFO is a good example of how to think about compact formats. The makedumpfile utility is used to make the small dump file from a kernel memory image in /proc/vmcore by filtering out unneeded data. It needs some basic symbol and type information, which can come from DWARF, "but that's a pain to use", especially in a kdump environment, "where there's limited memory, limited ... everything, honestly". VMCOREINFO is a tiny fraction of the size of the DWARF information.

Ideas for the future

Allowing makedumpfile to access kallsyms and BTF would provide ways to exclude more memory, such as GPU buffers, from a dump file. It would also mean that things like user-space stack memory could be added to the dump so that process stack traces could be examined. Brennan was working on adding that support when Tao Liu pointed to his patches that do much the same thing; Brennan said that they plan to work together on the feature. Another version was posted in mid-January 2026.

His final slide consisted of some "things that I was spitballing when I came up with these slides"; the intent was to try to get others thinking about better debugging tooling. For example, he noted that GDB and drgn can both produce nicely formatted output of structures in memory in a way that is useful to a developer, rather than just a hex dump. Perhaps it makes sense to add a new printk() format specifier that would use the BTF information, which could be helpful while developing and debugging. That could be extended to user space, as well, so that output from applications would use type information to pretty-print structures.

Another area that could be addressed is converting enum values to strings; it could be done via some kind of option to the compiler, which is, of course, open source, so he should simply write some code to do it, Brennan said. He also suggested combining kallsyms and BTF in the kernel as they currently carry a lot of the same information, but have separate string tables, so combining them would save space. In general, there is a lot of overlap between the two, so "we could probably combine them in interesting ways to further compact the formats".

The "perf mem" and "perf c2c" commands are used to look at memory accesses and cache sharing on a system, but their output is address-based. Instead it could use type information to say: "This is a slab address and it has this type object and I can tell you that that's the offset of this field in the kernel." That would help in finding problems like false sharing, for example.

He concluded by noting that "DWARF is really excellent, if you have it, definitely use it for debugging", but if not, there are options that can provide various pieces of that information. The compact formats can be used for more than debugging and can provide introspection features that bring those capabilities from higher-level languages to C. He believes there is a lot of room to rethink the tools that are being used in light of the availability of these other sources of information, which can lead to a more user-friendly experience.

The YouTube video of the talk is available for those interested.

[ I would like to thank our travel sponsor, the Linux Foundation, for assistance with my travel to Tokyo for Linux Plumbers Conference. ]