The Compact C Type Format in the GNU toolchain

I'm not talking about *now* -- and I'm also not suggesting that most binaries should be built with this. They probably shouldn't except if the distro knows it has tools that can use CTF on arbitrary programs (that's why you need a non-default compiler option, -gt, to turn CTF emission on), or unless the tool uses CTF to introspect itself. What I'm suggesting is that if you *have* built something with it, you probably did so because you *need* it -- and if you strip the CTF out of the binary there are literally zero libraries in existence that will know how to get at it unless you explicitly specify the path to the stripped-out CTF. And why would anyone compile with CTF only to render it immediately useless?

I have... painful experiences here. Back when we were converting DWARF to CTF at kernel link time and linking it into kernel modules, we had to actually *hack RPM at build time* via PATH shuffling and patching of /usr/lib/rpm/find-debuginfo.sh to even make it possible for RPM to not just strip out all non-loaded sections on the grounds that they must be unnecessary, no matter what size they were or whether RPM had never seen them before, including ripping all the CTF that we'd just gone to some lengths to link in.

To me that just seems like unwise behaviour on the part of a packaging system. RPM didn't know what that section was: why was it removing it? It might have been necessary. It *was* necessary for what we were doing, and RPM just removed it without so much as a by-your-leave. So... guess why strip(1) doesn't remove CTF? I don't want anyone who's actually using CTF to have to go through anything like that again just so they can package their own software without it being randomly broken by the packaging system.

And THAT is how CTF should be handled, too. If you *reaaaly* don't want it - ask compiler not to produce it. Most developers wouldn't care.

I peeked a bit. Will this also handle debuginfo files that are directly installed on the filesystem?

I hope that consideration of non-packaged / generic sources for debuginfo is being considered / allowed for. I've really wanted to start stripping out debuginfo from our binaries on my development team since they're so huge, but we don't create RPM or DEB files. When I propose using external debuginfo people are not happy about the developer overhead (even scripted) so we've never done it.

If it were possible to easily "register" debuginfo files created through Jenkins or some other build service without having to turn them into distro packages, then allow tools (i.e., GDB) to download them more or less invisibly when needed, that would be really nice.

Also is this limited to just debuginfo files?

The other big problem we have is cores being generated on remote systems which are using system libraries other than the local ones: in this situation we need to obtain the remote system's libc.so and other necessary system libraries. It would be really, really nice if we could register shared libraries from different systems, perhaps indexed via a hash of some kind, then have GDB automatically download them as well.

Of course, before this can be done we need to ensure that the core file contains enough information about the shared libraries to perform the lookup, which I doubt it does today, so this is requires more work in other places... however it would be good if the design of this service was able to be extended in this way in the future if/when it becomes feasible. For example in our system we use Google coredumper library rather than the kernel to dump cores and this allows us to add "notes" into the generated core file, so we could take advantage of this without Linux kernel support.

Ooh this is very nice, very nice indeed.

I wonder if it can handle more than just debuginfo... musing about having libctf automatically launch dbgserver queries for missing CTF sections now -- so people can have separated CTF if they really want *and* it is as if it were always present. Best of all worlds! For that matter they can also do both -- perhaps an option at CTF generation time which automatically emits separated CTF *if* its size passes some threshold, or some percentage threshold of the total binary size, or the .text size, or something. Of course then you'd have to arrange for the dbgserver to see it, but presumably whatever method is used for separated debuginfo would work for this too.

This is very interesting. We have a tool for NixOS that basically does except it is based on a FUSE filesystem and a content addressable packaging system: https://github.com/edolstra/dwarffs

Essentially, every version of every package has a unique hash. We build a reverse mapping from the buildids of the binaries in a package to its unique hash, and upload that metadata along with the package to the package server. We then patch GDB (and elfutils) to look in a specific directory for debug info. This directory is a FUSE filesystem, and when any tool tries to look in `.build-id/...` for the debug info -- it does a query to the package server, obtaining the unique package ID containing the symbols, and transparently installs them through the package manager. It is effectively a version of Microsoft Symbol Server, which is basically what people want, from what I can tell...

Perhaps we could replace dwarffs with something like dbgserver, however. Or integrate them so there's a single UX. We could for instance, perhaps replace the client tooling with a separate "backend" for our case, and the tools can all just work around that instead...