Enabling debuginfod for Fedora by default

In early April, Fedora program manager Ben Cotton posted a proposal to use the distribution's debuginfod servers by default in Fedora 35. This feature would help developers who are trying to debug or trace their programs using various tools, but who are lacking the source code and debugging symbols needed. The servers can provide that data directly to the tools as needed, but there are some security and privacy concerns to work through before turning the feature on by default.

The required source code and debugging information is available for Fedora already, of course, but it lives in debuginfo and src RPMs that must be installed to be used by the tools. Those RPM files are quite large and generally cover much more than the symbols and source for a single file that a user might want to look at in a tracing or debugging session. In addition, installing them via DNF requires root privileges, which may not be available to the user. Grabbing just the pieces needed, at the right time and without extra privileges, is a highly useful service that the debuginfod feature can provide.

An October 2019 Red Hat blog post describes debuginfod and notes that it is a new feature coming in the elfutils tools. The idea is that the Build.ID hash that gets stored in object files by GCC and LLVM can be used to identify which version of the symbols and source code correspond to the object. Build.ID support was added for Fedora 8 in 2007. The Build.ID directly identifies the debugging symbols for the object file; the source code path is also stored in the object file, which can be used to identify (thus serve) the right source file package as well.

In October 2020, Frank Ch. Eigler posted a note to the fedora-devel mailing list about the debuginfod feature and a test server that had been set up. He wanted to gauge if there was interest in having Fedora set up its own:

The problem is that Fedora itself doesn't run a server, and our test server can afford to carry only a subset of debuginfo/debugsource rpms & architectures. So, fedora developers / users cannot get at all the info, or from an official source. I wonder if it's time to get one set up. If there is interest, I'd be happy to start discussing logistics with fedora infrastructure folks.

The overall reaction was positive, which led to the proposal to set the DEBUGINFOD_URLS environment variable in the Fedora 35 release so that the new Fedora debuginfod servers would be used when debugging information is needed. As Eigler, who is the owner of the feature proposal, pointed out, there is a staging server available at debuginfod.stg.fedoraproject.org, so Fedora users can already try things out:

[...] it already is there in most tools, try it.

% export DEBUGINFOD_URLS=https://debuginfod.stg.fedoraproject.org/

and shortly all F32+ packages/versions/architectures will be debuggable that way.

Once the staging version of the server has been populated and is working, the data will be moved to the production server at debuginfod.fedoraproject.org, which is where DEBUGINFOD_URLS would be pointed for Fedora 35. Overall, using the staging server has been working well, though Michael Catanzaro was concerned about its performance:

I started testing the staging server yesterday. It seems a little slow -- I worry for anyone debugging anything that links to WebKit, which on the desktop is a lot -- but otherwise it works *very* well. I'm impressed. Very nice.

Owen Taylor noted that the feature works for Flatpak applications as well. Eigler explained that some of the performance problems Catanzaro saw could have been caused by the processing that the staging server had been doing to pull in the RPM files from the Fedora Koji build system. But there are some inherent delays in the process, though caching may help:

In the case where a new file is sought from inside some large rpm, yeah there will be some inherent latency in decompressing it (CPU bound), saving the selected file to tmp disk (RAM/storage bound) and then sending you the file (network bound).

At the server, there are some prefetching/caching options available to take advantage of the spacious servers kindly provided by fedora-infra.

There is also aggressive caching on the client side, with an adjustable week-long retention for files.

David Malcolm wondered about Python programs that are currently part of the debuginfo packages for Python. Back in 2010, for the Fedora 13 release, he had added these programs as part of a feature for easier Python debugging with GDB. But he was worried about having them automatically installed:

I'm much more nervous about arbitrary python scripts being supplied over this service, as the barrier to entry for bad guys to do Bad Things would be so much lower as compared to malformed DWARF, so perhaps if people want the .py files, they have to install the debuginfo package in the traditional way? (It's still .py files from rpm payloads, but having them autodownloaded with little user involvement seems troublesome as compared to manually installing debuginfo rpms).

Eigler said that debuginfod has no way to serve standalone .py programs, but that a way could be added if it was deemed important. He noted that the provenance of the data is meant to provide the security:

[...] I'm not sure bad DWARF is inherently safer than bad Python. Nevertheless, all this is based on the proposition that the files are coming from a generally trusted build system, serving trusted artifacts maintained by trusted individuals. If malevolent enough files get in there, the service can be degraded and/or clients can be affected.

In a similar vein, Björn Persson pointed out that the proposal was lacking answers for a variety of security and privacy questions. The proposal does mention some privacy concerns raised by Debian users after a debuginfod service for Debian was announced in February, but also noted that openSUSE Tumbleweed enabled the feature by default (in February 2020) "and we have heard of no controversy". The Build.ID and source file name being debugged are leaked to the server if the local system lacks the proper files; Debian added an install-time query to decide whether to enable the feature.

Beyond the privacy issues, Persson also wanted the proposal to address the security implications of the server: what kinds of attacks might be possible, how are the files received from the server verified, and what kind of signing and authentication is done for the files. Eigler thanked him for the questions and added a "Security FAQ" section to the online version of the proposal with initial answers to Persson's questions.

As might be guessed, the scope of attacks depends in large part on the tool that is consuming the data, its privileges and robustness, as well as the privileges of the program being debugged. There is no special verification being added for the information provided from the server; as Eigler's response about Python noted, it all comes down to trusting the distribution—and trusting HTTPS. As the FAQ puts it:

Debuginfod servers provide the verbatim contents of the verbatim distro archives, and transmit them securely across HTTPS. There is no per-file signing infrastructure in Fedora, and debuginfod doesn't add one. Thus there is no mechanism to manually verify these files, beyond downloading a corresponding signed archive out-of-band and comparing. The client side code will be taking some rudimentary measures with file permissions to reduce risk of accidental change. In principle, if the received files were tampered with, then the same tamperers could mess with the user's consumer tools and/or take over the account.

There are multiple benefits to the feature, including making it easier for users who may not be able to install the debuginfo packages using DNF and reducing the amount of data that needs to be sent and stored in comparison to the, generally rather large, debuginfo packages. The debuginfod server only sends the pieces needed for the user's immediate debugging task, rather than all of the associated source code and debugging information for the entire package.

While there may be a need to add an explicit option for administrators to disable (or opt-out of) consulting the debuginfod servers, it seems highly likely that Fedora 35 will default to using them. In the meantime, users can already start consulting the server by setting their DEBUGINFOD_URLS environment variable appropriately.