Adding package information to ELF objects

While it is often relatively straightforward to determine what package provided a binary that is misbehaving—crashing for instance—on Fedora and other Linux distributions, there are situations where it may be harder to do so. A feature recently proposed for Fedora 36—currently scheduled for the end of April 2022—would embed information into the binaries themselves to show where they came from. It is part of a multi-distribution effort to standardize how this information is stored in the binaries (and the libraries they use) to assist crash-reporting and other tools.

On October 25, Fedora program manager Ben Cotton posted the proposal to the Fedora devel mailing list; it is also available on the wiki. The basic idea is that each ELF object that gets created for an RPM package will get a .note.package ELF section added to it. That section will contain a JSON-formatted description of exactly which RPM it was distributed with. So those binaries will contain information that can tie them directly to the package, even in the absence of RPM metadata on the system.

The facility would be used by the systemd-coredump utility to log package versions when crashes occur. For regular Fedora systems, which normally have the RPM metadata available, there is no large advantage. But for other situations where Fedora-created binaries might be run—and crash—this mechanism would allow administrators and tools to recognize where exactly the binary came from.

The feature was originally proposed back in April for Fedora 35, but was rejected by the Fedora Engineering Steering Committee (FESCo) with an explicit invitation to resubmit it with "a more detailed and understandable 'Benefit to Fedora' section". So the two feature owners, Zbigniew Jędrzejewski-Szmek and Lennart Poettering, added more information to the proposal for the resubmission. The new "Benefit to Fedora" section details why it makes sense for the distribution:

A simple and reliable way to gather information about package versions of programs is added. It enhances, instead of replacing, the existing mechanisms. It is particularly useful when reporting crash dumps, but can also be used for image introspection and [forensics], license checks and version scans on containers, etc.

If we adopt this in Fedora, Fedora leads the way on implementing the standard. Fedora binaries used in any context can be easily recognized. Fedora binaries provide a better basis to build things.

If other distros adopt this, we can introspect and report on those binaries easily within the Fedora context. For example, when somebody is using a container with some programs that originate in the Debian ecosystem, we would be able to identify those programs without tools like `apt` or `dpkg-query`. Core dump [analysis] executed in the Fedora host can easily provide useful information about programs from foreign builds.

When a program crashes, there is already an identifier that can be used: the build ID that is stored in the .note.gnu.build-id ELF section. But that ID is a long hexadecimal string that is not terribly useful to a human. In addition, the ID can only be related back to the RPM it came from by using the RPM database installed on the system or by doing some sort of network query. An example in the proposal shows how the human-readable JSON in note.package might look instead:

    {
      "type": "rpm",
      "name": "hello",
      "version": "0-1.fc35.x86_64",
      "osCpe": "cpe:/o:fedoraproject:fedora:33"
    } 

The proposal notes that the "directly motivating use case is display of core dumps". The build ID could be used if the RPM database is present, but, even then, there can be problems. It is not uncommon for the package containing a crashing program to have been upgraded behind the scenes, so the installed binary is different than the running one; other mishaps are also possible, so that correspondence cannot be assured. Also, crashes that happen in environments without the database, running from initrd or a sandboxed container for example, can use the JSON note to extract the exact versions of each component involved in the crash.

For users who build their own packages, once again the human-readable information will be more useful than the build ID, which would need to be maintained in some kind of database to map the ID to the source version. In addition, binaries are sometimes pulled from Fedora for use in other distributions—and vice versa. Being able to easily find out where a binary came from will be useful in those cases too:

Whilst most distributions provide some mechanism to figure out the source build information, those mechanisms vary by distribution and may not be easy to access from a "foreign" system. Such mixing is expected with containers, flatpaks, snaps, Python binary wheels, anaconda packages, and quite often when somebody compiles a binary and puts it up on the web for other people to download.

David Cantrell had a few questions about the proposal. He wondered why Fedora should care about mixing-and-matching its binaries on other systems; if there is no way to reproduce the problem on a vanilla Fedora system, bug reports are not likely to be entirely useful. Poettering said that having the information will be useful because it will help show that the problem happened in a mixed system. That will give Fedora the opportunity to either try to reproduce the bug, perhaps in conjunction with the other distribution, or at least allow it to be "more efficient with 'not caring' for non-fedora issues".

Cantrell also asked about whether the "NEVRA" (name-epoch-version-release-architecture) package information is sufficient, because it may not be unique and wondered if the build ID plus debuginfod servers would be enough. Debian developer Luca Boccassi noted that access to the network is not a given, nor is it desirable from the sandbox that systemd-coredump runs in. Adding the URL for the debuginfod information to the package note is a possibility, as well.

There is also a privacy issue to consider Jędrzejewski-Szmek said: "querying debuginfo servers may expose information (about what is running, in what versions, what is crashing, etc.) Thus such queries need to be opt-in and under user control." He also pointed out that the Fedora Koji build system ensures that the NEVRA information is unique for the packages it creates.

Kevin Kofler had a number of objections to the feature, however. In effect, he was objecting the whole use case of running Fedora binaries in environments where the RPM database was not present. He also claimed that the licenses for the code were being violated when pulling binaries out of RPMs without providing the source code. But the licensing question is largely irrelevant, Jędrzejewski-Szmek said; in some cases there may be a license violation, but in lots of others there is not. The package information in the binary will actually help figure out when there is a problem of that sort, he continued.

Not having an RPM database results in a non-functional Fedora installation, Kofler said: "how can this not be broken?" But lacking an RPM database is rather common, especially in the container world, Daniel P. Berrangé said. It would sometimes be useful to have that database available, but it is not a high priority in container-land, which does not mean the use case is broken: "It is simply a different approach / attitude / tradeoff towards using & maintaining the software stack."

"Bloating" all of the ELF objects in Fedora to support the use case is not reasonable, though, Kofler said. The proposal notes that the overhead is around 200 bytes per ELF object, which results in an increase of 13MB if every object in the distribution had the package information added to it. Since Kofler does not seem to accept that the use case is valid, any increase to support it is unnecessary in his eyes. But Boccassi pointed out that cost is tiny to support a use case that is prevalent:

[...] it has happened, it is happening and it will keep happening, because for others it is perfectly logical and highly desirable. So one can either stay here and complain all day long that containers are bad and they are all doing them wrong, and if they only listened to reason everything would be just perfect, or one can do something to significantly improve the baseline for everybody at a cost so ridiculously negligible that if the same standard were applied to compiler updates or changing build flags or whatnot nothing would ever, ever change.

Furthermore, Poettering noted that even vanilla Fedora systems have a piece without an RPM database:

You too run a system with no RPM database – all the time, and that thing still calls itself Fedora: a dracut initrd is exactly that: built from RPMs but without any RPM db.

Thing is, there are different ways to update stuff. rpm/dnf is one thing, dracut image rebuilds is another, containers are typically updated very differently too. rpm is a useful tool (and by embedding rpm meta info into the ELF objects it becomes even stronger), but your assumption that rpm/dnf based updates is the only right way to upgrade stuff is simply neither reality nor even desirable.

A possible security issue with the proposal was also raised by Cantrell. He wondered if there might be problems with using JSON, which has been the source of some security problems in the past. "Of concern to me are encoding formats, size limits or reporting, and structure formats." Poettering said that JSON was chosen because of the "battle-tested parsers" that are already used in systemd and elsewhere. Jędrzejewski-Szmek added that "the implementation in systemd is undergoing continuous fuzzing in oss-fuzz", so there is reason to hope that many of the parser bugs have been found and fixed.

Overall, the reception was largely favorable, though there are some concerns. It is a change with fairly minor effects on binaries—200 bytes hardly seems onerous—that can help in a number of scenarios. It also work toward a cross-distribution effort. Microsoft's CBL-Mariner container distribution has added support for the feature and it will be proposed for Debian; others may well follow suit. It will be up to FESCo, of course, but the objections and concerns do not seem to offset the benefits that it will bring.