The history, status, and plans for reproducible builds

On the second day of DebConf24 in Busan, South Korea, Holger Levsen provided a history lesson on the "first 11 years" of the Reproducible Builds project. He has been involved in the project for most of that time and has been a Debian user since the mid-1990s, contributor since 2001, and a Debian member since 2007; "I love Debian". Meanwhile, his aim is to make all free software be reproducible, so that anyone can check that a binary program comes from the source code it purports to.

He began by noting that the talk was not really only his, but was instead a talk that comes from the work of more than 100 people listed on the Reproducible Builds web site. He asked a few questions of the audience, such as who knows about the project, who has contributed to it, and who knows that the project itself is more than ten years old but that the idea of reproducible builds goes back more than 30 years? The goal of the talk is to recap and celebrate what has been done, he said, in order to get attendees excited and, thus, involved in the project. "Because there is still a lot of work to do."

The problem is that, while the source code of free software is available, most people install pre-compiled binaries. "No one really knows how they really correspond, even those building the binaries." The machine doing the build might have been compromised, for example. Because of this problem, there are various types of supply-chain attacks that can result.

Historical overview

He reviewed some "ancient history" noting that the idea had been raised in Debian back in a 2007 post to the debian-devel mailing list. The reaction at the time was not enthusiastic; some saw no benefit, others thought it was technically infeasible. In fact, the idea had popped up even earlier, in 2000, on the list. On the Reproducible Builds rb-general mailing list in 2017, John Gilmore pointed out that GCC and related tools could be built in a reproducible way for several architectures in the early 1990s, though that ability had bit-rotted.

"Fast forward to last year", there was a posting on the WireGuard mailing list that the project's VPN app for Android could be built reproducibly. The same binary is produced by WireGuard, F-Droid, and the Google Play Store. Moreover, the WireGuard project never informed the Reproducible Builds project that it was working on it; that made him happy because it means that making builds reproducible is just something that some projects do as a matter of course these days.

The Reproducible Builds project defines the term as follows: "A build is reproducible if given the same source code, build environment and build instructions, any party can recreate bit-by-bit identical copies of all specified artifacts." It goes on to say that the environment and instructions are specified by the authors of the package, while the artifacts are the "desired primary output" from the build.

The mission of the project is to enable anyone to verify that a given source produces bit-for-bit identical results. "We don't want that 'Debian says that the software is reproducible', each of you should be able to also independently confirm this." Reproducible builds are an important building block for making supply chains more secure, but they do not make insecure software secure. They simply ensure that "you are running the software that you mean to be running".

These days, reproducible builds are much better known. The project site lists talks and other resources, academic publications, and lots of other information on its documentation page. In 2021, the US President issued an executive order that specifically mentioned requirements for a software bill of materials (SBOM) to accompany software, so that users can determine what has gone into their binaries; an SBOM is needed for reproducible builds as well.

"So, how did we get there?" He said that money was one factor. The Bitcoin software was made reproducible in 2011, before Debian started working on reproducible builds, when all of the Bitcoins together at that time were worth around $4 billion. The Bitcoin developers were worried about backdoored binaries stealing all of that money, and the blame falling on them, so they made the software reproducible.

Another factor was Edward Snowden, Levsen said. The Tor browser was made reproducible in 2013, in part due to the information that Snowden released. The Tor browser is based on Firefox, "one of the biggest" software projects in the world, he said, with a 50MB binary. Beyond money and Snowden, though, it took "the work of many people over many years".

Debian

In 2013 at DebConf13, Jérémy Bobbio (also known as "Lunar") hosted a brainstorming session that was inspired by the success of the Tor browser reproducibility; there was another birds-of-a-feather (BoF) meeting at DebConf14. Levsen asked if anyone in the room had been to the 2013 meeting, but no one had, including him. At FOSDEM in 2014, Bobbio gave a talk on reproducible builds for Debian in the developer room (devroom) for the distribution.

The first patches for dpkg came in 2014; they included fixing the sort order of the elements in the data and control files and creating the .buildinfo format, which is, effectively, an SBOM format. The .buildinfo files have all of the information about the environment, source code, hashes for the tools used to build the package, and more.

In September 2014, Levsen began systematic builds of Debian packages to check their reproducible status. He was sitting in the audience of a talk by Mike Perry and Seth Schoen from the Tor project at the Chaos Communication Congress in December 2014 when they used "his" graph of the progress for Debian packages: "Wow". That talk clearly resonated with Levsen as he spent some time showing the slides and going over some of the points they raised.

In 2015, Bobbio and Levsen teamed up on a talk at FOSDEM to invite all of the free-software world to collaborate on the reproducible-builds problem. They had realized that it was better if not just Debian was working on it, so getting other projects involved was important. Bobbio gave a talk at Chaos Communication Camp in 2015 (slides) "showing many problems and their solutions" for making software build reproducibly; that talk is still useful today.

The main thing that makes code not build reproducibly is "timestamps and timestamps and more timestamps", Levsen said, "software likes timestamps". Another problem spot is build paths. The timestamp problem has largely been overcome with the SOURCE_DATE_EPOCH environment variable, while build paths are handled by recording and reusing the same build path. For many years, the project tried different approaches for build paths, but has now decided that the path is simply part of the build environment. There are, of course, other problem areas as well.

Build timestamps are largely meaningless, but they do change the binary. If SOURCE_DATE_EPOCH is set, lots of software, including all Debian packages, will use it instead; it can be set to the time of the last modification of the source, which does have some meaning. Many major projects, such as GCC, Clang, Python, OCaml, and others, all support building using SOURCE_DATE_EPOCH, he said. The original specification for it was created in 2015; there was also a small update in 2017, "since then, it is unchanged". It is around 2KB of text, "pretty short".

Diffoscope

The diffoscope tool also came about in 2015; Bobbio started it as debbindiff, but once it was not Debian-specific, it got renamed. Diffoscope tries to determine "what makes files and directories different"; it unpacks archives, recursively if needed, and transforms binary formats into a more human-readable form in order to compare them. For example, he said, if you compare two .deb files, each of which contains an ar archive that contains multiple tar archives, one of which has a PDF that contains a PNG, each of which has different timestamps in it, "diffoscope will show you that".

Diffoscope provides both text and HTML output. It can decode a huge number of formats; the list he put up spread over two slides (as can be seen in the WebM video of the talk) and was a year old, so "there are probably 20 or more file formats added since then". It started by only taking two .deb files to compare, but now "you can point it to two directories, two ISO images, two anything, and it will compare it". If it does not know the format, it will fall back to a hex-dump comparison

Diffoscope is used outside of software contexts too; he met a lawyer who uses it to compare the PDFs of laws to see what has been changed in the text. He showed an example of the output from two versions of the HTTPS Everywhere browser plugin. "So you can also use it to compare two different software versions" to see the changes that were made. The full package is around 1.5GB in size to support all of those formats, Levsen said, so it can also be used online by uploading files to the web application.

He put up a graph showing the reproducibility bugs that had been fixed in Debian packages over the years. Nearly 4000 bugs have been fixed, most of which have gone upstream, though there are still around 300 patches pending.

Beyond just timestamps and build paths, the project has discovered 430 different problems that lead to non-reproducible builds, which are documented in the reproducible-notes repository. People often ask what to "do to make a package reproducible, and that's hard to say; it's easier to say what makes packages unreproducible". To that end, the project created the unreproducible package, "which has lots of problems".

Over the years, the frequent rebuilds that the project does on Debian packages has also found lots of other bugs, roughly nine per day over the last 11 years, mostly "failed to build from source" (FTBFS) bugs. That was just one of the additional benefits that the project has found. Removing the differences in binary artifacts means that build systems can cache objects more easily, which increases development speed, thus lowering development cost; a lot of big companies are interested for that reason, he said. Reproducibility also helps verify that a change really only has the effect that is expected; that only the binaries that should be impacted have changed.

There have been seven reproducible-build summits held so far, with another coming in September, which he invited audience members to attend. He put up a long list of nearly 70 projects, organizations, and companies that have attended the summits, but noted that there are others who asked not to be listed. Another unexpected benefit of reproducible builds was discovered at the 2016 summit in Berlin, where bootstrappable builds came about in a breakout session. The idea is to bootstrap toolchain binaries from source code only, starting with a minimal assembler in handwritten machine code, then building successively more complicated compilers and other tools.

Reproducible Builds is a Software Freedom Conservancy project since 2018. There are three current sponsors, which allows the project to fund four people: Chris Lamb, Mattia Rizzolo, Vagrant Cascadian, and Levsen.

Levsen showed a graph of the full history of the continuous-integration (CI) system for reproducible builds of Debian unstable packages. It shows that the number of packages has nearly doubled in the 11 years, "and Debian is a lot more reproducible now". For Debian 13 ("trixie"), which is the next Debian release, there are only 764 non-reproducible packages, while there are more than 34,000 reproducible packages.

Policies

Since the end of 2023, the CI results for a package are shown on the testing migration page, though there are no penalties or bonuses for changes to a package's reproducibility status. In 2025, when the development of Debian 14 ("forky") begins, the release team wants to introduce penalties for a change that causes a reproducible package to become non-reproducible; in addition, new packages will be required to build reproducibly before being allowed into testing. Some packages that cannot be built reproducibly can be added to a list of exceptions if Debian cannot live without them.

In 2017, Debian policy changed to say that packages should (not must) build reproducibly. He is hoping that the two policies that the release team wants for forky can be added to the policy in 2025, followed by a switch to "must" in, say, 2027, though there would still be a list of exceptions. Getting to 100% reproducible is not really a technical problem, he said; rather, it is a political decision. An exception list is a means to the end goal, which, for him, is for all packages to build reproducibly.

He put up a slide showing the numbers for the last few releases, along with projections (or hopes) for the next few. Trixie would only have 256 non-reproducible packages, while forky would have 128—without any regressions or new packages that are non-reproducible. For forky+1 in 2029, there would only be 42 packages that fail to reproduce, all of which are on the list. Forky+2 would have 0, "in 2031 or whatever". "That's the timeline I'm hoping for, or aiming for", Levsen said.

The current testing for reproducibility builds packages twice in the same environment, but what is really desired is to compare a build with the package that is being (or has been) distributed by the project. The snapshot.debian.org server is meant to be an archive of all of the packages, but until quite recently, it was broken, he said. So now, the same environment that was used in the past can be recreated and the package can be built; the debrebuild command can be used to do so now that snapshot.debian.org is functional.

There are other Debian artifacts that have been made reproducible over the years. The Docker/podman images listed at docker.debian.net have been reproducible for the last two years or so. The live images available at cdimage.debian.org were made reproducible quite recently, he said. In addition, the Debian-Installer is reproducible in theory, though it is not currently being tested.

Other distributions are also working on reproducibility. Tails only creates a single ISO image, which has been reproducible for five years at this point. Arch Linux has 80-90% of its packages reproducible, while SUSE plans a version that is reproducible in the near future. NixOS and Guix are designed to be reproducible, so they are close to fully reproducible. Yocto, F-Droid, Alpine Linux, and the BSDs (other than OpenBSD) all have basic support for reproducible builds. While it may seem that Fedora, Red Hat, and Ubuntu are not interested, he said, there has been some work done on SOURCE_DATE_EPOCH for Fedora, Red Hat is "a bit more skeptical", and, for Ubuntu, he has "heard rumors" though there is nothing concrete.

So, today many projects support reproducible builds, though there are still gaps in user understanding. He calls it "reproducible in theory or in CI". It is, however, a "massive success" since it was thought to be impossible ten years ago. "In theory, we are almost done. In practice, we have shown that reproducible builds can be done in theory."

For Debian, there is a need to move to rebuilders rather than CI builders now that snapshot.debian.org is working; the results of that need to be stored somewhere and policies need to be worked out on how to use that data. Meanwhile, the number of non-reproducible packages needs to continue to be reduced; each 1% shortfall for Debian equates to around 300 packages. He went back to the Perry and Schoen presentation from 2014 to show that it, too, described the need for tools to independently confirm that binaries correspond to the source they claim, as well as to have some kind of external monitoring to ensure that nothing bad has happened to the build infrastructure. That is how reproducible builds move beyond theory and into practice.

Q&A

The first audience question was regarding the OpenBSD practice of randomizing libraries and binaries on each boot. What does it matter that the build artifact is reproducible if the binaries are deliberately altered on end-user systems each time they are booted? Levsen did not directly address that, but did note that OpenBSD has said that it is not interested in reproducible builds, because "they say 'it is not useful'". OpenBSD believes that its build system is secure so the project does not need them, he said.

Another question was about whether the non-reproducible packages change over time or whether it is the same ones that show up over and over again. Levsen said that "packages that become reproducible usually stay reproducible, it's very rare that a new upstream version introduces unreproducibility". At this point, required and build-essential packages are all reproducible, but there are 96 non-reproducible dependencies of the build-essential packages (out of 5500 dependencies). "It fits on a screen now, but it is still a bit of work."

The final "question" was an invitation to the Cloud BoF for anyone who might be interested in helping the cloud team make those images reproducible. They are, the attendee said, among the few Debian build artifacts that are not at this point. Levsen said that he would meet with the attendee about that right after the talk.

Overall, the mood of the talk was upbeat with what had been accomplished, though that was tempered with what still needs to be done. Reproducible (or "deterministic") builds are an important tool for helping to ensure the software supply chain; it is a topic that LWN has looked in on several times over the years as well. That reproducible-builds journey continues, but the light in the distance is clearly growing stronger.

[I would like to thank the Linux Foundation, LWN's travel sponsor, for its assistance in visiting Busan for DebConf24.]