Debian's "secret" sauce

While Debian's "sauce" is not actually all that secret, it is not particularly well-known either, Samuel Henrique said at the start of his DebConf24 talk. There is a lot of software-engineering effort that has been put in place by the distribution in order to create and maintain its releases, but "loads of people are not aware" of it. That may be due to the fact that all of that is not really documented anywhere in a central location that he can just point someone to. Recognizing that is what led him to give the talk; hopefully it will be a "first step toward" helping solve the problem.

Henrique said that he is a Brazilian and has been a Debian Developer since 2018. He mostly works in the security tools packaging team, but also maintains some packages outside of that team, including curl and rsync. In addition, he mentors Debian newcomers, mostly on packaging tasks. He works on the Amazon Linux security team. He noted that his web site contains links to the slides from the talk; a WebM video is also available. [Update: A YouTube video with subtitles in English and Portuguese is also available.]

Underpinnings

A distribution is a project set up to distribute software; it can choose defaults or tweak how a software package behaves to make it easier to use, for example, as part of that work. But it is important not to ship bugs or other issues, including security problems, in those packages. Some distributions try to stay as close as possible to the upstream code, but "in Debian, we like to improve things" by "applying our own judgment" to the code.

Beyond that, the project needs to support whatever it is that gets shipped. That means there is a need to put procedures in place to "make sure we are following up with reported issues" and fixing them. At all times, the project needs to be supporting the current Debian stable release, while developing the next stable release. "We have to do both things at the same time, and so that is a constant process."

The Debian social contract aligns the project in an "ideological manner"; it provides a definition for "free software", for example. The Debian constitution provides a framework for how the project operates, how it apportions power, how it makes its decisions, and so on. On a technical level, the policy manual describes how packages should (and should not) operate, as well as how they should be put together. And, finally, the developer's reference is meant to give recommendations to packagers on how to follow the policies and to describe tools that can help. Those documents help create a "tight organization where we are all sharing the same goals, following the same rules, roughly speaking", which serves as a basis for the work that the project does.

Forms of Debian

There are multiple releases and repositories that the project works with, Henrique said. For example, a Debian developer who wants to try something out might push it to the Debian experimental repository, which typically only has a few packages. Experimental is not a full release, so it requires being enabled on a system running Debian unstable, then packages from experimental can be installed.

Unstable, also known as "sid", is a full release, so it can be installed on a system. Unstable is a rolling release that is "constantly being updated" and is not officially supported, so "you should not be installing this on your production servers". Debian testing is also a rolling release, but it is "more stable than Debian unstable"

Finally, there is Debian stable, "which is what you should be running on production servers". There is also oldstable, which is the previous stable release that is still being supported. The backports repository provides "newer packages for users of Debian stable"; it is meant to give users flexibility, but it is "not officially supported" as a release.

Architectures

Debian supports 20 architectures, "nine of them are official, 11 of them are not official". Being official means that all of the packages in stable and oldstable are built for them, as are the packages in unstable, testing, and experimental. The non-official architectures only get packages built for stable unstable, testing, and experimental. The list of architectures is "constantly changing"; the Debian release team determines which architectures are official or not before each new stable release. That decision generally comes down to a question of long-term support for an architecture, he said: will there be sufficient infrastructure to build the packages and enough people to help support them?

Henrique put up a screen shot from the buildd.debian.org web site, which showed the build status of the coreutils package on sid for all of the architectures. It shows green for a successful build and red for a failure (only hurd-i386 currently and in his screen shot). The page shows the logs of the build failure; the results of earlier builds and other releases can also be seen at the site.

Supporting multiple architectures matters, he said, because it finds bugs. Currently, Debian supports two architectures for non-Linux kernels, both for GNU Hurd on x86 systems, though it used to also support kFreeBSD. It supports both big- and little-endian systems, as well as 32- and 64-bit architectures; there is also diversity in the size of C data types. All of that is summarized on the Debian wiki. That diversity makes it easier to find problems in upstream software, even when the developer does not have access to, say, a big-endian system to test with. "So if your software is packaged for Debian, you're at least going to know if it runs or not"; maintainers may care about that or might not, but they will at least know.

Henrique pointed to a fix to the tests for GoAWK, which is an AWK implementation in Go, when it is built on 32-bit systems. It had been packaged for Debian and entered the distribution as part of the unstable release when the problem was discovered; the Debian maintainer reported it upstream and developed a fix. Debian was apparently the first to build and test GoAWK on 32-bit systems.

Another example is a bug he reported in the Aircrack-ng WiFi-security tool. It turned out to be a problem building and running the tests on big-endian systems. This kind of thing has happened before, he said, and is fairly easy to spot because all five big-endian architectures suddenly start reporting build/test problems.

Repositories

Henrique created the following diagram to help attendees understand how packages flow into the development repositories: experimental, unstable, and testing. The Debian developer in the upper left has four different places where they can send a new or updated package. They can directly put the package into the experimental or unstable repositories, or they can send updates that are bound for testing to either the release team, for regular updates, or the security team, for security fixes.

He noted that the experimental repository was not further connected in the diagram, "it goes nowhere", which makes sense because of how it is used. The repository makes it "very easy to test things". For example, if he wanted to enable a new architecture for a package and make it available to others for testing, he could upload it to experimental; that will also cause some of the automated testing to happen, which will generate useful reports. "I get a preview of what would have happened if I sent this package to unstable."

It is easy to revert changes in experimental, as the package can simply be removed. In addition, successful changes can be pushed to unstable, which allows having two variants of the same package available at the same time. The curl packagers recently used that ability when the TLS backend was being changed from OpenSSL to GnuTLS; the GnuTLS version stayed on experimental for a month or so, while the OpenSSL version was updated several times on unstable.

The unstable repository does not provide a supported release, but it does have all of the packages available, unlike experimental. It is "installed and used by expert users", however; since it is the first place that new packages show up, a lot of developers make use of it, he said. Packages in unstable get full coverage from the testing and QA infrastructure. It is "important not to break things too badly" because it risks breaking other packages; for risky changes, experimental should be used instead.

Packages in unstable automatically migrate to testing "after some rules are followed"; the package needs to be passing its tests and spend some time in unstable before it gets promoted. While testing is not supported either, it is generally much more stable than unstable since the packages there are bound for the next Debian stable release. There were some recent problems with the 64-bit time_t transition that affected testing, however, which is the kind of thing "that happens every 20 years, maybe"; that shows that testing can still break, but it is "quite stable" overall.

Unlike experimental or unstable, testing has an installer available, so that the installer team can test it before releasing. That means testing can be installed from scratch. He recommends testing for non-server use cases, desktops for example. "I believe it is as stable as any other rolling release" due to all of the checks and procedures that the distribution has put in place.

Stable

When thinking about Debian stable, there are two parts to consider, Henrique said: its creation and its maintenance. The creation happens every two years, by using a snapshot of testing; once it is created, packages are not updated as they are for unstable or testing. Users running stable are looking for something that is predictable, not something that is bug-free; "the point is there won't be ten new bugs showing [up] each day".

But there needs to be a time when packages cannot freely flow from unstable to testing in order to stabilize something that will become stable. The release team has a freeze process that changes the ability of packages to migrate to testing. The requirements for package migration change and get stricter as the various phases of the freeze process progress; in the final phase, any changes need to be manually reviewed and approved by the release team.

The last freeze, for the Debian 12 ("bookworm") release, lasted around six months. During the freeze, there is less activity in the unstable and testing repositories, because the focus is on fixing what is in testing; but updates that are bound for testing are still being pushed to unstable.

Some of the packages pushed to unstable during a freeze may not be bound for stable, however; if there is a need for a fix to the version of that package in testing, a different path must be taken. That is where the testing-proposed-updates and testing-security repositories (from the diagram) come into play; with manual review from the release or security team, patches can go into those repositories and then to testing from there.

Once there has been a stable release, though, the path for package updates also requires manual review by one of the teams depending "on the nature of the fix", Henrique said. Critical security fixes, such as for a CVE, might go directly into stable, but others will "spend baking time in the proposed-updates" repository; the normal requirement is that packages spend a week in proposed-updates before moving to stable. Periodically, the release team will pause migrations from proposed-updates to stable in order to create a point release.

Tools

There are "a million tools in Debian" that can be used for doing various kinds of testing, such as for continuous integration (CI) or general QA. He chose to focus on a few of those in his talk, but all of them are freely available; "anybody can just query all of the build logs of Debian packages, identify a pattern of something that's wrong, and provide patches" to fix the problems found. For those who are curious, the Debian QA group wiki page covers the full set of tools and the processes the group uses.

The dh_auto_test tool is hooked into the package-building system and tries to run the upstream tests once the package is built. It can detect things like testing targets in makefiles and run them automatically, but if it cannot determine how to run the tests, it can be configured to run any tests of interest. It is meant to confirm that the package works in the Debian environment, for all of its architectures, and with the build flags (including hardening options) that the distribution uses. Even though the upstream project is also running its tests, the dependencies that Debian uses may be different; "sometimes we spot issues" because of all of those differences.

Lintian does lots of different tests on packages to find "silly things, like spelling mistakes" or more serious problems like the presence of binary blobs without source files. It is hooked into the build process and if it detects problems that are serious enough, the package will be rejected. His first contribution to the curl package was for a problem detected by lintian: curl was still using Python 2 in its tests after the distribution had switched to Python 3, which was easy to fix.

Autopkgtest standardizes the test definitions for packages so that they can be run as part of the CI system. These tests are not build-time tests, but are used to run end-to-end tests of the packaged software. The autopkgtest scripts often include running the upstream tests, but additional tests are generally added; "you have a lot of freedom here" to add dependencies needed to test the software in various ways.

While autopkgtest is the mechanism used, debci is the service that runs the tests continuously, the results of which can be seen at ci.debian.net. It runs these tests "on a long list of architectures" for each change, though not all of the architectures that the distribution supports. The main artifact that it produces is reports of the tests, which can be used to determine whether the package should migrate from unstable to testing.

A given package will be tested, but there is more to it than that; its dependencies are also being tested, as are the packages which either depend on it or have tests that do. The intent is to find regressions where the previous version of the package was passing, but it no longer does; so, when tests fail, the previous version is tested to see if it also fails. So if package A gets uploaded to unstable, debci will look at which packages depend on it and find package B, for example; it will then run the tests for B using both the old and new versions of A and compare the results. That can all be seen on tracker.debian.org on a per-package basis, such as for the glibc package.

He gave two examples of where this process finds bugs that may have gone unnoticed otherwise. A 2021 test failure for the aeskeyfind utility, which is a forensic tool to find AES keys in memory, silently failed because it was relying on undefined behavior that changed due to a different GCC version. Since aeskeyfind is unmaintained, other distributions may also have the problem, though Henrique has tried to get the word out on it.

The other was a failure of UDPTunnel test due to a change in the Nmap port scanner. The problem was a hidden regression in Nmap that was detected by autopkgtest and debci, which resulted in an Nmap bug report and subsequent fix.

Salsa CI is one of his favorite tools; it is based on the Debian GitLab instance, Salsa, and helps "make sure our packages are all nice and stable". The Salsa CI team provides recipes for tests that run on individual commits for packages. He showed a Salsa CI page for curl, which lists a bunch of different tests that were run, including build tests, autopkgtest, a test for possibly missing build flags, such as for hardening, a build reproducibility test, and more. There were a few other tools he briefly mentioned, including one to ensure there are no multi-arch problems and Janitor, which will proactively scan source repositories and send merge requests with suggested improvements.

An audience member asked about what other distributions do with regard to QA. Henrique said that he knew a bit about others, including Fedora, which has some equivalent systems and processes, but thought that Debian is "the best one doing this today". With that, the session ran out of time. Overall, he provided a nice whirlwind tour of Debian development and maintenance in his presentation—and helped show the recipe for the sauce.

[I would like to thank the Linux Foundation, LWN's travel sponsor, for its assistance in visiting Busan for DebConf24.]

hppa             : 8
m68k             : 2
mips             : 5
powerpc          : 50
ppc64            : 37
s390x            : 10
sparc            : 2
sparc64          : 22