Understanding Debian's security processes

Providing security updates for a Linux distribution, such as Debian, involves a lot of work behind the scenes—and requires much more than simply shipping the latest code. On July 15, at DebConf25 in Brest, France, Samuel Henrique walked through the process of providing security updates to users; he discussed how Debian learns about security vulnerabilities, decides on the best response, and the process of sending out updates to keep its users safe. He also provided guidance on how others could get involved.

Henrique introduced himself as a member of Debian's security-tools packaging team; he has been a Debian developer since 2018, maintains a several packages, and serves as a mentor for newcomers trying to learn how to package software. He is also a senior system development engineer for the Amazon Linux security team, but he noted that his contributions to Debian are made as a volunteer and not something he does during work hours.

Henrique presented a similar security talk at DebConf24 in Busan, South Korea, called "Fixing CVEs on Debian: Everything you probably know already"; the video and slides are online. That talk, he said, would be a good reference for anyone who wanted to contribute security fixes to Debian.

This year, though, he wanted to give a more user-focused talk because he had noticed that even experienced users and contributors don't understand the way that Debian handles security vulnerabilities. Many users do not understand why Debian backports security fixes, and they often do not know about the Debian backports repository, which contains packages from Debian testing built for the current stable release.

CVEs and vulnerability information

Most of the time, when a discussion comes up about computer vulnerabilities, we're talking about those that have been enumerated as Common Vulnerabilities and Exposures (CVEs). Not always, but "there's a big overlap there". In fact, there are many different areas of security, and he wanted to acknowledge the areas he was not going to talk about. For example, how Debian manages its infrastructure and the way it builds packages are important to the security of the distribution—but he intended to focus solely on vulnerabilities in Debian packages for the talk.

The CVE program was created in 1999 by MITRE, with sponsorship by the US government. The main thing MITRE does, he said, is to assign CVE numbers and coordinate announcement of vulnerabilities by maintaining a public database. CVEs describe a single vulnerability even across platforms, so if a vulnerability affects Firefox on Windows, Linux, and macOS, it will still have a single CVE ID. Anyone can request a CVE ID; there is a process, and it should involve reporting a vulnerability to the developer of the software first, but there are no special credentials required to request a CVE.

Then there are organizations that enrich the data, such as NIST, which maintains the National Vulnerability Database (NVD). It provides a numerical score to rate a CVE's severity, called the Common Vulnerability Scoring System (CVSS). Vendors often use CVSS in service-level agreements to determine how quickly a patch for a CVE must be released. Many vendors, including Canonical, Red Hat, and SUSE, also release their own assessments of a vulnerability's severity that is specific to their products.

Henrique noted that Debian also provides vulnerability information via its security tracker. He showed a page from the tracker for CVE-2023-4911, which describes a buffer overflow in the GNU C Library (glibc). The page contains the CVE description, links to various vulnerability databases, various notes on the CVE, and a list of Debian packages with each one's status related to the vulnerability. For example, the page lists the glibc packages for current Debian releases, which ones are affected, and the version of the package that contains a fix for the vulnerability.

Debian users can get security updates for a release for up to five years from the Debian project, and an additional five years in some cases. Henrique explained that each release receives three years of support from Debian's security team and an additional two years of updates from its long-term support (LTS) project.

He advised the audience that it should update to the newest release after five years; if that was not feasible then there is the extended long-term support (ELTS) from Freexian. While ELTS is a commercial offering, the company makes updates available to everyone for free. However, there are limitations to the ELTS; Freexian only supports a subset of architectures and packages for each Debian release, depending on what its customers request. For example, its Debian 9 ELTS only supports the x86_64, i386, and armhf architectures. The base system packages for Debian 9 are supported for the life of the ELTS, but other packages may not be.

Triaging: anyone can do it

Day and night, weekdays and weekends, the Debian security team is dealing with "the firehose" of CVE information. There were 40,274 CVEs published in 2024, so there is a lot to sift through. Henrique said that the team has automated the import of security data that is queued up for triage by anyone: "if anybody is interested in this, talk to me, and I can teach you". The team has to determine which versions of a package may be affected; that involves "very cool investigative work", such as reading the vulnerability description and searching for the commit that introduced the vulnerability.

After determining the versions of software that are affected, then it's a matter of researching whether it affects a package in Debian. For example, a vulnerability may only be present on a hardware architecture that Debian does not support. That is a bit rare, he said, "because Debian runs on everything".

There are also security embargoes, which are "the most fun thing" about security work, Henrique said, possibly sarcastically. There are a few embargo processes that Debian has to deal with, he said, but they all involve representatives of different organizations coordinating to get fixes ready for when a vulnerability is published.

The most famous embargo group, according to Henrique, is the distros mailing list. It includes representatives from "all the big distros", and access to the list is severely limited. Members of the list may handle security problems themselves, or work with the maintainers of affected packages. It is safe to assume that there is an embargo in effect right now, he said, "there's one going on at all times".

Backports

Henrique circled back to his statement near the beginning of the talk, "people don't understand why Debian backports security fixes" rather than shipping the latest version of software that contains a security fix.

He described a scenario where Debian is shipping version 4 of a project in the stable release, and version 3 in the old stable release. A security vulnerability is announced in the software, and the upstream project fixes it with version 5. Users will think they need version 5, but do not understand that it might break their whole workflow. Just shipping the latest release of software to provide a security fix "is just not suitable for Debian stable".

Instead, the project will try to identify the minimal fix for the problem and backport it to the version currently in each Debian release. If a vulnerability is severe enough, Henrique said, Debian may release a Debian Security Advisory (DSA) to announce it to users and expedite release of the fix to the security repository. (Users can find the most recent DSAs on LWN's Debian security alerts page as well.) If it is a less severe vulnerability, then it will go through the normal proposed updates mechanism and may take up to two months to be released with the next point release for Debian stable.

So far, he had primarily talked only about Debian stable releases. Henrique said that there is no official security support for the experimental, unstable, and testing versions of Debian. However, he said that vulnerabilities in those releases are "all fixed in a timely manner, because we care that vulnerabilities are fixed quickly" even if it the releases are not officially supported. He pointed out that the fix for the XZ backdoor was released for unstable and stable at the same time and took about 24 hours to reach the testing distribution. No promises, he said, but the experimental, unstable, and testing versions of Debian are "more secure than a lot of distros out there".

Sometimes Debian decides not to fix a CVE because it is either too risky, too complex, or the vulnerability is not severe enough to require a fix. For example, Henrique said that some software has support for different network protocols and when the upstream finds a vulnerability it might decide to just remove support for a protocol rather than implement a fix. "That is valid from the upstream's perspective, but we can't just yank out a feature" in a stable release. At least, not unless the severity is too great, "we do an analysis on which is worse". If it's really bad, then it's removed. If not, then the fix will wait until the next release.

In some cases, Debian will decide not to bother with a fix because the vulnerability is essentially impossible to exploit in real-world circumstances. He used the example of finding buffer overflows using fuzzing to target an application's internal functions. "From outside an app, it's impossible to trigger" the vulnerability, Henrique said.

Source packages

Debian has a tooling problem, he said, where a vulnerability is only present in the source package and is not built into the binary package—but Debian's security tooling currently has no way of distinguishing the two. So, to users, it looks like the binary has a security flaw as well. The good news is that there is a fix in the works, contributed by "the Freexian folks" and waiting for Debian's security team to review.

Once that fix is available, Henrique said, he would be able to flag the CVEs he knows are false positives. "I have done some investigation, and roughly 20% of the CVEs in Debian container images" are false positives due to vulnerabilities in source packages that are not present in binaries.

By the numbers

The signal-to-noise ratio of CVEs is pretty low when one considers how many CVEs actually affect Debian. Out of more than 40,000 CVEs in 2024, Henrique said that 2,046 CVEs were fixed and 255 DSAs were issued for Debian stable. Only 56 CVEs that were relevant to Debian, or about 0.01% of all CVEs issued in 2024, had an entry in the US Cybersecurity and Infrastructure Security Agency's known exploited vulnerabilities (KEV) catalog. That does not mean there are no exploits of other CVEs, of course, but Henrique noted the ratio of KEVs to CVEs is an important indicator of actual security vulnerabilities compared to CVEs issued.

People who want to research CVEs on their own should start with Debian's security tracker, Henrique said. He also suggested checking the NVD for extra references and suggested reading "Hacker News, LWN, all those spaces" to keep up with security vulnerabilities. If a vulnerability is serious enough it will have its own name and logo, he said. It may seem silly or funny to brand vulnerabilities, but it works. "Nobody will recognize a CVE number a few years later, but everybody remembers Spectre."

He urged users to pay attention to all mitigating factors for a vulnerability, because it may only be a problem in certain situations or in non-default configurations. While it's critical to apply security fixes, eventually, "you need to understand if you are actually exploitable at that time". He also said people should be "very critical of descriptions in CVEs", as they often imply things that are not true. For example, just because a version of a project is not listed in a CVE does not mean it is not vulnerable.

Understand risk

Users should also be wary of Debian derivative distributions, Henrique said. He did not name any distribution specifically, but said that distributions that pull from Debian may have security problems that Debian does not.

He advised the audience to update their machines frequently. "The real risk is not zero days; it's machines that have not been updated in a long time". He also coached users to pay attention to "unofficial stuff on machines", such as IDE plugins, editor extensions, and other software from "random corners of the internet". If there is a vulnerability in software that is installed from third-party repositories, will users even notice? Will they pull the fix? "I'm not saying don't do this; I know it's required. But understand how you're changing your risk when you do it". If something is unused, it should be uninstalled.

In summary, he said that the Debian security tracker is the canonical source of information for Debian vulnerabilities. Just because a package is affected, he said, does not mean that a system with the package is affected. "A lot of CVEs are just noise".

Q&A

There was a little time left over for questions, so I asked Henrique if there were any practices from other Linux distributions that he would like to see Debian adopt. He said that he did not like that Debian over-reported on CVEs with false positives, but that was changing "with some help from Freexian", and he expected to have fewer false positives in the future.

Another member of the audience asked what his recommendation would be for security scanners that look for software version numbers to determine if a system is vulnerable, which means that Debian is sometimes falsely flagged. He said that he loves that question: "any security scanner that is good enough is reading Debian's data instead of guessing versions; tell your vendor their scanner is broken."

The slides from the talk are available on Henrique's website. Video from the talk should be made available by the DebConf25 video team soon.

[Thanks to the Linux Foundation, LWN's travel sponsor, for funding my travel to Brest for DebConf25.]