Glibc project revisits infrastructure security

The GNU C Library (glibc) is the core C library for most Linux distributions, so it is a crucial part of the open-source ecosystem—and an attractive target for any attackers looking to carry out supply-chain attacks. With that being the case, securing the project's infrastructure using industry best practices and improving the security of its development practices are a frequent topic among glibc developers. A recent discussion suggests that improvements are not happening as quickly as some would like.

On May 9, glibc maintainer Carlos O'Donell wrote to the libc-alpha mailing list to ask other glibc developers to review a secure software development life-cycle process document that he had drafted for glibc. He also provided a similar top-level document for the GNU toolchain that includes GNU Binutils, GCC, glibc, and the GNU Project Debugger (GDB). The goal is to define "what we expect from the infrastructure, developer end points, and our process" in order to figure out what services are needed to create a more secure development process.

The glibc project is hosted on Sourceware, which provides project hosting for free-software toolchain and developer tools, including those that comprise the GNU Toolchain. O'Donell noted that some of the items in his document were taken from Sourceware Cyber Security FAQ in its section "suggested secure development policies for projects", but had been rearranged into a structure that matched the NIST Secure Software Development Framework, which is the standard he recommended as "the simplest and least prescriptive".

In a nutshell, the document suggests top-level practices to be adopted "in order to develop a secure and robust GNU C Library". This includes treating infrastructure as a zero-trust environment in which it is assumed that any of the services, developers, administrators, or systems have been compromised and attempting to limit the consequences of such a compromise. It carries a host of recommendations such as defining security requirements for developers, implementing security tooling, and separating services into distinct systems or VMs.

Hosting

O'Donell emphasized that he was not talking about where to host the project's infrastructure, though the document does discuss hosting. This was worth noting, as the topic of toolchain infrastructure and hosting has come up a number of times, almost as an annual ritual at this point. It was, for example, raised in 2023 by O'Donell, and again in 2024, with O'Donell unsuccessfully trying to drive a core-toolchain-infrastructure (CTI) project that would have moved some of glibc's core collaboration services to infrastructure managed by the Linux Foundation. The statement of work for CTI proposed moving glibc infrastructure to a cloud vendor, adding multiple points of redundancy, as well as 24/7 monitoring and engineering support to handle service outages or "high-risk security events". The annual running cost for CTI was estimated at $276,000.

Sourceware became a member of the non-profit Software Freedom Conservancy (SFC) in 2023, in part a response to a push by O'Donell and others to move GNU Toolchain services to the Linux Foundation in 2022. The current costs of glibc's infrastructure are somewhat nebulous, as much of Sourceware's infrastructure is provided as donations rather than billed by a provider.

The infrastructure and services for Sourceware are managed by volunteers, with hardware, bandwidth, and hosting, and other services donated by number of individuals, companies, and other organizations. Red Hat provides the main server (singular) for several services, as well as a backup server should that one fail. The Oregon State University Open Source Lab, which recently had a funding scare, hosts the server that provides automated source and documentation snapshots. There are a number of machines provided and administrated by other organizations and individuals for building software on various architectures, such as arm64, RISC-V, and s390x.

Mark Wielaard, who serves on the Sourceware project leadership committee, posted a report on Sourceware's second year with the SFC on May 27. According to that report, Sourceware's total income over the last year was about $3,000 from personal donations, and it has spent about $240 on PayPal fees and spare disks for its servers. In total, it has a little more than $10,000 in the bank.

According to Sourceware's infrastructure security page, the site hosts more than 25 projects that have more than 350 active developers and 1,000 contributors. The page has a plans section at the bottom with a list of high-level goals to improve the security of Sourceware's processes and services. This includes isolating services, modernizing account-management processes, improving the release-upload process, and hiring a part-time junior system administrator. The list of plans is unchanged since the page was first captured by the Internet Archive on May 28, 2024. Wielaard noted in his report that Sourceware is looking for sponsors to help "accelerate" its security plans.

CTI

The CTI discussion in 2024 was contentious, with glibc maintainers objecting to both the way the proposal was developed and the choice of Linux Foundation services. Zoë Kooyman weighed in on behalf of the Free Software Foundation (FSF) to say that it opposed the effort to move glibc to CTI. She noted that the proposal would mean that only Linux Foundation IT staff would have administrative access to the servers for CTI, thus no one outside the foundation would be able to improve, maintain, or audit the infrastructure. Sourceware, on the other hand, "accepts technical contributions, and LF IT could be making them right now".

Andrew Pinski asked why the proposal was not developed on the glibc development list, and said that it "gives the vibes of being too closed and being done in a rush" without thinking the proposal through. Alfred M. Szmidt complained that it smelled like a corporate push and was not something the community wanted. Wielaard questioned why O'Donell was "pushing for something that was already highly controversial" and received negatively when it had been proposed before:

I thought we had consensus that the community wasn't really helped by setting up a corporate controlled directed fund or by having a highly disruptive change of infrastructure providers. [...]

Personally, as a glibc developer, I don't want a messy migration of some of the services separating glibc from the rest of the core toolchain and developer tool projects hosted at Sourceware. And looking at some of the other replies I think there is sustained opposition to this idea.

That opposition has not abated. Pinski said of the new proposal that the glibc document was less about security and more about pushing glibc toward the CTI project. He said that it would be better to step back and discuss glibc's model of submitting patches and approvals. Wielaard thought that the proposed policy would be better and clearer if it concentrated solely on the secure-development process. "We have better/separate documents for the hosting infrastructure security parts."

Isolation

Joseph Myers, however, worried about Sourceware running many services that were not isolated to separate virtual machines or containers. That may have been fine 25 years ago, but the project should assume now that it is "at risk of targeted attacks from state-level actors". Its practices were outdated ten years ago, he said, and certainly outdated when he raised similar concerns during GNU Cauldron in 2022. That was likely a reference to a Birds-of-a-Feather session on Sourceware's toolchain infrastructure that included a presentation by O'Donell and David Edelsohn about using managed services from the Linux Foundation. LWN covered this session, which was "loud, contentious, and bordered on physical violence at one point".

In 2022, Myers floated the idea that Sourceware administrators could move to "a modern high-security setup with isolated services", so that compromises of one project or service would not impact other projects or services. He said he had not seen much progress on isolation since 2022, though there had been a few security improvements, such as disabling inactive user accounts:

If Sourceware doesn't do such a migration to more secure, isolated hosting of services (within a reasonable time starting from 2022), that also serves as evidence as far as I'm concerned of the advantages of different hosting arrangements. If in fact lots of such migrations have happened since the 2022 Cauldron and are continuing to happen for the last few unisolated services, that serves as evidence that Sourceware provides suitable hosting arrangements but needs to work on improving how configuration changes and administrative actions get reported to the hosted projects.

Wielaard said that the Sourceware organization was working on it, though progress might not be as fast as Myers might like. Sourceware had started isolating processes using systemd services and resource controls, and there would be an opportunity to move to separate containers or VMs in Q3 when Red Hat's "community cage" servers move to a datacenter in Raleigh, NC. (An update on this move, specific to Fedora's services, was posted in April by Kevin Fenzi on the Fedora Community Blog.)

Security check list

While much of the conversation focused on the project's hosting infrastructure, there was some discussion of the other elements of O'Donell's document. Wielaard questioned whether the NIST format was the right one. It contains useful elements, he said, but "in general it isn't really a good way for a community project to document its cyber security practices". He added that the topic of a "secure development policy champion" had come up during the Sourceware office hours the day before.

O'Donell replied and volunteered to be glibc's secure-development policy champion. He disagreed that the NIST framework was not suitable for glibc, and pointed to a document he had created that compared Sourceware's cybersecurity policy to NIST's framework. His analysis concludes that items in Sourceware's checklist "do not clearly flow from any top-level requirements for security e.g. why would I do this particular step?", and recommends that the checklist should be rewritten to match NIST's framework.

Wielaard said he appreciated that there were interesting points from NIST, but a free-software project is unlike the organizations described in its document. "Pretending it is distracts from the strengths of collaboratively working together on Free Software." He added that Sourceware had been mostly looking at the European Union Cyber Resilience Act (CRA), and the checklist aimed to help create a documented, verifiable project security policy to prepare for the CRA becoming law. He said that it was great that O'Donell was volunteering: the best way forward would be to go over the checklist to document things that are already implemented or how to adopt any item that glibc is not already doing:

At the meeting several people said that we shouldn't mandate any specific policy item, but that we should look at making it attractive for contributors to follow policies because they agree it is good for the project as a whole. At the moment only the retiring of inactive accounts is mandated.

One suggestion was to use some kind of gamification between projects to see who did most. e.g. each quarter we publish a "signed-commit census report". We could turn that into a kind of leaderboard by sorting the projects by number of signed commits or number of people pushing signed commits. Last quarter glibc had just 8% of signed commits, that percentage could certainly be higher for Q2!

With that, the conversation seems to have sputtered out for now. The matter of glibc process security will, no doubt, come up again in the future. The project, and Sourceware, do seem to be inching toward better security practices and more secure infrastructure. However, the current status is less than comforting given the importance of glibc and the overall GNU Toolchain. Given the history of attacks on free-software projects (like last year's XZ backdoor) and infrastructure, one might expect a little more urgency (and industry support) in seeing to those improvements.