Safeguarding GNOME.org with an upload lockdown

Some effects of the kernel.org break-in were immediate, but others have rippled out more slowly. One example of the latter is the GNOME project's system administration team taking a hard look at the security of its own infrastructure. The team is drafting a plan to tighten down access to its servers and looking at the development processes of the project — starting with the module upload process.

Olav Vitters sent an email to the GNOME desktop-devel list on November 10 that outlines a plan to preemptively secure the master.gnome.org server against some preventable compromise paths. His root concern was that more than 350 GNOME developers have shell access, which provides a large attack surface even though most (if not all) of those accounts are used only to upload tar packages — a process that could be more securely performed in other ways. Considering the lengthy process that the kernel.org team had to undertake to restore the server after its compromise, the concern is a very real one.

The GNOME project maintains a variety of public-facing services on different machines (Bugzilla, mailing lists, wikis, web sites, etc.), which are managed by one full-time system administrator paid by the GNOME Foundation, and assisted by a small team of volunteers. Vitters is one of those volunteers. His message to the list, which he styled as an IETF-like "request for comments" (RFC), targeted just one service: the FTP server used to distribute GNOME tarballs.

In the current FTP release process, package maintainers upload tarballs to a GNOME project server using SCP, then log in over SSH and execute an "install-module" script. The script calculates the checksum(s) of the newly-uploaded package(s), copies them to the proper location in the public FTP directory, and triggers another script that starts an FTP sync between the GNOME project server and the "real" master FTP server (which is located off-site) and the official mirrors.

Restricted SSH and DOAP

Due to the interactive SSH step, Vitters said, if any of the 350-or-so package maintainers has their personal machine compromised, an attacker could gain shell access to the GNOME project FTP server. That was the path used by the kernel.org attackers, who then presumably used a local privilege escalation to completely compromise the machine. It is very probable that one or more of the GNOME user's machines has been compromised, but that has evidently not led to unauthorized access to the GNOME servers.

Vitters proposed a replacement module-upload plan, in which the initial file transfer would be performed with rrsync (the Perl-based program that restricts rsync to a pre-determined subdirectory) over an SSH tunnel. The user would then call a revised version of the install-module script remotely over SSH, where the SSH daemon on the server was configured to only permit execution of that script, rather than a full shell.

The server-side scripts already in use would also be modified to check the user account that uploaded the new file against a list of approved uploaders for the module, taken from the module's Description of a Project (DOAP) file. DOAP is an RDF format used to define the semantics of a project and its team; GNOME already stores DOAP files for each module in the project's Git repository.

The primary benefit of the plan is that it eliminates the need for the 350+ shell accounts. Even if a module maintainer's home machine is compromised, keeping the server locked down so that remote shell access is unavailable makes it more difficult to wreak large-scale havoc. It would still be possible to replace a package, but the identity check performed against DOAP would, in theory, mean that the attacker could only replace a small subset of packages for the majority of accounts. This is an improvement, because currently anyone with a shell account on the server can upload any package.

Vitters posed several questions for the list, starting with whether or not the proposed process was so much more arduous than the existing system that it would meet with resistance. But he also pointed out potential technical hurdles such as uncertainty over the reliability of rsync, whether rsync could be used to delete files, and the fact that although most GNOME modules have a DOAP file in Git, those GNOME modules that are not tracked using Git do not.

Reaction and feedback

Maciej Marcin Piechotka asked whether scp might be preferable to rsync, and suggested that the Git DOAP files are not secure, because anyone with Git access could upload a new DOAP file — a change less likely to be noticed by others than is the upload of a new tarball. Vitters replied that he considered, but ruled out, scp and sftp for the upload step because both permitted more file operations than rsync alone. In the interest of "just allowing exactly what is needed," he said, rsync was the better fit. "I don't want people to be able to modify anything [other] than uploading a tarball." He confirmed the vulnerability of the DOAP file in Git, and proposed modifying the upload scripts to notify existing maintainers whenever the DOAP file is modified.

Ray Strode then weighed in against restricting uploads to individuals in the DOAP file on the grounds that there are a lot of less-actively-maintained modules, and an unpredictable set of users who periodically need to make releases of them. "I think the lack of fine grained ACLs for this sort of thing is one part of GNOME that work better than projects that try to lock things down more aggressively," he said. Matthias Clasen and Bastien Nocera added that they were two such users, who occasionally update maintenance-mode modules when a change in such a module is mandated by an update to one of their own modules.

Vitters then suggested that the allowed-user-list could include the maintainers of each module plus all members of the release team, since both Clasen and Nocera are on the release team. Clasen and Nocera seemed happy with that adjustment, but Strode did not:

Strode's concern is not easily reconcilable with the overall plan, because it boils down to a philosophical, not a technical question: how locked-down is too locked-down. A lax security policy towards the less-frequently-updated modules simplifies the work for the administrators and allows more people to contribute, but the less-active modules are an easier target for attackers. Inserting malicious code into an actively-developed module is likely to get noticed in short order; by instead inserting it into a nearly-forgotten Makefile that is still pulled in and executed when building GNOME, one accomplishes the same goal with less risk of getting caught.

Alan Cox suggested that the team take a look at the tools used by the kernel.org team, which went through much the same locking-down process in September and October in response to the August break-in.

Lessons from kernel.org

The kernel.org lockdown began with the same concern as Vitters' RFC: eliminating the several-hundred shell accounts used mainly for uploading. Since the break-in, H. Peter Anvin has developed a suite of utilities for kernel.org that implement much the same process that Vitters is pursuing for the GNOME FTP server. Namely, allowing secure uploads, with access control measures limiting what individual users can upload, and without requiring shell access on the receiving machine. Anvin's work is the kernel.org upload tool, or "kup." It includes an upload client called kup and a server-side script called kup-server.

The kup-server script needs to be modified for each user account, to specify the pathnames for allowable uploads, Git trees, lock files, and the user's public keyring. The keyring file is read-only, and is used to verify signatures; kup-server checks that uploads are signed with a trusted OpenPGP key. Establishing the trust between the identities and keys was one of the lengthiest parts of bringing kernel.org back online.

Beyond simply utilizing a custom client instead of rsync, there are other differences, between kup and the GNOME.org proposal. The kup client contains multiple levels of sanity checks, even dry-running the requested commands in order to trap errors. It also supports other file operations in addition to uploads, including deletion, moving/renaming files, and creating links.

In addition, David Woodhouse said that he has set up a two-factor authentication scheme for kernel.org that has not yet been rolled out. It requires that users have both an SSH key and a one-time-password (OTP) generated by Google Authenticator. Google Authenticator includes a PAM module to secure applications, and a collection of OTP generators for various mobile phone OSes.

As of press time, Vitters told the list that he had gotten in touch with Anvin to follow-up about kup, but has not yet announced any alterations to the GNOME FTP proposal based on those conversations. The kernel.org team learned the hard way how much hassle a break-in on an upload server can produce. Although other, somewhat lower-profile projects may not present nearly as enticing an objective for attackers, they can learn from its example, and it is good to see GNOME undertaking a preventative lock-down. One hopes that the project is also taking steps to ensure that its infrastructure is not already compromised as well.

Ultimately a more secure infrastructure may inconvenience package maintainers, system administrators, and developers that (like Strode) prefer a less formal structure, but for a project of any size, what other choice is there? Package uploading and verification requires a security policy; it is too inviting of a target to not lock it down.