Some effects of the kernel.org break-in were immediate, but
others have rippled out more slowly. One example of the latter is the
system administration team taking a hard look at the security of its own
infrastructure. The team is drafting a plan to tighten down access to its
servers and looking at the development processes of the project
with the module upload process.
Olav Vitters sent an email to the GNOME desktop-devel list on November
10 that outlines a plan to preemptively secure the master.gnome.org server
against some preventable compromise paths. His root concern was that more than
350 GNOME developers have shell access, which provides a large attack
surface even though most (if not all) of those accounts are used only to
upload tar packages — a process that could be more securely performed
in other ways. Considering the lengthy process that the kernel.org team
had to undertake to restore the server after its compromise, the concern is
a very real one.
The GNOME project maintains a variety of public-facing services on different machines (Bugzilla, mailing lists, wikis, web sites, etc.), which are managed by one full-time system administrator paid by the GNOME Foundation, and assisted by a small team of volunteers. Vitters is one of those volunteers. His message to the list, which he styled as an IETF-like "request for comments" (RFC), targeted just one service: the FTP server used to distribute GNOME tarballs.
In the current FTP release process, package maintainers upload tarballs to a GNOME project server using SCP, then log in over SSH and execute an "install-module" script. The script calculates the checksum(s) of the newly-uploaded package(s), copies them to the proper location in the public FTP directory, and triggers another script that starts an FTP sync between the GNOME project server and the "real" master FTP server (which is located off-site) and the official mirrors.
Restricted SSH and DOAP
Due to the interactive SSH step, Vitters said, if any of the 350-or-so
package maintainers has their personal machine compromised, an attacker
could gain shell access to the GNOME project FTP server. That was the path
used by the kernel.org attackers, who then presumably used a local
privilege escalation to completely compromise the machine. It is very
probable that one or more of the GNOME user's machines has been
compromised, but that has evidently not led to unauthorized access to the GNOME
Vitters proposed a replacement module-upload plan, in which the initial file transfer would be performed with rrsync (the Perl-based program that restricts rsync to a pre-determined subdirectory) over an SSH tunnel. The user would then call a revised version of the install-module script remotely over SSH, where the SSH daemon on the server was configured to only permit execution of that script, rather than a full shell.
The server-side scripts already in use would also be modified to check the user account that uploaded the new file against a list of approved uploaders for the module, taken from the module's Description of a Project (DOAP) file. DOAP is an RDF format used to define the semantics of a project and its team; GNOME already stores DOAP files for each module in the project's Git repository.
The primary benefit of the plan is that it eliminates the need for the 350+ shell accounts. Even if a module maintainer's home machine is compromised, keeping the server locked down so that remote shell access is unavailable makes it more difficult to wreak large-scale havoc. It would still be possible to replace a package, but the identity check performed against DOAP would, in theory, mean that the attacker could only replace a small subset of packages for the majority of accounts. This is an improvement, because currently anyone with a shell account on the server can upload any package.
Vitters posed several questions for the list, starting with whether or
not the proposed process was so much more arduous than the existing system
that it would meet with resistance. But he also pointed out potential
technical hurdles such as uncertainty over the reliability of rsync,
whether rsync could be used to delete files, and the fact that although
most GNOME modules have a DOAP file in Git, those GNOME modules
that are not tracked using Git do not.
Reaction and feedback
Maciej Marcin Piechotka asked whether scp might be preferable to rsync, and suggested that the Git DOAP files are not secure, because anyone with Git access could upload a new DOAP file — a change less likely to be noticed by others than is the upload of a new tarball. Vitters replied that he considered, but ruled out, scp and sftp for the upload step because both permitted more file operations than rsync alone. In the interest of "just allowing exactly what is needed," he said, rsync was the better fit. "I don't want people to be able to modify anything [other] than uploading a tarball." He confirmed the vulnerability of the DOAP file in Git, and proposed modifying the upload scripts to notify existing maintainers whenever the DOAP file is modified.
Ray Strode then weighed in against restricting uploads to individuals in the DOAP file on the grounds that there are a lot of less-actively-maintained modules, and an unpredictable set of users who periodically need to make releases of them. "I think the lack of fine grained ACLs for this sort of thing is one part of GNOME that work better than projects that try to lock things down more aggressively," he said. Matthias Clasen and Bastien Nocera added that they were two such users, who occasionally update maintenance-mode modules when a change in such a module is mandated by an update to one of their own modules.
Vitters then suggested that the allowed-user-list could include the maintainers of each module plus all members of the release team, since both Clasen and Nocera are on the release team. Clasen and Nocera seemed happy with that adjustment, but Strode did not:
What I'm saying is there shouldn't be a lot of process involved here, because people informally help out pretty frequently. If you keep track of it already you can see probably see how often it happens by looking at the stats, I guess. A number of modules are sort of "maintained by the masses". It's not just release-team members who do the releases, either.
Strode's concern is not easily reconcilable with the overall plan,
because it boils down to a philosophical, not a technical question: how
locked-down is too locked-down.
lax security policy towards the less-frequently-updated modules
simplifies the work for the administrators and allows more people to
contribute, but the less-active modules are an easier target for
attackers. Inserting malicious code into an actively-developed module
is likely to get noticed in short order; by instead inserting it into a
nearly-forgotten Makefile that is still pulled in and executed when
one accomplishes the same goal with less risk of getting caught.
Alan Cox suggested that the team take a look at the tools used by the kernel.org team, which went through much the same locking-down process in September and October in response to the August break-in.
Lessons from kernel.org
The kernel.org lockdown began with the same concern as Vitters' RFC: eliminating the several-hundred shell accounts used mainly for uploading. Since the break-in, H. Peter Anvin has developed a suite of utilities for kernel.org that implement much the same process that Vitters is pursuing for the GNOME FTP server. Namely, allowing secure uploads, with access control measures limiting what individual users can upload, and without requiring shell access on the receiving machine. Anvin's work is the kernel.org upload tool, or "kup." It includes an upload client called kup and a server-side script called kup-server.
The kup-server script needs to be modified for each user
account, to specify the pathnames for allowable uploads, Git trees, lock
files, and the user's public keyring. The keyring file is read-only, and
is used to verify signatures; kup-server checks that uploads are
signed with a trusted OpenPGP key. Establishing the trust between the
identities and keys was one of the lengthiest parts of bringing kernel.org
Beyond simply utilizing a custom client instead of rsync, there are other differences, between kup and the GNOME.org proposal. The kup client contains multiple levels of sanity checks, even dry-running the requested commands in order to trap errors. It also supports other file operations in addition to uploads, including deletion, moving/renaming files, and creating links.
In addition, David Woodhouse said that he has set up a two-factor authentication scheme for kernel.org that has not yet been rolled out. It requires that users have both an SSH key and a one-time-password (OTP) generated by Google Authenticator. Google Authenticator includes a PAM module to secure applications, and a collection of OTP generators for various mobile phone OSes.
As of press time, Vitters told
the list that he had gotten in touch with Anvin to follow-up about
kup, but has not yet announced any alterations to the GNOME FTP
proposal based on those conversations. The kernel.org team learned the
hard way how much hassle a break-in on an upload server can produce.
Although other, somewhat lower-profile projects may not present nearly as
enticing an objective for attackers, they can learn from its example, and
it is good to see GNOME undertaking a preventative lock-down. One hopes
that the project is also taking steps to ensure that its infrastructure is
not already compromised as well.
Ultimately a more secure infrastructure may inconvenience package
maintainers, system administrators, and developers that (like Strode)
prefer a less formal structure, but for a project of any size, what other
choice is there? Package uploading and verification requires a security
policy; it is too inviting of a target to not lock it down.
to post comments)