LWN.net Weekly Edition for August 21, 2008

One week of infrastructure issues

On August 14, Fedora leader Paul Frields sent out a terse announcement regarding "an issue in the infrastructure systems" supporting the project. This "issue" could lead to some service outages, for which he apologized. Also included in the note was this ominous warning:

As this article is written (August 20, just barely in time for the LWN weekly publication deadline), there have been a couple of uninformative updates, but the situation persists and nobody seems to know what is really going on. The Fedora team, it would seem, is quite good at keeping secrets when the need arises. As a result, Fedora users worldwide have spent almost a full week wondering what has happened and whether they need to be worried about it.

In such a situation, there is a delightful amount of space for wild speculation. Your editor does not usually start his drinking binge until after publication, but, for the purposes of interpreting the following, one should assume that it was already well underway. This "issue" could be explained by any of the following:

• Maybe a Fedora developer - on a drinking binge of his own, perhaps - tripped over a power cord. The resulting mess not only deprived an important server of power, but said developer, on his way toward the floor, managed to take the entire rack down with him. Ever since, the infrastructure team has been trying to reassemble a set of working systems from the rubble.

• Last month, Fedora slipped a small patch into gcc designed to ensure that the results from the most recent board election - where one slot went to a candidate who was not a Red Hat employee - would never be repeated. But the patch was botched, and most mathematical operations in gcc-compiled programs have been returning random numbers ever since. Now the Fedora team is trying to quietly replace the broken binaries before anybody notices.

• It turns out that the rights to the Fedora name had never actually been secured, and the real owner got an injunction shutting the project down. As soon as all the branding has been changed, Fedora will be reborn as Leopard-Skin Pillbox Hat Linux. Just wait until you see the new desktop themes.

• The package signing key has been compromised, as have the build servers. For the last six months, every version of Firefox shipped by Fedora has reported account names, passwords, and credit card numbers to a server located on a ship in international waters near Colombia. The openssh client has been similarly modified. The Fedora team has been slow to get an explanation out because it takes time to relocate your home and family to an undisclosed location on a different continent.

• A vulnerability in RPM has enabled the creation of a large ecosystem of hostile mirrors operated by competing criminal groups. Most Fedora users have been installing compromised updates for the last year or so.

• No less than three Fedora system administrators turned out to be the type of people who will give out their password for a bar of chocolate. The provider of sweets really only wanted to fix the longstanding claws-mail dependency problems in Rawhide, but the project hit the panic button anyway.

• The Fedora team simply wanted to take a vacation in an undisclosed location on a different continent and didn't want to deal with a bunch of email on their return.

The real point of this being, of course, that none of us know what is going on, creating a situation described by Alan Cox as "leaving people in the dark assuming the worst - a very bad way to create long term trust." Distributors occupy a crucial part of our ecosystem; they absolutely need to have the trust of their users. There is just too much that can go wrong at that level.

One can only assume that something fairly serious has happened. By all accounts, the Fedora team has been working flat-out to get things resolved as quickly as possible; they seem to be doing an exceptional job under a great deal of pressure. They have undoubtedly earned a big round of thanks - and lots of beers - from the Fedora community as a whole.

But Fedora's leadership appears to have failed here. If Fedora users need to be concerned about the software running on their systems, they should have been told by now. If they can relax and stop worrying, they should have been told that as well. Instead, the Fedora user community has been left wondering for nearly a week while the infrastructure they count on is torn down and rebuilt from the beginning. Given that, Fedora users have shown a tremendous amount of patience and restraint; the user community clearly has a high degree of confidence in the project in general, and has been willing to wait until the project is ready to come clean.

To retain that confidence, the Fedora project will have to tell the full story in a clear manner - and sooner would certainly be better. A good explanation of why Fedora users were made to wait so long before hearing anything about how this "infrastructure issue" affects them will also be needed. Fedora users are concerned about what has happened so far, but their real response will be determined by what Fedora does next.

Comments (38 posted)

Standards, the kernel, and Postfix

Standards like POSIX are meant to make life easier for application developers by providing rules on the semantics of system calls for multiple different platforms. Sometimes, though, operating system developers decide to change the behavior of their platform—with full knowledge that it breaks compatibility—for various reasons. This requires application developers to notice the change and take appropriate action; not doing so can lead to a security hole like the one found in the Postfix mail transfer agent (MTA) recently.

The behavior of links, created using the link() system call—on Linux, Solaris, and IRIX—is what tripped up Postfix. In particular, what happens when a hard link is made to a symbolic link. Many long-time UNIX hackers don't realize that you can even do that, nor what to expect if you do. Postfix relied on a particular, standard-specified behavior that many operating systems, including early versions of Linux, follow.

Links can be a somewhat confusing, or possibly unknown, part of UNIX-like filesystems, so a bit of explanation is in order. A link created with link()—also known as a hard link—is an alias for a particular file. It simply gives an additional name by which a particular chunk of bytes on the disk can be referenced. For example:

    link("/path/to/foo", "/link/to/foo");

creates a second entry in the filesystem (called /link/to/foo) which points to the same file as /path/to/foo. The file being linked to must exist and reside on the same filesystem as the link.

Symbolic links, on the other hand, are aliases of a different sort. A symbolic link creates a new entry (e.g. inode) in the filesystem which contains the path of the linked-to file as a string. There is no requirement that the file exist or be on the same filesystem—the only real requirement is that the path conform to standard pathname rules. The symlink() system call is used to create them:

    symlink("/path/to/foo", "/symlink/to/foo");

Both symbolic links and hard links can also be created from the command line using the ln command (adding a -s option for symbolic links).

So, when making a hard link to a symbolic link, there are two choices: either follow the symbolic link to its, possibly nonexistent, target and link to that or link to the symbolic link inode itself. POSIX requires that the symbolic link be fully resolved to an actual existing file, which is the behavior that Postfix relies upon.

The exact sequence of events is lost in the mists of time, but Linux changed to non-standard behavior—at least partially for compatibility with Solaris—in kernel version 1.3.56 (which was released in January 1996). Some discussion prior to that change adds an additional reason for it: user space has no way to make a link to a symbolic link without it. Some saw that as a flaw in the interface and proposed the change. An application developer that wanted the original behavior would be able to implement it by resolving any symbolic links before making the hard link.

To further complicate things, it appears that the POSIX behavior was restored in the 2.1 development series, only to be changed back in late 1998. This change led to the comments currently in fs/namei.c for the function implementing link():

    /*
     * Hardlinks are often used in delicate situations.  We avoid
     * security-related surprises by not following symlinks on the
     * newname.  --KAB
     *
     * We don't follow them on the oldname either to be compatible
     * with linux 2.0, and to avoid hard-linking to directories
     * and other special files.  --ADM
     */

Where oldname is the file being linked to and newname is the name being created. For the curious, KAB is Kevin Buhr and ADM is Alan Modra.

Unfortunately, according to Postfix author Wietse Venema, the link(2) man page didn't change until sometime in 2006. This makes it fairly difficult for application developers to learn about the change, especially because they may not follow kernel development closely.

Postfix allows root-owned symbolic links to be used as the target for local mail delivery, specifically to handle things like /dev/null on Solaris, which is a symbolic link. Because an attacker can make a link to a root-owned symbolic link on vulnerable systems, Postfix can get confused and deliver mail to files that it shouldn't. This can lead to privilege escalation (via executing code as root) by making a hard link to a symbolic link of an init script (CVE-2008-2936).

As Venema outlines in the Postfix security advisory, the problem can be resolved by requiring that symbolic links used for local delivery reside in a directory that is only writeable by root. It is not a perfect solution, though: "This change will break legitimate configurations that deliver mail to a symbolic link in a directory with less restrictive permissions." There are other workarounds for people who don't want to use the provided patch to Postfix. Protecting the mail spool directory is one solution; Venema provides a script to use to do that. Some systems can be configured to disallow links to files owned by others, which is another way to avoid the problem.

This issue has given Postfix a bit of a black eye, but that is rather unfair. The problem was found by a SUSE code inspection, but it has existed in certain kinds of Linux installations of Postfix for a long time. It could be argued that testing should have found it—there is a simple test for vulnerable systems—but relying on documented behavior that is part of an important standard that Linux is said to support is not completely unreasonable either. It is likely that the full implications of not supporting the standard were not completely understood until recently.

Linux was still in its infancy when the original change went in. One would like to think that a change of that type today would be nearly impossible because it breaks the kernel's user-space interface. If it were to happen, somehow, the resulting hue and cry would be loud enough that application developers would hear. But that's for intentional changes; a bug introduced into a dark corner of the kernel's API might go unnoticed for quite some time. Hopefully, none of those lingers for ten years before being discovered.

Update: The original article referred to CVE-2008-2937 as also being a consequence of the link issue, which it is not. It is an unrelated issue that was found during the same code review.

Comments (40 posted)

Why the JMRI decision matters

The Java Model Railroad Interface (JMRI) project is not one to sit at the top of the Debian popularity contest results; it provides tools for model railroad enthusiasts. But the legal wrangling around JMRI has made it one of the more important projects in our community at this time. JMRI has suffered some legal setbacks, but much of that was turned around by the US Federal Circuit Court of Appeals on August 13. The result is a vindication for much of the legal reasoning behind free software licenses.

JMRI was charged with patent infringement back in 2006. As part of the legal counterattack, JMRI developer Robert Jacobson charged patent holders Michael Katzer and Kamind Associates, Inc. with copyright infringement for its use of JMRI code. The Federal District Court in this case had concluded that the terms of the Artistic License were contract terms, and not condition on the copyright license itself.

That ruling was seen as a major setback. The authors of free software licenses have gone to great lengths to restrict themselves to copyright licensing and to avoid contract law altogether. There are a couple of important reasons for this:

• A contract is only binding if all parties have voluntarily entered into it. There have been mutterings from some corners for years that licenses like the GPL are not truly enforceable because the recipients of software under those licenses have never signed the relevant contracts. Such mutterings have become relatively hard to hear, but they are still out there. A software license is, instead, a unilateral grant of privilege which does not require agreement. As such, it should be easier to enforce.

• Violation of the terms of a contract sets up the guilty party to be sued for damages. Copyright infringement, instead, allows for injunctive relief, allowing the copyright owner to immediately shut down the infringing activity. Many of those who would ignore the terms of free software licenses fear injunctions far more than they fear suits for damages.

Both points are crucial. If you look at clause 5 of the GNU General Public License (version 2, in this case), you read:

Anybody who distributes a copyrighted work will be doing so in violation of the author's exclusive rights. If a distributor has a license from the owner, though, then this distributor has a legal defense. The question raised in this case was, in summary, this: if somebody distributes free software without adhering to the terms of the license, does that somebody still have a license at all? The District Court ruled that this person did, indeed, still have a license to distribute the software, though they might be liable for damages for not having followed all of the terms. The Appeals Court, instead, said that failure to hold to the conditions meant that the license simply did not exist; distributing free software in a manner contrary to its license is copyright infringement, not breach of contract.

This decision was reached in a sufficiently high court that the conversation should be finished in the United States; we now have a high-level legal precedent that software licenses are licenses, and that they can be enforced with injunctions. In US-style law, precedents are everything; the absence of a clear precedent always causes a certain degree of legal uncertainty. We now have that precedent; as a result, anybody seeking to enforce a free software license in the US is now standing on firmer ground.

There are some other interesting conclusions to be drawn from this ruling. Copyright law in the US does not recognize any sort of moral rights to copyrighted works; it is, in classic American style, all about the protection of economic rights. Some have argued that, since free software is, well, free of charge, there is no economic harm in violating its licenses, and, thus, copyright law has nothing to say. But the Appeals Court saw things differently, stating that there was a clear economic interest in the Artistic license:

So the reasoning that free software licenses are unenforceable due to the lack of an economic interest fails to hold water. Similarly, the interesting idea that free software license incompatibility does not really exist, recently promoted on LWN by Brian Cantrill, seems unlikely to stand up to serious scrutiny.

Some voices on the net have worried that this ruling could also give sharper teeth to exploitive proprietary end user license agreements. The Electronic Frontier Foundation is one example:

If this comes to pass, restrictions on reverse engineering, publication of reviews, lack of bunny ears, etc. may, indeed, become easier to enforce. Such an outcome would not necessarily be a bad thing for users of free software, though. If anything, it will simply make the value of freedom that much more clear.

Finally, it is worth noting well that this outcome did not just happen on its own. Behind the scenes, concerned lawyers from groups like the Stanford Center for Internet and Society and the Electronic Frontier Foundation, who have understood all along what was at stake here, have put in a great deal of work to get this ruling. They were successful despite the fact that the old Artistic License was not the strongest position to be arguing from. Many of us would prefer to not have to think about legal issues much of the time. But we should be happy and grateful that some very capable people have been willing to put in the effort to defend our rights in cases like this one.

(The full ruling is available in PDF format, or in plain text on Groklaw).

Comments (34 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

• Security: Injunction lifted against MIT students; New vulnerabilities in amarok, postfix, yum-rhn-plugin
• Kernel: Triggers: less busy busy-waiting; Regulating wireless devices; Tangled up in threads.
• Distributions: In defense of Ubuntu; Pie Box Enterprise Linux 4AS U7; Intrepid Ibex Alpha-4; Debian news; Fedora issues; EOL for SUSE Linux 10.1
• Development: Desktop talks from LinuxWorld 2008 Conference, new versions of libnetfilter_log, PorkBind, Ecasound, Mercator, Pogolyn, Elisa, CLAM, Task Coach, ETS, AgileTrack, Tail, dlib, Parrot, bbfreeze, HDF5 for Python, PyUseCase, TextTest, GIT, Mercurial.
• Press: What Linux Will Look Like In 2012, Akademy mobile and embedded day, LinuxWorld coverage, IBM's Software Appliance Initiatives, interviews with Torvalds, Debian leaders, Corbet, Pyne and van Rossum, improving Firefox Find, greener IT departments, Dell's latest laptops.
• Announcements: Lessig on the model train decision, MIT Students ungagged, Canonical joins LF, Perl Foundationon Jacobsen v. Katzer, SFLC Guide to GPL Compliance, Fedora scholarship program, Microsoft and Novell renew agreement, RailsConf report, PostgreSQL West cfp, FOSS.IN/2008, Linux Plumbers speakers, pyArkansas, Java Game Tome.

Next page: Security>>