On August 14, Fedora leader Paul Frields sent out a terse announcement
"an issue in the infrastructure systems" supporting the project. This
"issue" could lead to some service outages, for which he apologized. Also
included in the note was this ominous warning:
We're still assessing the end-user impact of the situation, but as
a precaution, we recommend you not download or update any
additional packages on your Fedora systems.
As this article is written (August 20, just barely in time for the LWN
weekly publication deadline), there have been a couple of uninformative
updates, but the situation persists and nobody seems to know what is really
going on. The Fedora team, it would seem, is quite good at keeping secrets
when the need arises. As a result, Fedora users worldwide have spent
almost a full week wondering what has happened and whether they need to be
worried about it.
In such a situation, there is a delightful amount of space for wild
speculation. Your editor does not usually start his drinking binge until
after publication, but, for the purposes of interpreting the following, one
should assume that it was already well underway. This "issue" could be
explained by any of the following:
- Maybe a Fedora developer - on a drinking binge of his own, perhaps -
tripped over a power cord. The resulting mess not only deprived
an important server of power, but said developer, on his way toward
the floor, managed to take the entire rack down with him. Ever since,
the infrastructure team has been trying to reassemble a set of working
systems from the rubble.
- Last month, Fedora slipped a small patch into gcc designed to ensure
that the results from the most recent board election - where one slot
went to a candidate who was not a Red Hat employee - would never be
repeated. But the patch was botched, and most mathematical operations
in gcc-compiled programs have been returning random numbers ever since.
Now the Fedora team is trying to quietly replace the broken binaries
before anybody notices.
- It turns out that the rights to the Fedora name had never actually
been secured, and the real owner got an injunction shutting the
project down. As soon as all the branding has been changed, Fedora
will be reborn as Leopard-Skin Pillbox Hat Linux. Just wait until you
see the new desktop themes.
- The package signing key has been compromised, as have the build
servers. For the last six months, every version of Firefox shipped
by Fedora has reported account names, passwords, and credit card
numbers to a server located on a ship in international waters near
Colombia. The openssh client has been similarly modified. The Fedora
team has been slow to get an explanation out because it takes time to
relocate your home and family to an undisclosed location on a
- A vulnerability in RPM has enabled the creation of a large ecosystem
of hostile mirrors operated by competing criminal groups. Most Fedora
users have been installing compromised updates for the last year or
- No less than three Fedora system administrators turned out to be the
type of people who will give out
their password for a bar of chocolate. The provider of sweets
really only wanted to fix the longstanding claws-mail dependency
problems in Rawhide, but the project hit the panic button anyway.
- The Fedora team simply wanted to take a vacation in an undisclosed
location on a different continent and didn't want to deal with a bunch
of email on their return.
The real point of this being, of course, that none of us know what is going
on, creating a situation described by Alan
Cox as "leaving people in the dark assuming the worst - a very bad
way to create long term trust." Distributors occupy a crucial part
of our ecosystem; they absolutely need to have the trust of their
users. There is just too much that can go wrong at that level.
One can only assume that something fairly serious has happened. By all
accounts, the Fedora team has been working flat-out to get things resolved
as quickly as possible; they seem to be doing an exceptional job under a
great deal of pressure. They have undoubtedly earned a big round of thanks
- and lots of beers - from the Fedora community as a whole.
But Fedora's leadership appears to have failed here. If Fedora users need to be
concerned about the software running on their systems, they should have
been told by now. If they can relax and stop worrying, they should have
been told that as well. Instead, the Fedora user community has been left
wondering for nearly a week while the infrastructure they count on is torn
down and rebuilt from the beginning. Given that, Fedora users have shown a
tremendous amount of patience and restraint; the user community clearly has
a high degree of confidence in the project in general, and has been willing
to wait until the project is ready to come clean.
To retain that confidence, the Fedora project will have to tell the full
story in a clear manner - and sooner would certainly be better. A good
explanation of why Fedora users were made to wait so long before hearing
anything about how this "infrastructure issue" affects them will also be
needed. Fedora users are concerned about what has happened so far, but
their real response will be determined by what Fedora does next.
Comments (38 posted)
Standards like POSIX are meant to make life easier for application developers
by providing rules on the semantics of system calls for multiple different
platforms. Sometimes, though, operating system developers decide to change
the behavior of their platform—with full knowledge that it breaks
compatibility—for various reasons. This requires
application developers to notice the change and take appropriate action;
not doing so can lead to a security hole like the one found in the Postfix
mail transfer agent (MTA) recently.
The behavior of links, created using the link() system call—on
Linux, Solaris, and IRIX—is what tripped up Postfix. In particular, what
happens when a hard link is made to a symbolic link. Many long-time UNIX
hackers don't realize that you can even do that, nor what to expect if you
do. Postfix relied on a particular, standard-specified behavior that many
operating systems, including early versions of Linux, follow.
Links can be a somewhat confusing, or possibly unknown, part of UNIX-like
filesystems, so a bit of explanation is in order. A link created with
link()—also known as a hard link—is an alias for a
particular file. It simply gives an additional name by which a particular
chunk of bytes on the disk can be referenced. For example:
creates a second entry in the filesystem (called /link/to/foo
which points to the same file as
. The file being linked to must exist and reside on
the same filesystem as the link.
Symbolic links, on the other hand, are aliases of a different sort. A
symbolic link creates a new entry (e.g. inode) in the filesystem which
contains the path of the linked-to file as a string. There is no
requirement that the file exist or be on the same filesystem—the only
real requirement is that the path conform to standard pathname rules.
The symlink() system call is used to create them:
Both symbolic links and hard links can also be created from the command line
using the ln
command (adding a -s
option for symbolic
So, when making a hard link to a symbolic link, there are two choices:
either follow the symbolic link to its, possibly nonexistent, target and
link to that or
link to the symbolic link inode itself. POSIX requires that the symbolic
link be fully resolved to an actual existing file, which is the behavior
that Postfix relies upon.
sequence of events is lost in the mists of time, but Linux changed to
non-standard behavior—at least partially for compatibility with
Solaris—in kernel version 1.3.56 (which was released in January
1996). Some discussion
prior to that change adds an additional reason for it: user space has no
way to make a link to a symbolic link without it. Some saw that as a flaw
in the interface and proposed the change. An application developer that
original behavior would be able to implement it by resolving any symbolic
links before making the hard link.
To further complicate things, it appears that the POSIX behavior was
restored in the 2.1 development series, only to be changed back in late 1998.
This change led to the comments currently in fs/namei.c for
the function implementing link():
* Hardlinks are often used in delicate situations. We avoid
* security-related surprises by not following symlinks on the
* newname. --KAB
* We don't follow them on the oldname either to be compatible
* with linux 2.0, and to avoid hard-linking to directories
* and other special files. --ADM
is the file being linked to and newname
is the name being
created. For the curious, KAB is Kevin Buhr and ADM is Alan Modra.
Unfortunately, according to Postfix author Wietse Venema, the
link(2) man page
didn't change until sometime in 2006. This makes it fairly difficult for
application developers to learn about the change, especially because they
may not follow kernel development closely.
Postfix allows root-owned symbolic links to be used as the target for local
mail delivery, specifically to handle things like /dev/null on
Solaris, which is a symbolic link.
Because an attacker can make a link to a root-owned symbolic link on
vulnerable systems, Postfix can get confused and deliver mail to files that
This can lead to privilege escalation (via executing code as root)
by making a hard link to
a symbolic link of an init script (CVE-2008-2936).
As Venema outlines in the Postfix
security advisory, the problem can be resolved by requiring that
symbolic links used for local delivery reside in a directory that is only
writeable by root.
It is not a perfect solution, though: "This change will break
legitimate configurations that deliver mail
to a symbolic link in a directory with less restrictive
There are other workarounds for people who don't want to use the provided
patch to Postfix. Protecting the mail spool directory is one solution;
Venema provides a script to use to do that. Some systems can be configured
to disallow links to files owned by others, which is another way to avoid
This issue has given Postfix a bit of a black eye, but that
is rather unfair.
The problem was found by a SUSE code inspection, but it has existed in
certain kinds of Linux installations of Postfix for a long time. It could
be argued that testing should have found it—there is a simple test for
vulnerable systems—but relying on documented behavior that is part of
an important standard that Linux is said to support is not completely
unreasonable either. It is likely that the full implications of not
supporting the standard were not completely understood until recently.
Linux was still in its infancy when the original change went in. One would
like to think that a change of that type today would be nearly impossible
because it breaks the kernel's user-space interface. If it were to happen,
somehow, the resulting hue and cry would be loud enough that application
developers would hear. But that's for intentional changes; a bug
introduced into a dark corner of the kernel's API might go unnoticed for
quite some time. Hopefully, none of those lingers for ten years before
Update: The original article referred to CVE-2008-2937
as also being a consequence of the link issue, which it is not. It is an
unrelated issue that was found during the same code review.
Comments (40 posted)
The Java Model Railroad
(JMRI) project is not one to sit at the
top of the Debian popularity contest results; it provides tools for model
railroad enthusiasts. But the legal wrangling around JMRI has made it one
of the more important projects in our community at this time. JMRI has
suffered some legal setbacks, but much of that was turned around by the US
Federal Circuit Court of Appeals on August 13. The result is a
vindication for much of the legal reasoning behind free software licenses.
JMRI was charged with patent
infringement back in 2006. As part of the legal counterattack, JMRI
developer Robert Jacobson charged patent holders Michael Katzer and Kamind
Associates, Inc. with copyright infringement for its use of JMRI code. The
Federal District Court in this case had concluded that the terms of the
Artistic License were contract terms, and not condition on the copyright
That ruling was seen as a major setback. The authors of free software
licenses have gone to great lengths to restrict themselves to copyright
licensing and to avoid contract law altogether. There are a couple of
important reasons for this:
- A contract is only binding if all parties have voluntarily entered
into it. There have been mutterings from some corners for years that
licenses like the GPL are not truly enforceable because the recipients
of software under those licenses have never signed the relevant
contracts. Such mutterings have become relatively hard to hear, but
they are still out there. A software license is,
instead, a unilateral grant of privilege which does not require
agreement. As such, it should be easier to enforce.
- Violation of the terms of a contract sets up the guilty party to be
sued for damages. Copyright infringement, instead, allows for
injunctive relief, allowing the copyright owner to immediately shut
down the infringing activity. Many of those who would ignore the
terms of free software licenses fear injunctions far more than they
fear suits for damages.
Both points are crucial. If you look at clause 5 of the GNU General Public
License (version 2, in this case), you read:
You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify
or distribute the Program or its derivative works. These actions
are prohibited by law if you do not accept this License.
Anybody who distributes a copyrighted work will be doing so in violation of
the author's exclusive rights. If a distributor has a license from the
owner, though, then this distributor has a legal defense. The question
raised in this case was, in summary, this: if somebody distributes free
software without adhering to the terms of the license, does that somebody
still have a license at all? The District Court ruled that this
person did, indeed, still have a license to distribute the software, though
they might be liable for damages for not having followed all of the terms.
The Appeals Court, instead, said that failure to hold to the conditions
meant that the license simply did not exist; distributing free software in
a manner contrary to its license is copyright infringement, not breach of
This decision was reached in a sufficiently high court that the
conversation should be finished in the United States; we now have a
high-level legal precedent that software licenses are licenses, and
that they can be enforced with injunctions. In US-style law, precedents
are everything; the absence of a clear precedent always causes a certain
degree of legal uncertainty. We now have that precedent; as a result,
anybody seeking to enforce a free software
license in the US is now standing on firmer ground.
There are some other interesting conclusions to be drawn from this ruling.
Copyright law in the US does not recognize any sort of moral rights to
copyrighted works; it is, in classic American style, all about the
protection of economic rights. Some have argued that, since free software
is, well, free of charge, there is no economic harm in violating its
licenses, and, thus, copyright law has nothing to say. But the Appeals
Court saw things differently, stating that there was a clear economic
interest in the Artistic license:
The clear language of the Artistic License creates conditions to
protect the economic rights at issue in the granting of a public
license. These conditions govern the rights to modify and
distribute the computer programs and files included in the
downloadable software package. The attribution and modification
transparency requirements directly serve to drive traffic to the
open source incubation page and to inform downstream users of the
project, which is a significant economic goal of the copyright
holder that the law will enforce.
So the reasoning that free software licenses are unenforceable due to the
lack of an economic interest fails to hold water. Similarly, the
interesting idea that free software license incompatibility does not really
exist, recently promoted on
LWN by Brian Cantrill, seems unlikely to stand up to serious scrutiny.
Some voices on the net have worried that this ruling could also give
sharper teeth to exploitive proprietary end user license agreements. The
Electronic Frontier Foundation is one
While we're pleased to see a panel of learned judges endorse the
legal foundations of the open source software paradigm, the
decision may also encourage proprietary software vendors who
frequently fill their "end user license agreements" with
restrictions that are denominated as "conditions" on the license.
If violating a "condition" in a EULA results in copyright
infringement liability, what's to stop a software vendor from
imposing conditions that are unrelated to copyright law (e.g. an
agreement not to disparage the copyright owner, or to wear pink
bunny ears on Tuesdays), or even antithetical to copyright law
(e.g. a waiver of fair use rights)?
If this comes to pass, restrictions on reverse engineering, publication of
reviews, lack of bunny ears, etc. may, indeed, become easier to enforce. Such an
outcome would not
necessarily be a bad thing for users of free software, though. If
anything, it will simply make the value of freedom that much more clear.
Finally, it is worth noting well that this outcome did not just happen on
its own. Behind the scenes, concerned lawyers from groups like the
Stanford Center for Internet and Society and the Electronic Frontier
Foundation, who have understood all along what was at stake here, have
put in a great deal of work to get this ruling. They were successful
despite the fact that the old Artistic License was not the strongest
position to be arguing from. Many of us would prefer to
not have to think about legal issues much of the time. But we should be
happy and grateful that some very capable people have been willing to put
in the effort to defend our rights in cases like this one.
(The full ruling is available in PDF format,
plain text on Groklaw).
Comments (34 posted)
Page editor: Jonathan Corbet
Inside this week's LWN.net Weekly Edition
- Security: Injunction lifted against MIT students; New vulnerabilities in amarok, postfix, yum-rhn-plugin
- Kernel: Triggers: less busy busy-waiting; Regulating wireless devices; Tangled up in threads.
- Distributions: In defense of Ubuntu; Pie Box Enterprise Linux 4AS U7; Intrepid Ibex Alpha-4; Debian news; Fedora issues; EOL for SUSE Linux 10.1
- Development: Desktop talks from LinuxWorld 2008 Conference, new versions of libnetfilter_log, PorkBind, Ecasound, Mercator, Pogolyn, Elisa, CLAM, Task Coach, ETS, AgileTrack, Tail, dlib, Parrot, bbfreeze, HDF5 for Python, PyUseCase, TextTest, GIT, Mercurial.
- Press: What Linux Will Look Like In 2012, Akademy mobile and embedded day, LinuxWorld coverage, IBM's Software Appliance Initiatives, interviews with Torvalds, Debian leaders, Corbet, Pyne and van Rossum, improving Firefox Find, greener IT departments, Dell's latest laptops.
- Announcements: Lessig on the model train decision, MIT Students ungagged, Canonical joins LF, Perl Foundationon Jacobsen v. Katzer, SFLC Guide to GPL Compliance, Fedora scholarship program, Microsoft and Novell renew agreement, RailsConf report, PostgreSQL West cfp, FOSS.IN/2008, Linux Plumbers speakers, pyArkansas, Java Game Tome.