Like most community-run events, the second SouthEast LinuxFest (SELF)
featured the standard set of positive talks on Linux and open source. It
also featured a somewhat more controversial talk about failures to get some
features merged into the Linux kernel by Ryan "icculus" Gordon.
The talk was delivered without slides, and Gordon started by admitting
the talk was biased — unlikely to win many friends in the kernel
community. The focus of the talk, which was not terribly obvious from the
was on several high-profile attempts to add new features into the Linux
kernel that failed, due — according to Gordon — to kernel
politics or personality conflicts. He said he had spoken to a number of
kernel developers who experienced failure, but few were willing to go "on
the record," for the talk. As examples he used his own experiences, Eric
S. Raymond's CML2, Con
with the Completely Fair Scheduler, along with Hans Reiser and the Reiser4 filesystem.
Gordon is behind icculus.org, and says
that he spends most of his time porting video games to Linux. In the course
of working with Linux, Gordon says that he discovered that "Linux
sucks at a lot of important tasks." He noted that Apple has solved a
number of the things that Linux does poorly (though he ceded that Mac OS X also does many things badly), and that Linux developers should be "stealing some stuff" from Apple. Gordon pointed to the Time Machine backups and universal binaries that allow users to install software on PowerPC or Intel-based machines and have them "just work."
Gordon wasn't using universal binaries as an idle example. Gordon attempted to solve that problem himself by creating FatELF, a universal binary format for Linux. He described the process of creating the patch, sending it to the kernel list for acceptance, and expecting success. Instead of succeeding, Gordon said it was a "spectacular failure."
The problem? Gordon says that he misinterpreted a response from Alan Cox as being in favor of the patch, when actually he was against it. According to Gordon, "the worst thing you can do is have a kernel maintainer tell you what they don't like and ignore it. Once Alan Cox was openly hostile, people came out of the woodwork." He then drew an analogy between the kernel community and the movie Mean Girls, likening Cox to one of the popular girls and other community members as following Cox's example. Gordon says that the kernel community has a "herd mentality" and that "you can't move the kernel maintainers."
After recounting his own experience with FatELF, Gordon talked about
Raymond and the problems getting CML2 — a replacement for the kernel
build system — into the Linux kernel. Gordon highlighted the fact
that CML2 going into the kernel seemed a foregone conclusion to Raymond,
and that it was originally supposed to go into the 2.5.1 kernel. He also
talked at length about Raymond and Linus Torvalds's discussions about CML2
and Torvalds's general agreement that CML2 could go into the kernel.
Gordon said that the kernel community was hostile towards Python and
Raymond, but little was mentioned about Raymond's sometimes caustic
responses to the kernel developers. Gordon also mentioned a little, but
very little, of the technical problems of CML2,
such as 30-second wait time to compile its kernel-rules.cml into a binary
format for use with the cmlconfigure program. He did discuss the
— perhaps not entirely reasonable — objection that CML2
consisted of a major change to the kernel where the maintainers typically
prefer a set of small incremental changes. He wondered: how does one implement a new configuration language as an incremental change?
One might question the wisdom of using Hans Reiser as an example of the
kernel development process gone wrong, as Gordon did. Reiser had a
contentious relationship with the kernel community to begin with, and the
fact that it was necessary to cite correspondence from Reiser's cell tends
to undermine his credibility. But Gordon is to be credited with, at least, being thorough in interviewing his sources. Gordon showed several letters from Reiser gathered over months of correspondence about the failure to get Reiser4 into the mainline kernel.
He described the problems that Reiser had in attempting to get Reiser4
into the kernel. In an unfortunate analogy, he said that Reiser failed to
get a "fair trial" from the kernel community in considering Reiser4. Gordon
also hinted that corporate interests may have sabotaged Reiser4 from being
adapted into mainline — a longstanding
contention of Reiser's — because of conflicting interests on behalf of the
maintainers, though he cited no evidence for it. Gordon painted the kernel community as abusive towards Reiser, but omitted any mention that Reiser could also be caustic in return. Certainly, Gordon's point that the FAQ on KernelNewbies is unnecessarily personal is well-taken.
But little was said about any of the real technical problems with
Reiser4. Personality issues aside, real technical problems stood
(and still stand) in the way of merging Reiser4 into the mainline kernel.
Due to time, Gordon rushed through the discussion of Kolivas's failure
to get the Completely Fair Scheduler into the kernel. He characterized
Kolivas's treatment as "rude," and suggested that it was
particularly bad treatment that Kolivas's ideas made it into the kernel but his actual code didn't.
Gordon ended his talk by throwing out a few ideas to improve the
kernel development process. He suggested that the audience, and other
Linux users, join the kernel mailing list and lobby for features that
they want. He also challenged the idea that developers should have a
"thick skin" to participate in kernel development, and suggested that
the atmosphere of lkml needs to improve.
Despite the focus on the "spectacular failures," Gordon did
acknowledge that this was a sporadic problem at worst — not a
systemic failure. He also took great pains to say, both during and after
the talk, that everyone he spoke with held Andrew Morton in the highest regard. Gordon said that developers should "study Andrew Morton" with great intensity.
The talk was interesting because it provides a different view of the kernel development process than is normally given to general audiences. The kernel development community, and larger FOSS development community, has an understanding of the development process as imperfect. It is well-known that it can be political and personal, as Torvalds himself pointed out in response to Kolivas's departure. The SELF audience, by and large, was composed of Linux enthusiasts that are far removed from the development process. It will be interesting to see if any of the audience decides to take Gordon's advice and begin lobbying the kernel list.
Kernel development is often held up as a shining example of the open source development process to larger audiences. Gordon's talk presented a narrative that demonstrates the less pleasant side of kernel hacking, and the disappointment that developers feel when their contributions are rejected. It might have been more valuable by presenting a less biased view, but it is a story that should be heard.
to post comments)