Among many in the real-time community, it is a matter of accepted faith
that a general-purpose kernel (such as the Linux kernel) cannot be expected
to perform properly in a situation where deterministic, real-time response
is required. Things may work most of the time, but one never knows when
such a kernel may get distracted for too long, with disastrous results for
real-time applications. On the other hand, general-purpose kernels do tend
to provide nicer programming environments than hard real-time kernels. So
real-time developers can be faced with a fundamental conflict:
deterministic response or a rich environment?
One longstanding attempt to resolve this conflict is RTLinux. At its core,
RTLinux is a small, real-time kernel without a great deal of
functionality. One of the things RTLinux can do, however, is run a
normal Linux kernel as a low-priority task. The RTLinux kernel responds
to interrupts, passing them through to the real-time code when appropriate;
Linux only gets a chance to run when the real-time code has finished. In
an RTLinux system, a small amount of real-time code can perform data
acquisition or other real-time tasks while leaving much of the more
time-flexible processing to Linux-based code.
One interesting thing to know about RTLinux is that the basic technique is
This patent - first covered in LWN
in February, 2000 - was a relatively early indication of just how software
patent claims can affect free software users. The core RTLinux code was
licensed under the GPL, but it was not truly free; anybody wanting to use
it was subject to the terms imposed by the patent owner. Those terms were
eventually spelled out in the RTLinux
patent license which allowed royalty-free use provided that either
(1) the "Open RTLinux" distribution was used without modifications, or
(2) the entire application was licensed under the GPL. Not everybody
was happy with this license, but most of the world found ways of living
with it or avoiding the patent, and things got quiet on the RTLinux front
for some years.
On February 20, however, Wind River Systems announced
the acquisition of RTLinux - including the patent. Interestingly, nothing
to be found in Wind River's press release or acquisition FAQ mentions the
patent license in any way. The text of that license, meanwhile, has
disappeared from the FSMLabs site and has yet to reappear on the Wind River
site. LinuxWorld ran an article
on the acquisition with a verbal statement from Wind River that the
license would be maintained, which is a step in the right direction, but it
hardly adds up to a commitment on Wind River's part.
It is entirely possible that Wind River will continue with the current
policy. Perhaps Wind River will even make new "Open RTLinux" releases
allowing licensees to run reasonably contemporary software. At the moment,
however, this code does not appear to be downloadable from anywhere, and
there is no indication of when that situation might change. Along these
lines, it's worth looking at some text from the
acquisition FAQ [PDF]:
There are other real-time Linux products available in open source
today. However, RTLinux is the only commercially supported, hard
real-time product available today. Other open source versions of
RTLinux are based on much older versions of the technology or on
Given that Wind River sees an advantage to having a newer RTLinux than the
"open source" versions, updated free releases of RTLinux from Wind River
For anybody who is concerned, there are alternative approaches to real time
and Linux which are worthy of consideration. At the lowest level, there is
Adeos, a "nanokernel" which makes
RTLinux-like functionality available while avoiding the claims of the
RTLinux patent. Rather than run the general-purpose kernel as a task of
the real-time kernel, Adeos runs both as tasks underneath itself. Adeos,
in turn, is used at the base of RTAI, a
longstanding RTLinux competitor. Things have been relatively quiet on the
RTAI front in recent times, but a look at the RTAI-Lab
project suggests that interesting things are happening there still.
Beyond that, work on the real-time preemption project, which aims to make
Linux, itself, a real-time capable kernel, continues, and much of that work
has found its way into the mainline. It will always be harder to prove
that a full Linux kernel can provide deterministic response times, but, for
many applications, the real-time performance of this kernel will be more
than good enough. Some real-time vendors are already shipping products
based on this work.
There may well be an ongoing market for the RTLinux technology that Wind
River has just bought. It would be nice if Wind River could find a way to
exploit that market while, simultaneously, using RTLinux to increase its
contributions back to the community. There are few indications that Wind
River sees RTLinux as anything more than a product, though, so those hoping
for a more community-oriented stance may well be disappointed. The good
news is that the alternatives are plentiful and quickly getting better.
Comments (4 posted)
There have been a few events of interest in the Fedora community recently;
this article will attempt to provide a quick overview thereof. For the
this page, "events of interest" do not include personalities who have
decided to switch loudly to a different distribution.
The Fedora project has been trying to open itself up to contributions from
the community, with slow (but real) success. The community is not just
made up of developers and packagers, however; it turns out there is a group
of motivated people who would like to help out with the Fedora artwork.
Good design can be as hard as good code, and one would think that this sort
of contribution would be welcome. And, to an extent, it is - to an extent.
There has been a conversation happening on the fedora-art list recently;
some of the themes can be seen in this
posting. It seems, frankly, that the Red Hat-based Fedora folks are
concerned about the quality of artwork contributions and (though they don't
say so in so many words) loss of control over the default look of the
distribution. The end result is that the Fedora board has decided that contributed artwork will not be
part of the default Fedora theme; instead, that work will be done within
Red Hat. The project is trying not to close the door completely:
But the default theme is not all there is to the Artwork project.
There are many things left to do, including the Echo icon set.
Redesign and new art is needed for the Wiki, infrastructure
applications, the "Some Day Soon" Plone site, and so forth. In
addition, Fedora is not limited to just the default release art.
As part of the initiative to give users the ability to spin their
own distributions built on Fedora, we'd like contributor art to be
able to function as a drop-in RPM package replacement for the
default release art.
Nonetheless, there is a fair amount of disappointment in the artwork
community at the moment.
On a related issue, the recent revelation that Dell's customers are asking
for preinstalled Linux systems has created some interested in the Fedora
community. Having a vendor as large as Dell preinstall Fedora would have
clear benefits in helping the project to expand its user base. The Fedora
folks would like to help make that happen, but it seems that there are some potential roadblocks on the
Unless we create the second logo set, I don't think we'll get very
far with pre-installation. Most vendors will want to sweeten the
user experience, and possibly add branding. Any of that will make
it no longer Fedora, and the vendor would be unable to make such
claims under the trademark policy. They'd have to remove all the
Fedora/RH trademarked logos and such too.
Some members of the advisory-board list have pointed out that worrying
about the trademark policy is getting ahead of the game; making the
distribution work seamlessly on, say, Dell laptops should maybe come
first. Still, this issue points out the hazards of mixing trademark
licensing and free software. Sometimes the results are not even in the
trademark holder's interest.
Dell laptops were mentioned because the project knows that a surprisingly
large number of its users are installing Fedora on those systems. How does
Fedora know this? The answer is a tool called "smolt," which gathers
information on the underlying hardware and phones home with it. The
project is quite careful about how this communication is done - no
connection is made until the user explicitly agrees to it happening. Even
so, there have been some complaints on the lists, along with suggestions that it
could be illegal under the privacy laws of some countries, especially in
The project is currently working on a
fairly tight; the project really is just interested in the sort of hardware
its distribution is running on, not the people who are running it.
Nonetheless, if anybody has concerns about the use of this information
(which might be expanded to include a list of packages installed on the
system), now would be the time to express them.
During a recent Fedora board meeting, there was discussion of the Fedora 7 release delay,
and, in particular, whether support for Fedora Core 5 and 6 would
be extended to compensate. It came out that, while a number of people
assume that the new 13-month support policy came into effect when it was
adopted, that is not how the project understands it. The Fedora Core
releases are currently expected to be supported under the old way of doing
things: support for Fedora Core 5 will end when the second
Fedora 7 test release (which just went into freeze mode) comes out. Support for
Fedora Core 6 will end during the Fedora 8 development cycle.
The full 13-month (or "2n+1") support mode is only expected to begin with
Fedora 7. There has been some talk of trying to extend security
support for FC5 and FC6, but it is not at all clear that it will happen.
Finally, it has been noted that a number of Fedora tasks seem to be going
more slowly than many people would like. The word that your editor has
heard is that much of this has to do with the impending release of
RHEL 5. Getting that release into final form has been causing some
heavy demands on Red Hat's developers, with the result that less time is
available for working on Fedora. Once the RHEL release is out, things can
be expected to pick up a bit on the Fedora side.
Comments (4 posted)
Time recently published an article entitled Getting
rich off those who work for free
which, among other things, talked
about free software this way:
Open-source, volunteer-created computer software like the Linux
operating system and the Firefox Web browser have also established
themselves as significant and lasting economic realities.
It is not uncommon to see Linux referred to as a volunteer-created system,
as opposed to the corporate-sponsored, proprietary alternatives. There has
been little research, however, into how much work on Linux is truly
"volunteer" - done on a hacker's spare, unpaid time. In general, the
assumption that Linux is created by volunteers is simply accepted.
Determining the real provenance of free software can be a daunting task.
There is a wealth of information available for those who look, however. In
an attempt to shine some light in this area, your editor hacked up some
scripts to do a lot of digging around in the kernel git repository. The
idea was that, by looking at who is putting changes into the kernel, we can
get a sense for where our source is coming from.
Who got patches into 2.6.20
This study looked at the stream of patches that changed the 2.6.19 kernel
into the current 2.6.20 release. There were, as it turns out 4983
non-merge changesets in this release, contributed by 741 different
developers. (Merge changesets mark where the contents of other
repositories were pulled into the mainline, but they do not carry any code
changes, so the analysis skipped them).
These patches added 286,439 lines of code and removed 159,812
others, for a total growth of 126,627 lines over the 2.6.20 development
Your editor's scripts looked over every non-merge commit in 2.6.20. For
each, the developer listed as the "author" was given credit for the patch.
This approach is not entirely fair, since one developer will, in some
cases, be submitting code written by a group of people. In general,
though, there is no easy way of getting around this problem - the true
breakdown of authorship of a joint work simply is not available in the
mainline repository. Your editor believes that this inaccuracy affects the
accounting of a relatively small portion of the patches merged into the
Beyond that, how one generates statistics from a patch stream is an
interesting question. How does one measure the productivity of
programmers? One possibility is to look at the number of changesets
merged. By that metric, this is the list of the most prolific contributors
|Developers with the most changesets|
|David S. Miller||48||1.0%|
Looking at patch counts rewards developers who put in large numbers of
small patches. Al Viro's patches include a vast number of code annotations
(to enable better checking with sparse), include file fixups,
etc. Many of the changes are small - many do not affect the resulting
kernel executable at all - but there are a lot of them. Even so, as the
biggest contributor, Al generated less than 5% of the
total changesets added to the kernel. The top 20 contributors, all
together, generated 28% of the total changesets in 2.6.20.
One could make the argument that a better way to look at the problem is by
the number of lines affected by a patch. In this way, a contributor's
portion of the whole will not depend on whether it has been split into a
long series of small patches or not. On the other hand, simply renaming a
file can make it look like a developer has touched a large amount of code.
Be that as it may, by looking at lines changed (defined
as the greater of the number of lines added or removed by each individual
changeset), one gets a table like this:
|Developers with the most changed lines|
|Amit S. Kale||9537||2.7%|
|Mauro Carvalho Chehab||7390||2.1%|
Jeff Garzik comes out on top of this particular measurement by virtue of
having deleted the long-unmaintained floppy tape subsystem. Patrick
McHardy's work includes a number of additions to the netfilter subsystem,
Jiri Slaby did a great deal of driver cleanup work, Avi Kivity was
the contributor of the KVM
virtualization code, and Andrew Victor contributed a number of
ARM-related patches and the Atmel AT91 i2c driver. (The contributions made
by other authors can be found by searching out their name in the 2.6.20 short-form changelog).
Most of the developers in the above list got there by adding code to the
kernel. It can be said, however, that the true heroes in the development
community are those who remove code and make the kernel smaller. The
developers who were best at removing more code than they added were:
|Developers with the most lines removed|
Once again, Jeff Garzik's removal of ftape comes out on top, by far. Chris
Zankel cleaned up the Xtensa architecture, removing a number of files in
the process. Adrian Bunk worked on the ftape removal, got rid of the frame
diverter code, removed an old, broken block driver, and generally performed
cleanups all over the tree. Mr. Bunk is, in fact, the bane of old code;
over the last year (since 2.6.16) he has removed a full 127,000 lines from
the kernel source tree. Arnd Bergman got rid of a bunch of
syscall*() macros. Linus Torvalds removed the broken x86 stack
Finally, one could look at a different measure entirely: the number of
patches signed off by each developer. A Signed-off-by: line is an
indication that the person involved believes that the code is suitable for
merging into the kernel; it implies that some degree of attention has been
paid to the patch. Authors sign off their code, as do the subsystem
maintainers who pass it up the chain. The top signers-off in 2.6.20 were:
|Developers with the most signoffs|
|David S. Miller||483||4.7%|
|Mauro Carvalho Chehab||170||1.6%|
|Arnaldo Carvalho de Melo||119||1.1%|
There were a total of 10,354 signoff lines in the 2.6.20 patch stream, so
each changeset, on average, was signed off just over two times. It is interesting that
Linus, who ultimately merges every patch, only signed off 13% of them. It
seems that most patches, these days, go directly into the mainline from
subsystem repositories without a signoff from Linus or Andrew. Most of the
other names on that list, with just a few exceptions, are the maintainers
of subsystem or architecture trees.
Who paid them
So now we have a sense for who got their fingers on the code which went
into 2.6.20. But one interesting question still has not been answered: to
what extent was that code contributed by volunteers (or "hobbyists")?
Finding an answer to that question is somewhat trickier than looking at who
wrote the patches, mostly because very few developers say "I wrote this on
behalf of my employer."
The approach taken by your editor was relatively simplistic, but, perhaps,
the best that is practical. Any patch whose author's given email address
indicates a corporate affiliation is assumed to have been developed by an
employee of that corporation. So any patch posted by somebody with an
ibm.com email address is accounted as having been done by an IBM
employee. Things are complicated by the fact that many people who work for
companies do not use corporate
addresses; it is not unheard-of for companies to have policies explicitly
prohibiting code contributions associated with their domains. Your editor
has coped with this problem by filling in the relevant developer's
affiliation whenever it is known to him; in some cases, the developer was
asked for this information.
This method has the effect of crediting all of an employee's work to
his or her employer. In many cases, the situation is probably more
complicated than that; one assumes, for example, that a certain kernel
hacker's employer has not directed him to hack on
Battle for Wesnoth. When looking only at kernel code, however,
crediting all work to the employer is probably relatively safe.
Using this approach, the top sources of changesets were:
|Top changeset contributors by employer|
|University of Aberdeen||79||1.6%|
Looking instead at the number of lines of code changed, the results become:
|Top lines changed by employer|
|University of Aberdeen||4324||1.2%|
[Note that these tables have been updated once since the article was
originally published; the curious can see what the original versions looked like.]
In these tables, the line marked "(Unknown)" is exactly that: patches for
which existence of a supporting employer could not be determined. The line
marked "(None)", instead, indicates the patches from developers
known to be working on their own time.
Either way, the results come out about the same: at least 65% of the code
which went into 2.6.20 was created by people working for companies. If the
entire "unknown" group turns out to be developers working on a volunteer
basis - an unlikely result - then just over 1/3 of the 2.6.20 patch stream
was written by volunteers. The real number will be lower, but it still
shows that a significant portion of the code we run is written by
developers who are donating their time.
One year's worth of changes
Looking at a single kernel release is instructive, but it can also be
deceptive. The relatively short release cycle used by the kernel project
makes it fairly easy for prolific developers to see few of their patches go
into a specific release. In an attempt to gain a longer-term perspective,
your editor forced his suffering system to crank through the entire history
from 2.6.16 (released almost exactly one year ago) to the present. Some
28,000 non-merge changesets have been added to the mainline (by 1,961
developers) over that time,
replacing 1.26 million lines of old code with 2.01 million lines
of new code - the kernel grew by 754,000 lines.
The developers who touched the most lines over that time were:
|Developers with the most changed lines|
|Mauro Carvalho Chehab||68568||2.7%|
|David S. Miller||35958||1.4%|
|Divy Le Ray||17909||0.7%|
The results for employers were:
|Top lines changed by employer|
|Open Grid Computing||20505||0.8%|
The end result of all this is that a number of the widely-expressed
opinions about kernel development turn out to be true. There really are
thousands of developers - at least, almost 2,000 who put in at least one
patch over the course of the last year. Linus Torvalds is directly
responsible for a very small portion of the code which makes it into the
kernel. Contemporary kernel development is spread out among a broad group
of people, most of whom are paid for the work they do. Overall, the
picture is of a broad-based and well-supported development community.
There are many other interesting things to be learned by looking at the
kernel's development history. Expect more articles along these lines as
your editor finds the time to improve his scripts.
Comments (61 posted)
Page editor: Jonathan Corbet
Inside this week's LWN.net Weekly Edition
- Security: A PostgreSQL flaw; New vulnerabilities in clamav, fail2ban, kernel, php, ...
- Kernel: More changes for 2.6.21; The managed resource API; A new Intel wireless driver; Clockevents and dyntick.
- Distributions: A look at dyne:bolic 2.4; Debian 3.1 r5, Lunar 1.6.1, BLFS 6.2.0, openSUSE 10.3 Alpha, Ubuntu Herd 4; new security distribution ProTech
- Development: Progress on the Linux Desktop Testing Project,
new versions of wxSQLite3, Apache SpamAssassin, MAILOOK, CUPS, CUPS DDK,
Mod_python, Mammut, Grace, GARNOME, PasswordSafe, Wine, Bongo, Elisa,
CLAM, HylaFAX, Laplock, GCC, Lython, PHP, PyPy, Wing IDE.
- Press: BBC and Windows DRM, IDC predicts $40B Linux ecosystem,
Relationship Economies, the ACCURATE meeting,
LinuxWorld OpenSolutions Summit, KDE at SCALE 5x, Vancouver PHP Conf,
Dell users vote for Linux, Flash issues,
Cuba goes open-source, FOSDEM interviews, Samba's Jerry
Carter, Jonathan Miller podcast, Inge Wallin interview,
Firewall comparison, using Icecast2, suspending laptops, RPM revival,
Xfce 4.4 reviewed.
- Announcements: EFF protects humor site, PUBPAT Exec testifies before US house, ADempiere
announces ERP support, LinuxQuestions awards, Nokia N800 challenge,
LinuxBasics course, Python Bootcamp, OpenVZ at LinuxWorld, Electronic Voting
CFP, Bossa conf - Brazil, ESC 2007, KDE at OpenMind and Guademy, FOSS Meet
- India, OO.o conf selects Barcelona, Web 2.0 - Tokyo, Where 2.0 registration.