The opening keynote talk at the 2007 Embedded Linux
was given by Thomas Gleixner. Thomas has been a significant
contributor to the kernel for some time; most recently, he is the force
behind much of the high-resolution timer work which has been merged for
2.6.21. His experience with the embedded Linux industry has prompted him
to put together a talk on how that industry works (or doesn't) with the
development community. When things go badly, he says, the result is a true
Linux (and the kernel in particular) is, says Thomas, a sort of "mutual
benefit society" which is jointly maintaining a common good. This society
will only work as long as the stakeholders give to it as well as taking
from it. The giving part, unfortunately, is often lacking in the embedded
There are a lot of reasons given for the use of special, closed, vendor
kernels in embedded situations. According to Thomas, these reasons do not
hold water. They include:
- "Vendor kernels are developed by experts." Thomas looked at some
specific vendor kernels to see what level of expertise was to be found
there. In one kernel from a system-on-chip vendor, this allegedly
2.6.10 kernel had patches to about 10,000 different files - out of
just over 16,000 total. Another kernel, from a distribution vendor,
had modified 8,000 files. Yet another, from a board vendor, had only
patched 6500 files. Says Thomas: "don't ask me why" these vendors
felt the need to make so many changes.
To give some perspective, the patch from 2.6.10 to 2.6.11 only touched
5600 files. These vendor kernels are far larger than the (invasive)
real-time preemption patch set, which only hits 725 files. These
massive patches are not a sign of expertise - quite the opposite,
instead. Experts don't mess with things which do not need changing
and they get their changes back into the mainline.
- "Vendor kernels offer better time to market." Thomas's counterexample
here was an email from a vendor which had been struggling with a
(self-inflicted) driver problem for a month. Working with the
community, instead, allows vendors to avoid making silly mistakes and
to fix them when they do happen.
- "Users prefer vendor kernels." This is only true when there is no
choice. When there is a choice, users prefer kernels with
ongoing development and maintenance, and for which they can get
support from the community.
- "Vendor kernels help Linux." That help is hard to see. Thomas
pointed out this
discouraging note from the folks at Cirrus:
I think we will just maintain our own port for the 93xx. I
am not going to want to support code not written by Cirrus
Logic. So I give you kuddos for getting to the port first,
but using GIT makes it easy to remove your work and add ours.
It is hard to see how this sort of attitude helps Linux in any way.
Instead, we have vendors tossing aside the work done by the community
in the name of "not invented here."
What really flows from vendor kernels is user lock-in, community
detachment, and waste of resources. None of these are good for users, for
the vendor, or for the Linux community as a whole. They are, instead, the
embedded Linux nightmare.
As an example of community detachment, Thomas offered the linux-arm.org web site,
which describes itself as:
This site is the definitive resource for the community of
developers and users of the Linux Kernel on the ARM Family of
This site, Thomas points out, was launched in 2005 - ten years after the
community ARM port was launched. It does not even do the courtesy of
linking to the real community ARM
site. It is, instead, an example of a vendor trying to create its own
community which has little to do with the people actually creating the
With regard to waste of resources: a Linux developer recently rewrote a
system-on-chip driver to make it suitable for the mainline. In the
process, a 7,000-line driver became a much better 1,300-line driver. Using
the COCOMO model, Thomas estimates that about $180,000 was wasted in the
creation of this vendor driver.
An even more egregious example is a fork of the real-time preemption tree
by "an unnamed company" a couple of years ago. No patches have ever been
published from this fork, and there has not been a single email exchange
with the preempt-rt developers. The resulting code is still based on a
kernel from about the 2.6.14 era, and is completely unmaintainable.
Unfortunately, a customer now wants serial ATA support, putting this
company in a difficult situation. Thomas asks: "why the hell is this
company using Linux?" He estimates that at least ten staff-years have been
wasted in this fork.
The end result of this nightmare can be seen in the form of unhappy
customers, a bad reputation for free software, fragmentation of the code
base, a feeling of being ripped off among kernel developers, and wasted
resources. In addition, Thomas fears that the kernel development process
risks being dominated by the enterprise Linux companies, which do work with
the community. If the embedded world wants to avoid all of these problems,
it needs to start talking with the community and getting its code into the
mainline kernel. Then Tux can get a good night's rest, and world
domination will get back on schedule.
to post comments)