ELC: The embedded Linux nightmare

Linux (and the kernel in particular) is, says Thomas, a sort of "mutual benefit society" which is jointly maintaining a common good. This society will only work as long as the stakeholders give to it as well as taking from it. The giving part, unfortunately, is often lacking in the embedded world.

There are a lot of reasons given for the use of special, closed, vendor kernels in embedded situations. According to Thomas, these reasons do not hold water. They include:

• "Vendor kernels are developed by experts." Thomas looked at some specific vendor kernels to see what level of expertise was to be found there. In one kernel from a system-on-chip vendor, this allegedly 2.6.10 kernel had patches to about 10,000 different files - out of just over 16,000 total. Another kernel, from a distribution vendor, had modified 8,000 files. Yet another, from a board vendor, had only patched 6500 files. Says Thomas: "don't ask me why" these vendors felt the need to make so many changes.

  To give some perspective, the patch from 2.6.10 to 2.6.11 only touched 5600 files. These vendor kernels are far larger than the (invasive) real-time preemption patch set, which only hits 725 files. These massive patches are not a sign of expertise - quite the opposite, instead. Experts don't mess with things which do not need changing and they get their changes back into the mainline.

• "Vendor kernels offer better time to market." Thomas's counterexample here was an email from a vendor which had been struggling with a (self-inflicted) driver problem for a month. Working with the community, instead, allows vendors to avoid making silly mistakes and to fix them when they do happen.

• "Users prefer vendor kernels." This is only true when there is no choice. When there is a choice, users prefer kernels with ongoing development and maintenance, and for which they can get support from the community.

• "Vendor kernels help Linux." That help is hard to see. Thomas pointed out this discouraging note from the folks at Cirrus:
  I think we will just maintain our own port for the 93xx. I am not going to want to support code not written by Cirrus Logic. So I give you kuddos for getting to the port first, but using GIT makes it easy to remove your work and add ours.

  It is hard to see how this sort of attitude helps Linux in any way. Instead, we have vendors tossing aside the work done by the community in the name of "not invented here."

What really flows from vendor kernels is user lock-in, community detachment, and waste of resources. None of these are good for users, for the vendor, or for the Linux community as a whole. They are, instead, the embedded Linux nightmare.

As an example of community detachment, Thomas offered the linux-arm.org web site, which describes itself as:

This site, Thomas points out, was launched in 2005 - ten years after the community ARM port was launched. It does not even do the courtesy of linking to the real community ARM site. It is, instead, an example of a vendor trying to create its own community which has little to do with the people actually creating the code.

With regard to waste of resources: a Linux developer recently rewrote a system-on-chip driver to make it suitable for the mainline. In the process, a 7,000-line driver became a much better 1,300-line driver. Using the COCOMO model, Thomas estimates that about $180,000 was wasted in the creation of this vendor driver.

An even more egregious example is a fork of the real-time preemption tree by "an unnamed company" a couple of years ago. No patches have ever been published from this fork, and there has not been a single email exchange with the preempt-rt developers. The resulting code is still based on a kernel from about the 2.6.14 era, and is completely unmaintainable. Unfortunately, a customer now wants serial ATA support, putting this company in a difficult situation. Thomas asks: "why the hell is this company using Linux?" He estimates that at least ten staff-years have been wasted in this fork.

The end result of this nightmare can be seen in the form of unhappy customers, a bad reputation for free software, fragmentation of the code base, a feeling of being ripped off among kernel developers, and wasted resources. In addition, Thomas fears that the kernel development process risks being dominated by the enterprise Linux companies, which do work with the community. If the embedded world wants to avoid all of these problems, it needs to start talking with the community and getting its code into the mainline kernel. Then Tux can get a good night's rest, and world domination will get back on schedule.