NLUUG/ELCE: Embedded devices and free software

On successive days, Harald Welte and David Woodhouse gave different views of the relationship between embedded companies and the free software communities whose code the companies are increasingly using. Their outlooks were not contradictory, but instead complementary; each came at the topic from a different direction. Welte looked mostly at what companies, particularly chip vendors could do better, while Woodhouse looked at what things the community could do to improve.

Welte and Woodhouse spoke at the co-located NLUUG autumn Mobility conference and Embedded Linux Conference Europe in Ede, the Netherlands, November 6 and 7. The Congrescentrum De Reehorst facility was excellent, well-suited to an event of this type which is not surprising as NLUUG has been holding two events there each year for the last ten years or so. In addition, the conference was well-organized and run; clearly displaying the experience that comes from the 26 years that NLUUG has been in existence.

[ The following covers Welte's presentation, Woodhouse's talk will be covered in a subsequent article. ]

Welte kicked things off on Thursday with a talk entitled "How chipmakers should (not) support free software". As the conference got a bit of a late start and was already 15 minutes behind at that point, Welte said that he would make the time up because "everyone can understand gzip compressed speech". More seriously, he outlined his experience as a member of the Linux community, embedded developer, chip manufacturer from his recent work with Via, as well as a customer of consumer-grade embedded devices for gpl-violations.org; all of which result in multiple relevant points of view.

Linux is being found in more and more devices today—some less than obvious. Welte listed fairly well-known things like mobile phones and in-flight entertainment systems, but then noted that there are DSL Access Multiplexers (e.g. DSLAMs), payphones, ATMs, as well as vending and exercise machines that also run Linux.

Vendors of those devices are using free and open source software (FOSS) because of its strengths, which he outlined. There is a great deal of innovative and creative development done in FOSS because the barriers to entry are fairly low: the codebase is easy to read—at least in comparison to closed source—and there are standard development tools that are freely available. Because development is done in the open, developers will be embarrassed if their software architecture or code is bad. This also results in better security because of the code review that takes place.

The outcome of using FOSS this way is that "we should have a perfect world" with tons of embedded products, all secure and maintainable, that allow for additional or alternate functionality via third parties. The first of those, many embedded products, has been achieved, but we are still waiting for the other two, Welte said.

He contrasted a user's experience with Linux on PCs today with the experience provided by most embedded devices. For PCs, you can download the kernel, build it and it will run, with most hardware supported. You can choose from multiple distributions, any of which will have a kernel close to that of a mainline kernel and provide regular security updates. These are "things we are used to for many years", but things are not that way in the embedded space.

In the embedded world, every CPU or system-on-a-chip (SoC) has its own kernel tree, typically based on some ancient version of the kernel, that never gets cleaned up or submitted for mainline inclusion. So, they get no benefit from new features or security fixes in the kernel. There are no distributions to choose from, either for users or board makers and, even if updates are generated, there is generally no packaging system to use to update the code; re-flashing the entire device is required.

In Welte's words, "this sucks!" The embedded vendors get unstable and unmaintainable software with "security nightmares" and no innovation from elsewhere. The vendors have kernels that have diverged so far from the mainline that new features or fixes can't be backported, nor can their kernels get merged upstream. This is because the vendors tend to be very short-sighted, only focusing on getting one particular device out the door.

From Welte's perspective, embedded vendors do not understand the real potential of FOSS. They do not think in terms of creating platforms that others can build atop. In general, "they would rather sell a new [device] rather than improve the existing one". So, the vendors compete on the basis of the features their proprietary competitors implement rather than figuring out how to take advantage of the true strengths of FOSS. If, instead, they used FOSS to its fullest, they could outcompete the proprietary vendors in ways that could not be matched—except by using FOSS.

Turning to the chip vendors, Welte points out that there are two types of customers: Linux-aware and Linux-unaware. The Linux-aware customers—whose numbers are growing—will seek out vendors whose Linux support is better. It is already relatively late in the game: "if you don't have proper FOSS support, you will lose the 'openness competition'".

Chip manufacturers should be engaging in "sustainable development" by releasing kernels developed against the mainline in cooperation with the community. One large mistake these vendors make is to think their customers are only the tier-one companies that buy chips directly. There are many more downstream users of a chip once it has been integrated into other hardware; the buyers of those devices are also important as they will determine the success or failure of the product.

Unsurprisingly, Welte recommends that the development be done in the open, with a public development tree. Releases should not just be stable snapshots or big code drops; "post early, post often" should be the governing principle. FOSS is not just a technology, as chip vendors tend to think, it is a research and development philosophy that needs to be integrated into both the internal and external processes of the chip vendor.

On the external side, making documentation available, without a non-disclosure agreement (NDA)—or at worst a FOSS-friendly NDA—is essential. Internally, there is normally quite a bit of learning required to understand the FOSS philosophy. This will require training for engineers as well as product management folks. Having a clear FOSS support strategy, with clear goals, is important for making it work.

Product management needs to understand that supporting Linux is mostly a process of understanding the development model. The Linux APIs are not a particularly big hurdle, but understanding the community and how to work within it can be. Supporting Linux should mean supporting the mainline, not just N distributions, as N will grow over time, which leads to more problems. It is important to recognize that Linux-aware customers care as much about the quality of the code as they do about price and performance.

Engineering management needs to encourage engineers to communicate with the community, which requires real internet access. When faced with adding functionality to some FOSS code, they should be looking at ways to cooperate with others who have similar needs, rather than reinventing the wheel. Engineers need to figure out how and where to ask the right kinds of questions. They also need to learn that code is written to be read, not just executed; "this is something new to many people".

The community also has responsibilities to help the chip makers by providing "non-partisan" documentation because these manufacturers often have "no clue where to start or who to talk to" when they start considering supporting Linux. Commercial embedded distributors have a different perspective from the community so documentation from the community viewpoint is required. Welte says that various Linux Foundation sponsored efforts are helping in this area, but more needs to be done. A mentoring program of some sort might help by having FOSS developers willing to work with engineers to walk them through the process of getting their code upstream. The community must also work to keep from scaring chip vendor engineers away by being overly rude or terse; it is important that valid criticism be fully explained.

Welte sees a number of current or looming problems for chip vendors in supporting Linux, mostly involving patents or technology licensing issues. Various licensing regimes (like those for MPEG or Sony's memory stick) impose requirements that essentially preclude the development of free software drivers to talk to devices that implement those technologies. Everyone in the industry has these problems, though, so Welte suggests that they band together to present a case to the license holders; with enough smaller players working together, their voice can be heard.

On the whole, Welte is somewhat pessimistic about where embedded devices are headed. He certainly sees more FOSS being used in devices in the future, but expects to see them still be restricted so that they cannot leverage the full potential of FOSS. He does see "some very dim light at the end of a very far tunnel" with projects like Openmoko, but also efforts by some chip vendors, notably Intel, to fully support Linux.

It was not that many years ago when the desktop Linux situation looked as bleak as the embedded space does today, so there is hope. Presentations like Welte's can only help to bring that about. The audience contained many embedded developers, hopefully they can help their company's management see the benefits that Welte outlines so that his perfect world comes about sooner, but if the desktop is any guide, it will come about eventually.