On successive days, Harald Welte and David Woodhouse gave different views
of the relationship between embedded companies and the free software
communities whose code the companies are increasingly using. Their
outlooks were not contradictory, but instead complementary; each came
at the topic from a different direction. Welte looked mostly at what
companies, particularly chip vendors could do better, while Woodhouse
looked at what things the community could do to improve.
Welte and Woodhouse spoke at the
autumn Mobility conference and Embedded Linux
Conference Europe in Ede, the Netherlands,
November 6 and 7. The Congrescentrum De
Reehorst facility was excellent, well-suited to an event of this type
which is not surprising as NLUUG has been holding two events there each year
for the last ten years or so. In addition, the conference was
well-organized and run; clearly displaying the experience that comes from
the 26 years that NLUUG has been in existence.
[ The following covers Welte's presentation, Woodhouse's talk will be
covered in a subsequent article. ]
Welte kicked things off on Thursday with a talk entitled "How chipmakers
should (not) support free software". As the conference got a bit of a late
start and was already 15
minutes behind at that point, Welte said that he would make the time up
because "everyone can understand gzip compressed speech". More
seriously, he outlined his experience as a member of the Linux community,
embedded developer, chip manufacturer from his recent work with Via, as
well as a customer of consumer-grade embedded devices for gpl-violations.org; all of which result in
multiple relevant points of view.
Linux is being found in more and more devices today—some less
than obvious. Welte listed fairly well-known things like mobile
phones and in-flight entertainment systems, but then noted that there are
DSL Access Multiplexers (e.g. DSLAMs), payphones, ATMs, as well as vending and exercise machines that also
Vendors of those devices are using free and open source software (FOSS)
because of its
strengths, which he outlined. There is a great deal of innovative and
creative development done in FOSS because the barriers to entry are fairly
low: the codebase is easy to
read—at least in comparison to closed source—and there are
standard development tools that are freely available. Because development
is done in the open, developers will be embarrassed if their software
architecture or code is bad. This also results in better security because of
the code review that takes place.
The outcome of using FOSS this way is that "we should have a perfect
with tons of embedded products, all secure and maintainable, that allow for
additional or alternate functionality via third parties. The first of
those, many embedded products, has been achieved, but we are still waiting
for the other two, Welte said.
He contrasted a user's experience with Linux on PCs today with the
experience provided by most embedded devices. For PCs, you can
download the kernel, build it and it will run, with most hardware
supported. You can choose from multiple distributions, any of which will
have a kernel close
to that of a mainline kernel and provide regular security updates. These are
"things we are used to for many years", but things are not
that way in the embedded space.
In the embedded world, every CPU or system-on-a-chip (SoC) has its
own kernel tree, typically based on some ancient version of the kernel,
that never gets cleaned up or submitted for mainline inclusion. So, they
get no benefit from new features or security fixes in the kernel. There
are no distributions to choose from, either for users or board
makers and, even if updates are generated, there is generally no packaging
system to use to
update the code; re-flashing the entire device is required.
In Welte's words, "this sucks!" The embedded vendors get
unstable and unmaintainable software with "security
nightmares" and no
innovation from elsewhere. The vendors have kernels that have diverged so
far from the mainline that new features or fixes can't be backported, nor
can their kernels get merged upstream. This is because the vendors tend to
be very short-sighted, only focusing on getting one particular device out
From Welte's perspective, embedded vendors do not understand the real
potential of FOSS. They do not
think in terms of creating platforms that others can build atop. In
general, "they would rather sell a new [device] rather than improve
the existing one". So, the vendors compete on the basis of the
features their proprietary
competitors implement rather than figuring out how to take advantage of the
true strengths of FOSS. If, instead, they used FOSS to its fullest, they could
outcompete the proprietary
vendors in ways that could not be matched—except by using FOSS.
Turning to the chip vendors, Welte points out that there are two types of
customers: Linux-aware and Linux-unaware. The Linux-aware
customers—whose numbers are growing—will
seek out vendors whose Linux support is better. It is already relatively
late in the game: "if you don't have proper FOSS support, you will
lose the 'openness competition'".
Chip manufacturers should be engaging in "sustainable
development" by releasing kernels developed against the mainline in
cooperation with the community. One large mistake these vendors make is to
think their customers are only the tier-one companies that buy chips
directly. There are many more downstream users of a chip once it has been
integrated into other hardware; the buyers of those devices are also
important as they will determine the success or failure of the product.
Unsurprisingly, Welte recommends that the development be done in the open,
with a public development tree. Releases should not just be stable
snapshots or big code drops; "post early, post often" should
be the governing principle. FOSS is not just a technology, as chip vendors
tend to think, it is a research and development philosophy that needs to be
integrated into both the internal and external processes of the chip vendor.
On the external side, making documentation available, without a
non-disclosure agreement (NDA)—or at worst a FOSS-friendly
NDA—is essential. Internally, there is normally quite a bit of
learning required to understand the FOSS philosophy. This will require
training for engineers as well as product management folks.
Having a clear FOSS support strategy, with clear goals, is important
for making it work.
Product management needs to understand that supporting Linux is mostly a
process of understanding the development model. The Linux APIs are not a
particularly big hurdle, but understanding the community and how to work
within it can be. Supporting Linux should mean supporting the mainline,
not just N distributions, as N will grow over time, which leads to more
problems. It is important to recognize that
Linux-aware customers care as much about the quality of the code as they do
about price and performance.
Engineering management needs to encourage engineers to communicate with the
community, which requires real internet access. When faced with adding
functionality to some FOSS code, they should be looking at ways to
cooperate with others who have similar needs, rather than reinventing the
wheel. Engineers need to figure out how and where
to ask the right kinds of questions. They also need to learn that code is
written to be read, not just executed; "this is something new to many
The community also has responsibilities to help the chip makers by
providing "non-partisan" documentation because these manufacturers often
clue where to start or who to talk to" when they start considering
supporting Linux. Commercial embedded distributors have a different
perspective from the community so documentation from the community
viewpoint is required. Welte says that various Linux Foundation sponsored
efforts are helping in this area, but more needs to be done.
A mentoring program of some sort might
help by having FOSS developers willing to work with engineers to walk them
through the process of getting their code upstream.
The community must also work to keep from scaring chip vendor
engineers away by being overly rude or terse; it is important that
valid criticism be fully explained.
Welte sees a number of current or looming problems for chip vendors in
Linux, mostly involving patents or technology licensing issues. Various
licensing regimes (like those for MPEG or Sony's memory stick) impose
requirements that essentially preclude the development of free software
drivers to talk to devices that implement those technologies. Everyone in
the industry has these problems, though, so Welte suggests that they band
together to present a case to the license holders; with enough smaller
players working together, their voice can be heard.
On the whole, Welte is somewhat pessimistic about where embedded devices
are headed. He certainly sees more FOSS being used in devices in the
future, but expects to see them still be restricted so that they cannot
leverage the full potential of FOSS. He does see "some very dim
light at the end of a very far tunnel" with projects like Openmoko,
but also efforts by some chip vendors, notably Intel, to fully support Linux.
It was not that many years ago when the desktop Linux situation looked as
bleak as the embedded space does today, so there is hope. Presentations
like Welte's can only help to bring that about. The audience contained
many embedded developers, hopefully they can help their company's
management see the benefits that Welte outlines so that his perfect world
comes about sooner, but if the desktop is any guide, it will come about
to post comments)