recent LWN article
Jim Ready has a long history in the embedded systems market. Most
recently, he became the founder of MontaVista, now one of the most
successful embedded Linux companies. A
with some of Jim's comments; it only seemed fair to give him the
opportunity to present his side of the story. Thus, this interview. We
asked several questions about MontaVista and its approach to Linux
marketing, and Jim took quite a bit of time to answer them in detail. So,
without further ado...
You have been working in the embedded Linux market for some years. How has
that market changed over that time? What do you think are the prospects
for embedded Linux now?
The single biggest change, and one that gives me great pleasure, is that
embedded Linux is now mainstream, part of the landscape, and arguably the
fastest growing embedded OS. Believe me, when we started in 1999 that was
hardly the case. Of course, the complexity of the devices that our
customers are building continues to increase. The underlying hardware is
typically a highly integrated system with loads of I/O, for example the
SOCs (System On a Chip) such as the TI 3430. That in turn drives both
complexity in Linux as well as in the application and middleware software
stacks. It's pretty amazing to realize that a little Linux-based
handheld device running on batteries is powerful enough to have supplied
the State of California government's computing needs not so many years
Where do you think MontaVista's sweet spot is in that market?
Companies who are highly focused on their value-add who want a first class
partner to supply them a suitable Linux and associated services (consulting
and training etc.) upon which they will develop their application. The more
formal approach a company takes towards their own software development, the
more they care about meeting schedules and the higher their requirements
for quality, the "sweeter" MontaVista looks. When you're basing a
billion dollar product line on someone's OS you care about what's
going in your product. So when Motorola, NEC or Panasonic end up shipping
30 million phones, they need a supplier who can meet their technical,
schedule and quality requirements. The phones have to ship for the
Christmas season, and no one wants to recall millions of devices.
another example, our Carrier Grade Linux distribution is the core OS in
deployed NEC systems which have established 99.9999% availability
(that's no more than ~31.5 seconds of unscheduled downtime in a year,
which is a DoCoMo requirement). Our Professional Edition is the OS for two
different patient monitoring systems that have been through FDA
certification. We're truly fortunate to have thousands of customers,
both big and not-so-big.
Embedded systems vendors have, as a group, been criticized for their lack
of participation in the free software development process. Are you happy
with MontaVista's level of contribution? What, in your mind, are some of
the highlights of MontaVista's community participation?
Most contribution surveys show MV in the top 10 of Linux contributors. (No
other embedded Linux vendor even makes it into the top 30) Arguably
MontaVista contributes more to Linux relative to our size than any other
company in the world. It has always been a cornerstone of our strategy to
be a major contributor to Linux. We figured the more gas we poured on the
Linux fire the faster we could erode the RTOS suppliers installed base and
speed to movement towards Linux. We are perhaps best known for our work in
helping making Linux real-time capable but over time we have also made
significant contributions in the PPC, XScale, MIPS and ARM trees as well as
some other specific projects such as kgdb, LTT, DPM (Dynamic Power
Your recent article
in Military Embedded Systems was seized upon by a
proprietary embedded vendor as proof that Linux is too expensive and
difficult for embedded applications. Assuming you disagree with his
conclusions, where do you think his reasoning went wrong?
Well there really wasn't any reasoning, just ranting. But having said
that Dan (Dan O'Dowd Greenhills' CEO) implied that our business model
was to allow customers to more or less get in trouble by developing their
own from scratch Linux distribution and then charge them for support to
bail them out. Of course that's not what we do. Rather we build a robust
fully tested and supported embedded Linux distribution (MontaVista Linux
Professional Edition, for example) and deliver that to our customers. We
then maintain and support that specific version of MontaVista Linux over
time, even as the community dashes forward. In fact we have maintenance
obligations that can be as long as a decade from initial deployment.
approach gets the customer out of the business of making their own
distribution, maintaining and supporting it with all the accompanying
costs. So we shield the customer from the complexity and change rate that
they otherwise would be exposed to if they were on their own. They don't
have to watch all the patches, monitor the newsgroups and otherwise be tied
up, they can get on to building their product. Dan purposely ignored the
fact that a commercial embedded Linux distribution makes it very easy to
use Linux as an embedded OS. I suspect that's why he tried to hide it.
Your article suggests that an embedded systems manufacturer using Linux
would start by assembling the kernel and development toolchain by hand.
Why do you think they would do that? Even in the absence of vendors like
MontaVista, there are numerous options which do not require assembling
systems at such a low level; why would a vendor not use one of them?
We know from direct experience that even starting with what appears to be
"pre-assembled" distribution, from a semiconductor maker or
elsewhere, a developer sometimes isn't getting what they think they
Don't get me wrong, almost any Linux distribution can serve as a
starting point, maybe 99.99% perfect, but our customers demand more than
that. They want to be at the end of the Linux development cycle, not the
beginning. For example, a Linux distribution we recently started working
with had the following problems:
- The code explicitly ignored Linux coding standards by adding hardware
dependencies. That code would never be accepted into the upstream trees,
and this kind of fork creates debugging issues and additional maintenance
- The drivers were not SMP-safe, real-time safe nor did they support DPM, yet
the device was designed for applications where all three could well be
required. In order to take advantage of these advanced features, the
device driver would need to be re-written from scratch.
- The code contained numerous defects that caused the system to crash. Error
returns were not checked and other problems indicating very poor coding
practices. These are exactly the type of quality issues that should compel
businesses to find a Linux commercialization partner.
We had the great pleasure of fixing all these problems as we assembled our
distribution. Even with our standard practice of pushing back the changes,
as you well know, there is no guarantee by the community that these changes
make it back into the appropriate open source trees.
The fact is it is difficult for a prospective Linux developer to have any
idea of the state of the Linux distribution they might select. A high
quality, commercial distribution can give a developer some peace of mind
about what they are getting. For example, MontaVista has a formal
development process in place for each of its releases, with quantitative
criteria that must be met for defects (0 critical defects for example with a
sharply declining overall new defect detection curve.) before the
distribution can ship. Our processes have been formally audited by a number
of our largest customers in order to assure themselves of what they are
getting from us. And as we mentioned above, the proven results from devices
in the field speak to our abilities. As for other starting points,
you'll have to ask them about their process.
There were some interesting numbers in that article. Where did the
5000 messages/day for kernel.org come from - which lists?
Since our engineers live and breathe Linux and the other Open Source
components that make up and embedded Linux distribution, we have a pretty
good feel for the overall rate of traffic we keep up with. Based upon that
experience we measured the overall message traffic that a developer would
have to monitor to keep abreast of the daily ins and outs of the typical
mix of software they would use for an embedded Linux project. We aggregated
the total under kernel.org, which isn't precisely correct, but the
paragraph preceding that statement clearly referred to a set of lists one
would have to monitor.
For example, the monitoring would include not only
lkml, but also the lists for other significant parts of the software
typically used for an embedded project, including the list maintained for
the specific architecture used (MIPS, PPC ARM etc), the real-time list,
networking, IPv6, security, advanced filesystems etc. By the way, the lkml
list on May 21, 2008 contained ~500 messages, and gcc contained ~100, just
for starters. So it wasn't just lkml at 5000 but a total set that can
total up to 5000 per day. Does the fact that lkml is only ~500 a day (and
"only" ~200-300 on weekends) make it any less daunting? I don't
You say: "a recent security patch that took all of 13 lines of code to
implement against an embedded Linux system would have taken more than 800k
lines of source patches to implement if the previous trail of patches had
been ignored." How was that number arrived at? Which security patch were
you describing? How could it possibly require 800,000 lines of patches "to
implement" this security fix?
This example comes from a sequence back in 2006 (CVE-2006-1528 to be
precise), but the "problem" is just as true today. Here's the
A developer decides to use Linux and has taken the strategy of minimizing
their costs by using a community-maintained Linux kernel. (This story would
be true if the developer started with the typical semiconductor
distribution, by the way) The community has a good reputation for stability
and defect resolution and therefore the developers think they can minimize
their own effort.
They start with Linux 2.6.10 and base their device
application software on this release. During testing, they notice a defect
and find that the defect has been identified (the good news) and fixed in
2.6.13 (the bad news). So now they have a problem, moving up to 2.6.13,
where the defect is fixed also introduces 846,233 new lines of code (the
delta between 2.6.10 and 2.6.13).
This magnitude of change restarts their
QA process, since so much code has changed in the underlying Linux
kernel. Their other choice is to backport the fix, which in this particular
case is 33 lines (we know because we did it), but now the developer has
taken on maintenance of their own Linux, which was what they were trying to
avoid in the first place. This drift between a Linux release you have
baselined and the fact that defects are often fixed in newer releases
presents a less than perfect set of choices for developers. Whether you
wanted to or not, you're in the Linux maintenance business. This drift
problem is true of many distributions, not just dealing with kernel.org.
If our customer found the same defect, we have the obligation to fix it in
the release that they purchased from us; we don't force them to
potentially destabilize their environment by sending them a newer kernel
release where the defect was originally fixed. I guess it all depends how
cavalier one is about changing your underlying operating system after
you've developed and tested your application. In general our customers
are very strict about minimizing changes, and so are we.
At least some of MontaVista's marketing would appear to focus on making
Linux look scary. Are you not concerned that this approach might have the
effect of making Linux in general look less attractive and, thus, playing
into the hands of proprietary systems vendors?
No one is a bigger proponent of embedded Linux than MontaVista (and we have
the contributions to prove it). But it doesn't do us any good to have
folks try Linux, get over their heads and fail, and attribute that failure
We have seen over the past 8 years any number of projects that got into
trouble by not understanding what to expect when they downloaded some Linux
and started in by themselves. In fact one of our very earliest customers,
back in 1999, had started off building their own Linux, and hit a hardware
integration bug that stopped them dead in their tracks for weeks, putting
their project in real trouble. Had we not been able to help them out, their
alternative was Windows CE. Ugh!
Why shouldn't many millions of lines of complex operating system code
that changes daily be a little scary, especially when your business is
making devices, not operating systems? I think it is a mistake to
"trivialize" the difficulty in owning large amounts of any software,
including Linux. That's why I think it's important for folks to be
well informed about what they are getting into, so they can make good
decisions on how they will approach using Linux for their system, whether
they do-it-themselves or go commercial. In either case we want them to
Is there anything else you would like to pass on to LWN's readers?
We can all be quite proud of the enormous progress Linux has made in
transforming the embedded OS marketplace from one which was highly
fragmented and largely devoid of standards, to an environment based upon a
highly functional OS which is a truly open standard: Linux. A whole cast of
characters made this possible: visionary customers who dared think it was
possible to embed Linux in their devices, the semiconductor companies
making sure Linux was ported to their chips, commercial companies such as
MontaVista and others making rock solid distributions that were capable of
being deployed by the millions, and numerous individuals who made
significant contributions along the way. It's a pretty powerful
combination that's hard to beat.
We would like to thank Jim for taking the time to answer our questions.
to post comments)