An interview with Jim Ready [LWN.net]

By Jonathan Corbet
June 11, 2008

Jim Ready has a long history in the embedded systems market. Most recently, he became the founder of MontaVista, now one of the most successful embedded Linux companies. A recent LWN article took issue with some of Jim's comments; it only seemed fair to give him the opportunity to present his side of the story. Thus, this interview. We asked several questions about MontaVista and its approach to Linux marketing, and Jim took quite a bit of time to answer them in detail. So, without further ado...

You have been working in the embedded Linux market for some years. How has that market changed over that time? What do you think are the prospects for embedded Linux now?

The single biggest change, and one that gives me great pleasure, is that embedded Linux is now mainstream, part of the landscape, and arguably the fastest growing embedded OS. Believe me, when we started in 1999 that was hardly the case. Of course, the complexity of the devices that our customers are building continues to increase. The underlying hardware is typically a highly integrated system with loads of I/O, for example the SOCs (System On a Chip) such as the TI 3430. That in turn drives both complexity in Linux as well as in the application and middleware software stacks. It's pretty amazing to realize that a little Linux-based handheld device running on batteries is powerful enough to have supplied the State of California government's computing needs not so many years ago.

Where do you think MontaVista's sweet spot is in that market?

Companies who are highly focused on their value-add who want a first class partner to supply them a suitable Linux and associated services (consulting and training etc.) upon which they will develop their application. The more formal approach a company takes towards their own software development, the more they care about meeting schedules and the higher their requirements for quality, the "sweeter" MontaVista looks. When you're basing a billion dollar product line on someone's OS you care about what's going in your product. So when Motorola, NEC or Panasonic end up shipping 30 million phones, they need a supplier who can meet their technical, schedule and quality requirements. The phones have to ship for the Christmas season, and no one wants to recall millions of devices.

As another example, our Carrier Grade Linux distribution is the core OS in deployed NEC systems which have established 99.9999% availability (that's no more than ~31.5 seconds of unscheduled downtime in a year, which is a DoCoMo requirement). Our Professional Edition is the OS for two different patient monitoring systems that have been through FDA certification. We're truly fortunate to have thousands of customers, both big and not-so-big.

Embedded systems vendors have, as a group, been criticized for their lack of participation in the free software development process. Are you happy with MontaVista's level of contribution? What, in your mind, are some of the highlights of MontaVista's community participation?

Most contribution surveys show MV in the top 10 of Linux contributors. (No other embedded Linux vendor even makes it into the top 30) Arguably MontaVista contributes more to Linux relative to our size than any other company in the world. It has always been a cornerstone of our strategy to be a major contributor to Linux. We figured the more gas we poured on the Linux fire the faster we could erode the RTOS suppliers installed base and speed to movement towards Linux. We are perhaps best known for our work in helping making Linux real-time capable but over time we have also made significant contributions in the PPC, XScale, MIPS and ARM trees as well as some other specific projects such as kgdb, LTT, DPM (Dynamic Power Management) etc.

Your recent article in Military Embedded Systems was seized upon by a proprietary embedded vendor as proof that Linux is too expensive and difficult for embedded applications. Assuming you disagree with his conclusions, where do you think his reasoning went wrong?

Well there really wasn't any reasoning, just ranting. But having said that Dan (Dan O'Dowd Greenhills' CEO) implied that our business model was to allow customers to more or less get in trouble by developing their own from scratch Linux distribution and then charge them for support to bail them out. Of course that's not what we do. Rather we build a robust fully tested and supported embedded Linux distribution (MontaVista Linux Professional Edition, for example) and deliver that to our customers. We then maintain and support that specific version of MontaVista Linux over time, even as the community dashes forward. In fact we have maintenance obligations that can be as long as a decade from initial deployment.

That approach gets the customer out of the business of making their own distribution, maintaining and supporting it with all the accompanying costs. So we shield the customer from the complexity and change rate that they otherwise would be exposed to if they were on their own. They don't have to watch all the patches, monitor the newsgroups and otherwise be tied up, they can get on to building their product. Dan purposely ignored the fact that a commercial embedded Linux distribution makes it very easy to use Linux as an embedded OS. I suspect that's why he tried to hide it.

Your article suggests that an embedded systems manufacturer using Linux would start by assembling the kernel and development toolchain by hand. Why do you think they would do that? Even in the absence of vendors like MontaVista, there are numerous options which do not require assembling systems at such a low level; why would a vendor not use one of them?

We know from direct experience that even starting with what appears to be "pre-assembled" distribution, from a semiconductor maker or elsewhere, a developer sometimes isn't getting what they think they are.

Don't get me wrong, almost any Linux distribution can serve as a starting point, maybe 99.99% perfect, but our customers demand more than that. They want to be at the end of the Linux development cycle, not the beginning. For example, a Linux distribution we recently started working with had the following problems:

The code explicitly ignored Linux coding standards by adding hardware dependencies. That code would never be accepted into the upstream trees, and this kind of fork creates debugging issues and additional maintenance burden.
The drivers were not SMP-safe, real-time safe nor did they support DPM, yet the device was designed for applications where all three could well be required. In order to take advantage of these advanced features, the device driver would need to be re-written from scratch.
The code contained numerous defects that caused the system to crash. Error returns were not checked and other problems indicating very poor coding practices. These are exactly the type of quality issues that should compel businesses to find a Linux commercialization partner.

We had the great pleasure of fixing all these problems as we assembled our distribution. Even with our standard practice of pushing back the changes, as you well know, there is no guarantee by the community that these changes make it back into the appropriate open source trees.

The fact is it is difficult for a prospective Linux developer to have any idea of the state of the Linux distribution they might select. A high quality, commercial distribution can give a developer some peace of mind about what they are getting. For example, MontaVista has a formal development process in place for each of its releases, with quantitative criteria that must be met for defects (0 critical defects for example with a sharply declining overall new defect detection curve.) before the distribution can ship. Our processes have been formally audited by a number of our largest customers in order to assure themselves of what they are getting from us. And as we mentioned above, the proven results from devices in the field speak to our abilities. As for other starting points, you'll have to ask them about their process.

There were some interesting numbers in that article. Where did the 5000 messages/day for kernel.org come from - which lists?

Since our engineers live and breathe Linux and the other Open Source components that make up and embedded Linux distribution, we have a pretty good feel for the overall rate of traffic we keep up with. Based upon that experience we measured the overall message traffic that a developer would have to monitor to keep abreast of the daily ins and outs of the typical mix of software they would use for an embedded Linux project. We aggregated the total under kernel.org, which isn't precisely correct, but the paragraph preceding that statement clearly referred to a set of lists one would have to monitor.

For example, the monitoring would include not only lkml, but also the lists for other significant parts of the software typically used for an embedded project, including the list maintained for the specific architecture used (MIPS, PPC ARM etc), the real-time list, networking, IPv6, security, advanced filesystems etc. By the way, the lkml list on May 21, 2008 contained ~500 messages, and gcc contained ~100, just for starters. So it wasn't just lkml at 5000 but a total set that can total up to 5000 per day. Does the fact that lkml is only ~500 a day (and "only" ~200-300 on weekends) make it any less daunting? I don't think so.

You say: "a recent security patch that took all of 13 lines of code to implement against an embedded Linux system would have taken more than 800k lines of source patches to implement if the previous trail of patches had been ignored." How was that number arrived at? Which security patch were you describing? How could it possibly require 800,000 lines of patches "to implement" this security fix?

This example comes from a sequence back in 2006 (CVE-2006-1528 to be precise), but the "problem" is just as true today. Here's the setup: A developer decides to use Linux and has taken the strategy of minimizing their costs by using a community-maintained Linux kernel. (This story would be true if the developer started with the typical semiconductor distribution, by the way) The community has a good reputation for stability and defect resolution and therefore the developers think they can minimize their own effort.

They start with Linux 2.6.10 and base their device application software on this release. During testing, they notice a defect and find that the defect has been identified (the good news) and fixed in 2.6.13 (the bad news). So now they have a problem, moving up to 2.6.13, where the defect is fixed also introduces 846,233 new lines of code (the delta between 2.6.10 and 2.6.13).

This magnitude of change restarts their QA process, since so much code has changed in the underlying Linux kernel. Their other choice is to backport the fix, which in this particular case is 33 lines (we know because we did it), but now the developer has taken on maintenance of their own Linux, which was what they were trying to avoid in the first place. This drift between a Linux release you have baselined and the fact that defects are often fixed in newer releases presents a less than perfect set of choices for developers. Whether you wanted to or not, you're in the Linux maintenance business. This drift problem is true of many distributions, not just dealing with kernel.org.

If our customer found the same defect, we have the obligation to fix it in the release that they purchased from us; we don't force them to potentially destabilize their environment by sending them a newer kernel release where the defect was originally fixed. I guess it all depends how cavalier one is about changing your underlying operating system after you've developed and tested your application. In general our customers are very strict about minimizing changes, and so are we.

At least some of MontaVista's marketing would appear to focus on making Linux look scary. Are you not concerned that this approach might have the effect of making Linux in general look less attractive and, thus, playing into the hands of proprietary systems vendors?

No one is a bigger proponent of embedded Linux than MontaVista (and we have the contributions to prove it). But it doesn't do us any good to have folks try Linux, get over their heads and fail, and attribute that failure to Linux.

We have seen over the past 8 years any number of projects that got into trouble by not understanding what to expect when they downloaded some Linux and started in by themselves. In fact one of our very earliest customers, back in 1999, had started off building their own Linux, and hit a hardware integration bug that stopped them dead in their tracks for weeks, putting their project in real trouble. Had we not been able to help them out, their alternative was Windows CE. Ugh!

Why shouldn't many millions of lines of complex operating system code that changes daily be a little scary, especially when your business is making devices, not operating systems? I think it is a mistake to "trivialize" the difficulty in owning large amounts of any software, including Linux. That's why I think it's important for folks to be well informed about what they are getting into, so they can make good decisions on how they will approach using Linux for their system, whether they do-it-themselves or go commercial. In either case we want them to succeed.

Is there anything else you would like to pass on to LWN's readers?

We can all be quite proud of the enormous progress Linux has made in transforming the embedded OS marketplace from one which was highly fragmented and largely devoid of standards, to an environment based upon a highly functional OS which is a truly open standard: Linux. A whole cast of characters made this possible: visionary customers who dared think it was possible to embed Linux in their devices, the semiconductor companies making sure Linux was ported to their chips, commercial companies such as MontaVista and others making rock solid distributions that were capable of being deployed by the millions, and numerous individuals who made significant contributions along the way. It's a pretty powerful combination that's hard to beat.

We would like to thank Jim for taking the time to answer our questions.

(Log in to post comments)