LWN.net Logo

How not to sell embedded Linux

By Jonathan Corbet
May 6, 2008
Every now and then one should have a look at some unabashed fear, uncertainty, and doubt (FUD) material. It's good to know what the other side is saying, the level of unintended humor is often high, and, on occasion, one even learns something. Your editor's suggestion for FUD of the week is this Embedded.com article by Dan O'Dowd. Therein, one will learn about the impending death of embedded Linux as told by the companies which sell embedded Linux.

In particular, Mr. O'Dowd looks at some marketing material from MontaVista and Wind River, and concludes:

This embedded Linux bashing from embedded Linux's strongest proponents should give pause to those who are thinking through their embedded operating system strategy. If embedded Linux champions are saying that embedded Linux is terrible, why would anyone want to risk their products or their company on it?

One can easily pick holes in this article, starting with the assertion that MontaVista and Wind River are "Linux's strongest proponents." One could also recall that we have heard this kind of thing before; in 2004, Mr. O'Dowd (who happens to be the founder and CEO of a proprietary embedded systems software vendor) helpfully warned us that "intelligence agencies and terrorists" would contribute "subversive software" to Linux and lectured on the need for secret source code to achieve true security. One could point out that many of the points put forward by Mr. O'Dowd appear to be pure fantasy. All of these rebuttals would be valid, but they risk missing an important point to be gained from this article - though it's not quite the point Mr. O'Dowd is trying to make.

Mr. O'Dowd obtains his "facts" from two sources: an advertisement by Wind River Systems (which your editor was unable to find online) and, primarily, from a column by MontaVista founder Jim Ready in Military Embedded Systems magazine. Mr. Ready's evident purpose is to frighten embedded systems vendors into buying his company's services; to that end, he lays it on pretty thick:

To keep abreast of the changes occurring on a daily basis, a developer needs to monitor the email traffic of 11 different and unsynchronized open source projects: kernel.org, the core home of the Linux kernel; the gcc and glibc projects (the core tool chain and libraries from FSF at fsf.org); and at least nine other components that would typically comprise a useable Linux development environment.

Kernel.org itself may have up to 5,000 messages a day with 1,000 of these being patches that need to be evaluated and possibly applied to the source base. Simply ignoring the traffic, figuring that the system in use seems to be working well enough, can lead to disastrous consequences later. For example, a recent security patch that took all of 13 lines of code to implement against an embedded Linux system would have taken more than 800k lines of source patches to implement if the previous trail of patches had been ignored. It's a classic case of pay now or really pay later.

Somebody must have had a great deal of fun putting all of those numbers together. The generation of ordinary random numbers can be managed through traditional methods like a toss of the dice, picking numbers out of a hat, or reading corporate earnings estimates. Randomness on this scale, though, can only be achieved through the use of special-purpose software.

Even by kernel.org standards, 5,000 messages per day is fairly intense, though your editor, a subscriber to the linux-kernel, git-commits-head, and mm-commits lists, can attest that the order of magnitude is right at least. But your editor cannot even begin to grasp the thought process which turns a 13-line security patch into 800,000 lines of code. Imagine posting that to linux-kernel. "Pay now or really pay later" indeed.

But the provenance of the numbers is not really the point here. Mr. Ready is perpetrating the fallacy that, to build an embedded system with Linux, one starts with the various components and integrates them all by hand. If a company were to take that path, it might well incur the high costs that Mr. Ready warns about. Creating your own distribution - and maintaining it over a product's life - is, indeed, a difficult and expensive job.

But it is a rare vendor which does that; even Gentoo users outsource much of the integration work to their distributor. Why would any vendor create its own distribution when there are so many out there to base a product on? Customizing a distribution for an embedded application is not a trivial job, but it's not rocket science either. The distributor will keep up with most of those mailing lists, and, somehow, a reasonable distribution also manages to ship security updates which do not involve 800,000 lines of code. There is no reason for embedded systems vendors to wander into the expensive mess that Mr. Ready describes; the creation of a suitable distribution is much easier than that.

Even so, many vendors may decide that, in fact, they would rather not be in the business of customizing distributions. They might, instead, look to a vendor to do that work for them. It makes perfect sense for companies like MontaVista and Wind River (among others) to offer to provide a stable, integrated, and supported platform to embedded systems vendors for a fee. There is honest value in this line of business.

But one does have to wonder why these companies feel the need to scare companies into buying their services. Those services, properly rendered, have a real value which can be sold without resort to outright FUD. Failure to focus on that value gives encouragement to people like Mr. O'Dowd, who would be most pleased if embedded Linux were to go away altogether. This does not seem like a sensible business strategy. Companies which seek to make money from Linux might just want to think twice before poisoning the well they are trying to drink from. That is the real lesson to be learned from this particular piece of writing.


(Log in to post comments)

How not to sell embedded Linux

Posted May 6, 2008 17:29 UTC (Tue) by drag (subscriber, #31333) [Link]

> Mr. Ready is perpetrating the fallacy that, to build an embedded system with Linux, one
starts with the various components and integrates them all by hand. If a company were to take
that path, it might well incur the high costs that Mr. Ready warns about. Creating your own
distribution - and maintaining it over a product's life - is, indeed, a difficult and
expensive job. 


I've seen it done.

It's tough coming from a PC-Linux background and seeing people who are experienced developers
otherwise deal with trying to roll their own Linux system when they are just trying to get
started.

It's absolutely a great learning experience and for that that approach can be valuable.. but
they don't quite understand that Linux systems available from other organizations  don't have
be taken at face value.. the benefit of Linux is a flexible system, but you do not have to do
it all by yourself  from scratch to get that.  But these developers, I think, generally only
see things from two perspectives... source code and proprietary binary distributions. That
is.. its either something you have to do yourself or you buy in a packaged deal from somebody
else. 

It's the middle ground between those two extremes is were the value is at. Take what you can
directly from distributions or vendors and benefit from their experience and widely used and
tested systems... and then modify it to suite your needs only when you have to. 

Each modification you make is more work for you. 

Linux distros provide the 90%, you have to do the 10%. 

That is the sort of thing that embedded developers often don't 'get' right away. Once they
start down the path of total customization then they can get trapped in it and it just makes
things more and more difficult and expensive as they diverge with aging code bases, most of
which ends up being a in-house fork and the rest of the Linux world moves on.

I think this sort of problem is more common then is apparent. For us experienced in PC or
Server stuff it's a no-brainer. For embedded developers it's not so obvious.

How not to sell embedded Linux

Posted May 7, 2008 8:24 UTC (Wed) by abacus (subscriber, #49001) [Link]

You are right that it's better to start from an embedded distribution 
when building an embedded device. The only problem with this approach is 
that the software of a typical embedded device has to be supported during 
ten years or more, while MontaVista and WindRiver typically support each 
version of their own distributions during only two or three years. This 
means that if you want to benefit from their support, after this time you 
are forced to upgrade everything (kernel, glibc, gcc, ...). Such an 
upgrade can be very expensive because of all the retesting work it 
implies.

How not to sell embedded Linux

Posted May 6, 2008 17:48 UTC (Tue) by ewan (subscriber, #5533) [Link]

your editor cannot even begin to grasp the thought process which turns a 13-line security patch into 800,000 lines of code

I think the 'logic' for that probably goes like this:

  • You fork off a nice stable copy of (say) 2.6.n and use it happily for a few months.
  • A security patch comes along, but the original patch is only made available for current(ish) kernels, say 2.6.n+10
  • To apply the patch you first have to apply the 800,000 lines of diffs to take you from 2.6.n to 2.6.n+10
PANIC!!

How not to sell embedded Linux

Posted May 6, 2008 18:17 UTC (Tue) by darwish07 (guest, #49520) [Link]

Any developer that worth his salary can see the context of the security patch modifications
and apply it to the old codebase (if the old one is still affected).

How not to sell embedded Linux

Posted May 13, 2008 13:18 UTC (Tue) by IkeTo (subscriber, #2122) [Link]

What if the original bug is fixable in version 2.6.n+10 because of the infrastructure that
created 800,000 lines of code added dispersed across version 2.6.n+5, 2.6.n+7 and 2.6.n+10?  I
admit that this is a rather hypothetical situation, though.

Lazy marketing

Posted May 6, 2008 19:32 UTC (Tue) by dmarti (subscriber, #11625) [Link]

Don't forget -- that's 800,000 lines of code from FOREIGN SECRET AGENTS!

Seriously, though, it's way easier to write "OMG WTF LINUX IS REALLY HARD AND WE HOLD YOUR HAND" marketing than to actually write up a Customer Win as an example of what an embedded vendor can do for its customer. Customers don't like to give permission, and by the time a problem is hard enough that it requires an embedded Linux company's Solutions Engineer (or whatever they call them) to save the customer's project, it's hard to explain in enough detail.

So, the embedded Linux industry could really stand to stop feeding the trolls in advance, and communicate about its products and services better.

How not to sell embedded Linux

Posted May 6, 2008 22:29 UTC (Tue) by caitlinbestler (subscriber, #32532) [Link]

The real issue is what sort of kernel your embedded application needs.

If you need a very trim very secure kernel that only provides a few services and does not need to support very many devices then trying to track kernel.org just does not make sense. The mainline is just not focused on your needs.

On the other hand, if you your embedded product is really more of a specialized standalone server that benefits from running all sorts of current networking code then trying to keep your hand-crafted mini-kernel and its supporting stack up to date does not make much sense.

When Mr. Ready first entered the embedded kernel business, the typical embedded kernel could not afford the overhead of a general purpose networking stack, and the networking requirements of embedded devices were simple and straight forward.

For many projects the trade-offs have changed radically since then.

How not to sell embedded Linux

Posted May 7, 2008 0:06 UTC (Wed) by dlang (subscriber, #313) [Link]

I'd argue exactly the opposite.

if you need a trim, secure kernel with minimal features enabled, definantly track kernel.org.
with a minimal config it's far easier to test a new kernel and any security patches that come
out you can more easily apply (assuming that they are at all relavent to things that you have
turned on in your kernel)

it's when you want the 'everything and the kitchen sink, running on any available hardware'
that you want to think about backing off and running a distro provided kernel (so that they
can do a lot of the work maintaining it)

How not to sell embedded Linux

Posted May 7, 2008 4:02 UTC (Wed) by jordip (subscriber, #47356) [Link]

I guess he is not trying to scare customers from their services but scare customers from
replacing them. Montavista obviously adds value to their products, the problem is that
companies with good engineers simply won't need it. 

As comments and original post noted, we are not talking about rocket science here and these
companies' bigger competition may be their own customers. As Linux evolves (for instance a
great deal of RT patches and other stuff that helps embedded Linux users is being gradually
merged in the 2.6 line) it becomes increasingly easy to roll your own Linux (from a free
distribution) modified for your needs. And thus avoiding buying Montavista's or other
company's solutions.

They obviously want companies to use their services instead of build their own.

Bad way then

Posted May 7, 2008 6:20 UTC (Wed) by man_ls (subscriber, #15091) [Link]

In that case I guess that they would be better off providing value in other areas, instead of just the kernel. According to LinuxDevices' annual survey, development tools are an all time favorite, but documentation and support would also be fine. These are things that no sane devs want to build on their own. But of course they take more effort than just tracking kernel.org and integrating a stream of patches.

Monta Vista does hold your hand

Posted May 7, 2008 7:51 UTC (Wed) by rvfh (subscriber, #31018) [Link]

I have had to work with Montavista in a previous job because they were writing drivers for us,
and they do provide tools and doc. They also found bugs in our hardware and came up with
solutions.

I can see that in two sentences I have sold their stuff better than with that FUD stuff from
Ready, and that's real technical stuff, stuff the customer is asking for!

All in all, we did not really need it for the distrib and tools, and went on using the one I
was maintaining on a previous version (not a product so no real security issue at stake), but
for a company with no Linux expert around or just no time for that, I can see clear benefits
in going with a MV or now WindRiver.

Thanks Jon for pointing this BTW, it's very instructive on how FUD is always bad, and might
serve interests opposite to yours ITE.

Monta Vista does hold your hand

Posted May 7, 2008 9:20 UTC (Wed) by man_ls (subscriber, #15091) [Link]

Maybe all this FUD is directed to pointy-haired bosses, and maybe it is effective with them. But any possible positive effect is offset by the negative effect on technical people. Which side has more decision power (management or techies) depends on how technical a company is; in the end, if only Dilbert-esque companies use your product then your reputation will probably suffer.

Monta Vista does hold your hand

Posted May 7, 2008 14:01 UTC (Wed) by sepreece (subscriber, #19270) [Link]

Ready's comments are largely aimed at companies that build embedded products and have been
using specialized real-time OSs, but are now looking at Linux and thinking "Hmm - get rid of
those licensing costs and I could make more money per device."

Those folks are used to buying an OS from a vendor that provides support and it's reasonably
important to make sure they understand that while you don't need to license Linux, you can't
expect to use it in products without some kind of support costs - either an in-house
development OS-development team or a specialist embedded-Linux distributor.

At a previous employer we determined that using an embedded distributor (MontaVista, in fact)
cost less than it would cost to staff the work we relied on them to do.

[Aside to Jon - you make fun of "the assertion that MontaVista and Wind River are "Linux's
strongest proponents." However, the actual assertion was that they were "embedded Linux's
strongest proponents", which is a claim MV could probably make a reasonable argument for.]


Data

Posted May 7, 2008 14:08 UTC (Wed) by alex (subscriber, #1355) [Link]

I would like to see how much of mvista's code is in the mainline kernel and how much is in
their own distro kernels. Granted they have a better record than WR but it still doesn't look
like much in the grans scheme of things:


[linux-2.6.git] >git-log --pretty=format:"%ae" | wc -l
87767
[linux-2.6.git] >git-log --pretty=format:"%ae" | grep "mvista" | wc -l
569

Data

Posted May 7, 2008 14:21 UTC (Wed) by rvfh (subscriber, #31018) [Link]

The O(1) scheduler was developed by Robert Love while at Montavista IIRC.

Data

Posted May 8, 2008 21:43 UTC (Thu) by im14u2c (subscriber, #5246) [Link]

Huh? The O(1) scheduler was developed by Ingo Molnar while he was at RedHat.

Data

Posted May 9, 2008 7:08 UTC (Fri) by rvfh (subscriber, #31018) [Link]

O(1) != CONFIG_PREEMPT

Posted May 9, 2008 7:21 UTC (Fri) by im14u2c (subscriber, #5246) [Link]

I guess I missed where this is the O(1) scheduler? In fact, that press release is largely content free. It just says "It's Not RTLinux."

I think you're thinking of CONFIG_PREEMPT, not the O(1) scheduler.

Perhaps this patch (ingo-O1-sched/preempt-kernel-rml-2.4.18-rc1-ingo-K3-1.patch) is what confused you? It's Robert Love's preempt patch against Ingo Molnar's O(1) scheduler.... :-)

How not to sell Linux

Posted May 7, 2008 7:55 UTC (Wed) by Hanno (subscriber, #41730) [Link]

Almost related to the topic, not long ago I got an unsolicited sales call from Redhat (!),
with the classic fud-based strategy used against Debian:

http://www.hanno.de/blog/2007/fud-in-the-linux-world/

How not to sell Linux

Posted May 8, 2008 8:43 UTC (Thu) by nim-nim (subscriber, #34454) [Link]

In a business context the Red Hat question is spot-on.

I've seen several cases where a Debian lover fool managed to get any kind of Linux blacklisted
in his organisation by putting his head in the sand and refusing to consider the support
question.

Real-world IT deployments (as opposed to local experiments) are all about managing risks,
which means considering risks, and risk is fear, so by your definition anyone engaging in
real-world deployment is FUD-ing.

FUD?

Posted May 9, 2008 6:37 UTC (Fri) by man_ls (subscriber, #15091) [Link]

Yes, you may not have liked it, but the question is legitimate. And I'm not sure it is FUD: the salesperson is not saying "your machines will go down in flames", but "when they fail, what will you do?".

The link you provide in the comments to your blog counters "I'm not going to use some unsupported hacker-ware on my system" with "Point them at http://www.caldera.com or http://www.linuxcare.com and point out that they can buy a service contract there". See, support is actually a valid thing to request in a business context. I use Debian on all my machines at home, but at work I would rather recommend Red Hat.

Going it alone, the relevance of the vendors diminishes

Posted May 7, 2008 14:05 UTC (Wed) by alex (subscriber, #1355) [Link]

One problem the embedded Linux vendors have is they are still trying to behave in proprietary
ways - and with Linux at the core of your embedded project it becomes increasingly clear the
vendors are trying to lock you into them.

The last embedded project I did with Linux (in fact the first I did) we held the option of
going for vendor supplied kernel open in case we needed harder real-time support. In the end
it turned out we didn't need that so all they had left to offer was fancy (proprietary) GUIs
and distro's which would have tied the project to the vendor if we had used them. Unless you
are going for a heavily configured PC type application rolling your own distro isn't that hard
and it means you have a good understanding of what is in your system and why.

I suspect as more and more RT and embedded patches make it into the mainline kernel
(windriver's 81 patches seeming rather poultry) the value proposition of embedded Linux
companies will look increasingly weak. There is a role for them in the space somewhere but
they need to prove their worth.


Going it alone, the relevance of the vendors diminishes

Posted May 7, 2008 15:20 UTC (Wed) by ewan (subscriber, #5533) [Link]

(windriver's 81 patches seeming rather poultry)

That's why they sound so scared. They're clearly chicken.

Going it alone, the relevance of the vendors diminishes

Posted May 8, 2008 21:45 UTC (Thu) by im14u2c (subscriber, #5246) [Link]

I suspect fowl play.

How not to sell embedded Linux

Posted May 8, 2008 20:01 UTC (Thu) by nhippi (subscriber, #34640) [Link]

"by Dan O'Dowd, Green Hills Software"

While it is clear to embedded.com readers, lwn.net readers might not be 
aware that Green Hills is a company that sells a proprietary embedded OS.

So it's just marketing as usual.

How not to sell embedded Linux

Posted May 8, 2008 20:59 UTC (Thu) by magnus (subscriber, #34778) [Link]

Take a look at Wind River or Montavista's web pages, they're so full of glossy marketing stuff
and buzzwords. Instead of saying the truth, "for some cash, we can save you a few weeks of
work in bringing up your Linux board", they're trying to build up some kind of hype. 

I think a lot of people decide to roll their own system (and with the help of projects like
Buildroot it can be quite painless, if you're lucky). In part this is because they're trying
to save money, but it's also an uncertainty on how much value these commercial vendors really
add. You can compare the situation to the desktop Linux world, where many people (me at least)
prefer upstream fixes over distribution-specific add-ons. 

Going from a modern PC Linux distribution into the embedded Linux world can sometimes feel a
bit like going back in time 10 years. It can sometimes be a bit rough around the edges, but
for an engineer that's also part of the fun. The joy of bringing up a board for the first time
and seeing the boot messages scrolling by over the serial port, priceless!

I'm pretty sure

Posted May 13, 2008 2:41 UTC (Tue) by Baylink (subscriber, #755) [Link]

that the point he was trying to make was "here's this security critical 9-line patch, which is
entirely dependent on these other patches, which are in turn heavily dependent on these other
patches, and by the time you get back to the major release, you're at 800kloc."

Whether that's a reasonable approach to take is open to question.

But it's at least not blue sky.

Copyright © 2008, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds
Powered by Rackspace Managed Hosting.