Weekly Edition Return to the Front pageSponsored link Serve your customers, not your servers, with VERIO Linux VPS. Full-access test-drive here. |
How not to sell embedded Linux
Every now and then one should have a look at some unabashed fear,
uncertainty, and doubt (FUD) material. It's good to know what the other
side is saying, the level of unintended humor is often high, and, on occasion, one even
learns something. Your editor's suggestion for FUD of the week is this Embedded.com
article by Dan O'Dowd. Therein, one will learn about the impending
death of embedded Linux as told by the companies which sell embedded Linux.
In particular, Mr. O'Dowd looks at some marketing material from MontaVista and Wind River, and concludes:
This embedded Linux bashing from embedded Linux's strongest
proponents should give pause to those who are thinking through
their embedded operating system strategy. If embedded Linux
champions are saying that embedded Linux is terrible, why would
anyone want to risk their products or their company on it?
One can easily pick holes in this article, starting with the assertion that MontaVista and Wind River are "Linux's strongest proponents." One could also recall that we have heard this kind of thing before; in 2004, Mr. O'Dowd (who happens to be the founder and CEO of a proprietary embedded systems software vendor) helpfully warned us that "intelligence agencies and terrorists" would contribute "subversive software" to Linux and lectured on the need for secret source code to achieve true security. One could point out that many of the points put forward by Mr. O'Dowd appear to be pure fantasy. All of these rebuttals would be valid, but they risk missing an important point to be gained from this article - though it's not quite the point Mr. O'Dowd is trying to make. Mr. O'Dowd obtains his "facts" from two sources: an advertisement by Wind River Systems (which your editor was unable to find online) and, primarily, from a column by MontaVista founder Jim Ready in Military Embedded Systems magazine. Mr. Ready's evident purpose is to frighten embedded systems vendors into buying his company's services; to that end, he lays it on pretty thick:
To keep abreast of the changes occurring on a daily basis, a
developer needs to monitor the email traffic of 11 different and
unsynchronized open source projects: kernel.org, the core home of
the Linux kernel; the gcc and glibc projects (the core tool chain
and libraries from FSF at fsf.org); and at least nine other
components that would typically comprise a useable Linux
development environment.
Kernel.org itself may have up to 5,000 messages a day with 1,000 of these being patches that need to be evaluated and possibly applied to the source base. Simply ignoring the traffic, figuring that the system in use seems to be working well enough, can lead to disastrous consequences later. For example, a recent security patch that took all of 13 lines of code to implement against an embedded Linux system would have taken more than 800k lines of source patches to implement if the previous trail of patches had been ignored. It's a classic case of pay now or really pay later. Somebody must have had a great deal of fun putting all of those numbers together. The generation of ordinary random numbers can be managed through traditional methods like a toss of the dice, picking numbers out of a hat, or reading corporate earnings estimates. Randomness on this scale, though, can only be achieved through the use of special-purpose software. Even by kernel.org standards, 5,000 messages per day is fairly intense, though your editor, a subscriber to the linux-kernel, git-commits-head, and mm-commits lists, can attest that the order of magnitude is right at least. But your editor cannot even begin to grasp the thought process which turns a 13-line security patch into 800,000 lines of code. Imagine posting that to linux-kernel. "Pay now or really pay later" indeed. But the provenance of the numbers is not really the point here. Mr. Ready is perpetrating the fallacy that, to build an embedded system with Linux, one starts with the various components and integrates them all by hand. If a company were to take that path, it might well incur the high costs that Mr. Ready warns about. Creating your own distribution - and maintaining it over a product's life - is, indeed, a difficult and expensive job. But it is a rare vendor which does that; even Gentoo users outsource much of the integration work to their distributor. Why would any vendor create its own distribution when there are so many out there to base a product on? Customizing a distribution for an embedded application is not a trivial job, but it's not rocket science either. The distributor will keep up with most of those mailing lists, and, somehow, a reasonable distribution also manages to ship security updates which do not involve 800,000 lines of code. There is no reason for embedded systems vendors to wander into the expensive mess that Mr. Ready describes; the creation of a suitable distribution is much easier than that. Even so, many vendors may decide that, in fact, they would rather not be in the business of customizing distributions. They might, instead, look to a vendor to do that work for them. It makes perfect sense for companies like MontaVista and Wind River (among others) to offer to provide a stable, integrated, and supported platform to embedded systems vendors for a fee. There is honest value in this line of business. But one does have to wonder why these companies feel the need to scare companies into buying their services. Those services, properly rendered, have a real value which can be sold without resort to outright FUD. Failure to focus on that value gives encouragement to people like Mr. O'Dowd, who would be most pleased if embedded Linux were to go away altogether. This does not seem like a sensible business strategy. Companies which seek to make money from Linux might just want to think twice before poisoning the well they are trying to drink from. That is the real lesson to be learned from this particular piece of writing. (Log in to post comments)
How not to sell embedded Linux Posted May 6, 2008 17:29 UTC (Tue) by drag (subscriber, #31333) [Link] > Mr. Ready is perpetrating the fallacy that, to build an embedded system with Linux, one starts with the various components and integrates them all by hand. If a company were to take that path, it might well incur the high costs that Mr. Ready warns about. Creating your own distribution - and maintaining it over a product's life - is, indeed, a difficult and expensive job. I've seen it done. It's tough coming from a PC-Linux background and seeing people who are experienced developers otherwise deal with trying to roll their own Linux system when they are just trying to get started. It's absolutely a great learning experience and for that that approach can be valuable.. but they don't quite understand that Linux systems available from other organizations don't have be taken at face value.. the benefit of Linux is a flexible system, but you do not have to do it all by yourself from scratch to get that. But these developers, I think, generally only see things from two perspectives... source code and proprietary binary distributions. That is.. its either something you have to do yourself or you buy in a packaged deal from somebody else. It's the middle ground between those two extremes is were the value is at. Take what you can directly from distributions or vendors and benefit from their experience and widely used and tested systems... and then modify it to suite your needs only when you have to. Each modification you make is more work for you. Linux distros provide the 90%, you have to do the 10%. That is the sort of thing that embedded developers often don't 'get' right away. Once they start down the path of total customization then they can get trapped in it and it just makes things more and more difficult and expensive as they diverge with aging code bases, most of which ends up being a in-house fork and the rest of the Linux world moves on. I think this sort of problem is more common then is apparent. For us experienced in PC or Server stuff it's a no-brainer. For embedded developers it's not so obvious.
How not to sell embedded Linux Posted May 7, 2008 8:24 UTC (Wed) by abacus (subscriber, #49001) [Link] You are right that it's better to start from an embedded distribution when building an embedded device. The only problem with this approach is that the software of a typical embedded device has to be supported during ten years or more, while MontaVista and WindRiver typically support each version of their own distributions during only two or three years. This means that if you want to benefit from their support, after this time you are forced to upgrade everything (kernel, glibc, gcc, ...). Such an upgrade can be very expensive because of all the retesting work it implies.
How not to sell embedded Linux Posted May 6, 2008 17:48 UTC (Tue) by ewan (subscriber, #5533) [Link] your editor cannot even begin to grasp the thought process which turns a 13-line security patch into 800,000 lines of codeI think the 'logic' for that probably goes like this:
How not to sell embedded Linux Posted May 6, 2008 18:17 UTC (Tue) by darwish07 (guest, #49520) [Link] Any developer that worth his salary can see the context of the security patch modifications and apply it to the old codebase (if the old one is still affected).
How not to sell embedded Linux Posted May 13, 2008 13:18 UTC (Tue) by IkeTo (subscriber, #2122) [Link] What if the original bug is fixable in version 2.6.n+10 because of the infrastructure that created 800,000 lines of code added dispersed across version 2.6.n+5, 2.6.n+7 and 2.6.n+10? I admit that this is a rather hypothetical situation, though.
Lazy marketing Posted May 6, 2008 19:32 UTC (Tue) by dmarti (subscriber, #11625) [Link] Don't forget -- that's 800,000 lines of code from FOREIGN SECRET AGENTS!Seriously, though, it's way easier to write "OMG WTF LINUX IS REALLY HARD AND WE HOLD YOUR HAND" marketing than to actually write up a Customer Win as an example of what an embedded vendor can do for its customer. Customers don't like to give permission, and by the time a problem is hard enough that it requires an embedded Linux company's Solutions Engineer (or whatever they call them) to save the customer's project, it's hard to explain in enough detail. So, the embedded Linux industry could really stand to stop feeding the trolls in advance, and communicate about its products and services better.
How not to sell embedded Linux Posted May 6, 2008 22:29 UTC (Tue) by caitlinbestler (subscriber, #32532) [Link] The real issue is what sort of kernel your embedded application needs. If you need a very trim very secure kernel that only provides a few services and does not need to support very many devices then trying to track kernel.org just does not make sense. The mainline is just not focused on your needs. On the other hand, if you your embedded product is really more of a specialized standalone server that benefits from running all sorts of current networking code then trying to keep your hand-crafted mini-kernel and its supporting stack up to date does not make much sense. When Mr. Ready first entered the embedded kernel business, the typical embedded kernel could not afford the overhead of a general purpose networking stack, and the networking requirements of embedded devices were simple and straight forward. For many projects the trade-offs have changed radically since then.
How not to sell embedded Linux Posted May 7, 2008 0:06 UTC (Wed) by dlang (subscriber, #313) [Link] I'd argue exactly the opposite. if you need a trim, secure kernel with minimal features enabled, definantly track kernel.org. with a minimal config it's far easier to test a new kernel and any security patches that come out you can more easily apply (assuming that they are at all relavent to things that you have turned on in your kernel) it's when you want the 'everything and the kitchen sink, running on any available hardware' that you want to think about backing off and running a distro provided kernel (so that they can do a lot of the work maintaining it)
How not to sell embedded Linux Posted May 7, 2008 4:02 UTC (Wed) by jordip (subscriber, #47356) [Link] I guess he is not trying to scare customers from their services but scare customers from replacing them. Montavista obviously adds value to their products, the problem is that companies with good engineers simply won't need it. As comments and original post noted, we are not talking about rocket science here and these companies' bigger competition may be their own customers. As Linux evolves (for instance a great deal of RT patches and other stuff that helps embedded Linux users is being gradually merged in the 2.6 line) it becomes increasingly easy to roll your own Linux (from a free distribution) modified for your needs. And thus avoiding buying Montavista's or other company's solutions. They obviously want companies to use their services instead of build their own.
Bad way then Posted May 7, 2008 6:20 UTC (Wed) by man_ls (subscriber, #15091) [Link] In that case I guess that they would be better off providing value in other areas, instead of just the kernel. According to LinuxDevices' annual survey, development tools are an all time favorite, but documentation and support would also be fine. These are things that no sane devs want to build on their own. But of course they take more effort than just tracking kernel.org and integrating a stream of patches.
Monta Vista does hold your hand Posted May 7, 2008 7:51 UTC (Wed) by rvfh (subscriber, #31018) [Link] I have had to work with Montavista in a previous job because they were writing drivers for us, and they do provide tools and doc. They also found bugs in our hardware and came up with solutions. I can see that in two sentences I have sold their stuff better than with that FUD stuff from Ready, and that's real technical stuff, stuff the customer is asking for! All in all, we did not really need it for the distrib and tools, and went on using the one I was maintaining on a previous version (not a product so no real security issue at stake), but for a company with no Linux expert around or just no time for that, I can see clear benefits in going with a MV or now WindRiver. Thanks Jon for pointing this BTW, it's very instructive on how FUD is always bad, and might serve interests opposite to yours ITE.
Monta Vista does hold your hand Posted May 7, 2008 9:20 UTC (Wed) by man_ls (subscriber, #15091) [Link] Maybe all this FUD is directed to pointy-haired bosses, and maybe it is effective with them. But any possible positive effect is offset by the negative effect on technical people. Which side has more decision power (management or techies) depends on how technical a company is; in the end, if only Dilbert-esque companies use your product then your reputation will probably suffer.
Monta Vista does hold your hand Posted May 7, 2008 14:01 UTC (Wed) by sepreece (subscriber, #19270) [Link] Ready's comments are largely aimed at companies that build embedded products and have been using specialized real-time OSs, but are now looking at Linux and thinking "Hmm - get rid of those licensing costs and I could make more money per device." Those folks are used to buying an OS from a vendor that provides support and it's reasonably important to make sure they understand that while you don't need to license Linux, you can't expect to use it in products without some kind of support costs - either an in-house development OS-development team or a specialist embedded-Linux distributor. At a previous employer we determined that using an embedded distributor (MontaVista, in fact) cost less than it would cost to staff the work we relied on them to do. [Aside to Jon - you make fun of "the assertion that MontaVista and Wind River are "Linux's strongest proponents." However, the actual assertion was that they were "embedded Linux's strongest proponents", which is a claim MV could probably make a reasonable argument for.]
Data Posted May 7, 2008 14:08 UTC (Wed) by alex (subscriber, #1355) [Link] I would like to see how much of mvista's code is in the mainline kernel and how much is in their own distro kernels. Granted they have a better record than WR but it still doesn't look like much in the grans scheme of things: [linux-2.6.git] >git-log --pretty=format:"%ae" | wc -l 87767 [linux-2.6.git] >git-log --pretty=format:"%ae" | grep "mvista" | wc -l 569
Data Posted May 7, 2008 14:21 UTC (Wed) by rvfh (subscriber, #31018) [Link] The O(1) scheduler was developed by Robert Love while at Montavista IIRC.
Data Posted May 8, 2008 21:43 UTC (Thu) by im14u2c (subscriber, #5246) [Link] Huh? The O(1) scheduler was developed by Ingo Molnar while he was at RedHat.
O(1) != CONFIG_PREEMPT Posted May 9, 2008 7:21 UTC (Fri) by im14u2c (subscriber, #5246) [Link] I guess I missed where this is the O(1) scheduler? In fact, that press release is largely content free. It just says "It's Not RTLinux." I think you're thinking of CONFIG_PREEMPT, not the O(1) scheduler. Perhaps this patch (ingo-O1-sched/preempt-kernel-rml-2.4.18-rc1-ingo-K3-1.patch) is what confused you? It's Robert Love's preempt patch against Ingo Molnar's O(1) scheduler.... :-)
How not to sell Linux Posted May 7, 2008 7:55 UTC (Wed) by Hanno (subscriber, #41730) [Link] Almost related to the topic, not long ago I got an unsolicited sales call from Redhat (!), with the classic fud-based strategy used against Debian: http://www.hanno.de/blog/2007/fud-in-the-linux-world/
How not to sell Linux Posted May 8, 2008 8:43 UTC (Thu) by nim-nim (subscriber, #34454) [Link] In a business context the Red Hat question is spot-on. I've seen several cases where a Debian lover fool managed to get any kind of Linux blacklisted in his organisation by putting his head in the sand and refusing to consider the support question. Real-world IT deployments (as opposed to local experiments) are all about managing risks, which means considering risks, and risk is fear, so by your definition anyone engaging in real-world deployment is FUD-ing.
FUD? Posted May 9, 2008 6:37 UTC (Fri) by man_ls (subscriber, #15091) [Link] Yes, you may not have liked it, but the question is legitimate. And I'm not sure it is FUD: the salesperson is not saying "your machines will go down in flames", but "when they fail, what will you do?".The link you provide in the comments to your blog counters "I'm not going to use some unsupported hacker-ware on my system" with "Point them at http://www.caldera.com or http://www.linuxcare.com and point out that they can buy a service contract there". See, support is actually a valid thing to request in a business context. I use Debian on all my machines at home, but at work I would rather recommend Red Hat.
Going it alone, the relevance of the vendors diminishes Posted May 7, 2008 14:05 UTC (Wed) by alex (subscriber, #1355) [Link] One problem the embedded Linux vendors have is they are still trying to behave in proprietary ways - and with Linux at the core of your embedded project it becomes increasingly clear the vendors are trying to lock you into them. The last embedded project I did with Linux (in fact the first I did) we held the option of going for vendor supplied kernel open in case we needed harder real-time support. In the end it turned out we didn't need that so all they had left to offer was fancy (proprietary) GUIs and distro's which would have tied the project to the vendor if we had used them. Unless you are going for a heavily configured PC type application rolling your own distro isn't that hard and it means you have a good understanding of what is in your system and why. I suspect as more and more RT and embedded patches make it into the mainline kernel (windriver's 81 patches seeming rather poultry) the value proposition of embedded Linux companies will look increasingly weak. There is a role for them in the space somewhere but they need to prove their worth.
Going it alone, the relevance of the vendors diminishes Posted May 7, 2008 15:20 UTC (Wed) by ewan (subscriber, #5533) [Link] (windriver's 81 patches seeming rather poultry)That's why they sound so scared. They're clearly chicken.
Going it alone, the relevance of the vendors diminishes Posted May 8, 2008 21:45 UTC (Thu) by im14u2c (subscriber, #5246) [Link] I suspect fowl play.
How not to sell embedded Linux Posted May 8, 2008 20:01 UTC (Thu) by nhippi (subscriber, #34640) [Link] "by Dan O'Dowd, Green Hills Software" While it is clear to embedded.com readers, lwn.net readers might not be aware that Green Hills is a company that sells a proprietary embedded OS. So it's just marketing as usual.
How not to sell embedded Linux Posted May 8, 2008 20:59 UTC (Thu) by magnus (subscriber, #34778) [Link] Take a look at Wind River or Montavista's web pages, they're so full of glossy marketing stuff and buzzwords. Instead of saying the truth, "for some cash, we can save you a few weeks of work in bringing up your Linux board", they're trying to build up some kind of hype. I think a lot of people decide to roll their own system (and with the help of projects like Buildroot it can be quite painless, if you're lucky). In part this is because they're trying to save money, but it's also an uncertainty on how much value these commercial vendors really add. You can compare the situation to the desktop Linux world, where many people (me at least) prefer upstream fixes over distribution-specific add-ons. Going from a modern PC Linux distribution into the embedded Linux world can sometimes feel a bit like going back in time 10 years. It can sometimes be a bit rough around the edges, but for an engineer that's also part of the fun. The joy of bringing up a board for the first time and seeing the boot messages scrolling by over the serial port, priceless!
I'm pretty sure Posted May 13, 2008 2:41 UTC (Tue) by Baylink (subscriber, #755) [Link] that the point he was trying to make was "here's this security critical 9-line patch, which is entirely dependent on these other patches, which are in turn heavily dependent on these other patches, and by the time you get back to the major release, you're at 800kloc." Whether that's a reasonable approach to take is open to question. But it's at least not blue sky.
|
Copyright © 2008, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds
Powered by Rackspace Managed Hosting.