Interview: Wind River's John Bruggeman

The market changed, and I think that open source became a very, very important part of the addressable market we wanted to reach. And if Wind River was going to be relevant and going to be important in the marketplace, we would have to have an open source and specifically a Linux-based solution for our customers. So, basically, the market thrust us into it, demanded that we do it, and I think it was all for the best that that happened.

The embedded marketplace has requirements that aren't in the general enterprise computing market. Things like size becomes very critical, and memory utilization and power management and some other features like that. Standard Linux wasn't optimized or suited for device types that face those challenges.

Those are kind of software elements, but there is also a hardware element. In the enterprise computing space, you are basically living in an [Intel architecture] world and everything is pretty constant and stable and predictable. Well, that is the anti-case with what we see in embedded. You have a plethora of hardware environments. Each hardware environment has their own specific nuances and special techniques and tips and trips. And making Linux work really well with hardware is a tough problem.

VxWorks is where you need absolute real-time determinism, where you need things like safety and security, [and to] meet certain regulatory standards and certification standards: those kinds of applications are the sweet spot for our VxWorks software. More general solutions, where application availability, middleware integration, [and] where lots and lots of ecosystem partners are required, that's in the sweet spot of our Linux software.

I think, over time, probably not. But, that's a long time way. A great example of that would be security certification for an airplane. The standards and the requirements to meet those certifications are very, very complex. They are very difficult and I think Linux is a long way away from being able to do that.

Today about 80% of our revenue is VxWorks, but the fastest-growing segment of our business is Linux. It's growing in the triple digits quarter over quarter over quarter. We announced it well north of $50 million for us this year.

Wholly? I don't think so. There will always be certain types of devices in which VxWorks will be a superior solution. But the Linux portion of our business will continue to grow, and I see a day where our Linux business is every bit as big as the VxWorks business.

Let me start with Linux in general. The first is availability of the ecosystem. The need to accelerate the pace of development is becoming critical. Many, many of our customers used to be vertical integrators - they even manufactured their own silicon and they would go all the way up to the top. And we're seeing a change that's happening at light speed, where they are shifting from a vertical integrator to an application developer. And they are really differentiating themselves on the user experience, on the type of applications they develop.

The attraction of Linux is there's this massive development community developing that infrastructure stuff that they used to spend so much time on, that enabled application development: they don't have to do that anymore. The second thing is obviously cost. They really can get it at a significantly lower development cost than they did when they used to have to build it themselves.

We provide things like integration testing and validation. Open source is a bunch of packages and the magic is how well are they put together and how reliable are they, and how well has that been tested, and can you validate and stand behind that? We have over 300 support engineers located globally around the world, in different time zones. We have the richest indemnity and warranty program in the industry. We don't stand behind Wind River, we stand behind open source.

Linux has the opportunity to revolutionize the mobile phone space - not just smart phones, but feature phones, converged phones, [Mobile Internet Devices - MIDs]. What's holding it back right now is the fragmentation. There are just way too many different Linux distributions. What that means is the ecosystem can't aggregate and surround anything of any critical mass. So, two initiatives have broken out that seem to be aggregators or consolidators: one is LiMo and one is Android. We're not smart enough to know which one is going to be the ultimate consolidator, so we're tremendously active in both.

We joined LiMo as a board member and we work very, very hard with the architectural committee to become the Linux foundation for all LiMo-based development. What that means is the common integration environment, which is the Linux-built system, the tool chain, is all based on Wind River technology. And therefore any contribution that's made to LiMo [is] based on our technology - we contributed that common integration environment to the LiMo foundation.

[Open Handset Alliance's Android] was announced about six or nine months or so after LiMo, and Google came out and said Wind River is their Linux commercialization partner. We have been working with them for about two years. We've done a number of hardware integrations for them. That's one of our core competences: how do you get Android running on the hardware.

We have phones coming out for both. We see a lot of activity on both and a lot of momentum for both.

LiMo truly is a consortium of equals. There are multiple operators: Vodafone, Docomo, Verizon, Orange, others. A bunch of carriers and a bunch of handset OEMs: Motorola, Samsung, LG, Panasonic, NEC. And the board is made up of those guys and Wind River. And we see that really is sort of: how do we get a common ground between fierce competitors? How do we, for the good of the industry, standardize around that stuff that's non-differentiating?

OHA is really a Google-driven initiative. They make product decisions and they make feature decisions.

So, let's talk pros and cons about this. When it's not a democracy, when the decision-making is very clear, decisions can be made quickly and things move very fast. On the LiMo side, where it's a lot of people, with a lot of experience building phones, who know what really matters, and what's important and what works and what doesn't work, they can bring a lot of different experience, a wealth of different perspectives together.

Sometimes it might take a little longer to make a decision over here but I really understand and can see why that decision works over there. Where this one races ahead, this one's a little more methodical and carefully constructed. But they're both building compelling platforms and will both be successful in the marketplace.

If you look at the smartphone market, it's 7% today of the total phone marketplace. So, from a percentage basis, it's not big. But what we're seeing is more and more feature phone-like capabilities blurring with the smartphone. So even though it's a small part of the market today, it's very strategic, because it does have implications down-market on the feature phones.

Symbian's got 60% of the smartphone market. And Microsoft's 20 to 30% of that market. Certainly they are not among equals, but Microsoft's been gaining share against Symbian and against Nokia. So, I think this was an aggressive and a bold and clever move against Microsoft.

Vis-a-vis Linux, the Symbian move just endorsed what was going on. It said if you're going to be competitive, if you're going to relevant years from now, you'd better have an open source model. I love that endorsement of Linux.

On the other hand, their solution is years away. Nokia said: Well, we'll have it in the first half in 2010. Both Android and LiMo will have phones out by the end of this year. So, there should be a lot of activity. Now if I'm an ecosystem member, am I going to wait for 2010, or am I going to develop today, and address real design opportunities and real win opportunities today?

I think Linux has a window of opportunity. We're going to see mass adoption of Linux-based devices, whether they are phones, or converged devices or MIDs, or whatever they are. However this market evolves, Linux is going to have two years' worth of product out there in the marketplace, doing stuff, before we see Symbian open source. While Nokia made a brilliant and bold move, it might be too late, because there is enough Linux momentum, especially behind OHA and LiMo, that I think they left that too long.

Apple will always be what Apple is. Apple is just fantastic, touches the super, niche, high end - somebody willing to pay $700 for a phone. And there is a big market for that - if you think a big market is 10 million phones. That's going to be there and that's not threatened or messed with in any of this stuff, because they are always going to come out with some really creative form factor or killer application: they are going to touch 10 million people. Three years from now we'll see a couple billion phones in the marketplace. So, let Apple go be content with that [10 million]. Let RIM go hit their niche part of the market. I don't see that catching fire.

I think all three devices meet certain use cases. I don't see, in the near future, or even the mid-term future, a MID overtaking a phone. There's a reason people talk on phones, but there's this whole different class of people in different use scenarios, they need a MID.

What is becoming very, very clear is, it's not about voice and it's not about text or email, it's going to be about a true, rich Internet experience. Can a web page be represented on these devices at the same clarity, the same quality, the same speed, as they are on the PC? When I look at YouTube, I don't want to look at a fuzzy, webcam image. I want to see [High Definition] quality on that thing. So, the devices we're seeing today, they're being required to be able to deliver that level of video representation and audio, that's [as good as] my music device and that's as good as my home entertainment system.

One of the fastest-growing areas of Linux we see right now is in the automobile: in the in-vehicle entertainment, in the dashboard, in the navigation. Those, for years and years and years, have been relegated to proprietary software stacks, because there's this big stigma that an automobile is hard. It moves and it bumps and there's temperature and there's all these safety requirements, and that's proprietary stuff.

I think Apple helped change the game, because everybody wanted their iPod in their car without a bunch of wire striking around. Automobile manufacturers worked on the development cycle that is five to seven years, and all of a sudden the iPod hits and they have one quarter to figure out how to get that thing in there.

This is a whole new business and process problem that the automotive manufacturers had not been in before. They all stood up and said: We don't know how to do this. And then the next new application came in and the next new application and, all of a sudden, they said: There's been a tremendous disruption in the industry; we've got to change the underlying principles how we design these applications. And Linux is clearly the solution for that, because it's all about the application and how extensible can the platform be, and how well can we count on consumer-like speed in an automotive-like marketplace.

The second market that I would say we're seeing in the home. Things like broadband access points - how you get content into the house: that's going Linux now. Every new data standard, Linux is keeping pace with that better than anything else out there.

We're seeing a general theme here. There's a real need for content - I want YouTube and I want cable and I want satellite and I want data. We're seeing those three C's of content, of connectivity, and of complexity. When you have those three things there, Linux is a tremendous solution.

>A great example of that would be security certification for an airplane. The standards and
the requirements to meet those certifications are very, very complex. 

Now, are these standards complex because the problem domain is complex, or are these standards
complex because that situation suits somebody's (not necessarily Wind River's) business model?

It depends on what you would call complex. Describing a triply redundant system appears to me
to be of the same complexity as say parallel programming. Designing a high availability data
driven website (Google, Slashdot, et al) may even compare to a degree, it's a different
domain, but the problem is largely the same. Those inter-bank payment systems and flight
booking systems probably come close, where data lossage is an absolute no-no.

It might just be he calls it "very, very complex" because those standards are hard to read for
all the legalese they're couched in.

In other words, I'm guessing as much as you are.

I suspect it's more a matter of the process to get the software certified being complex,
rather than the standards themselves. 

For real high-integrity software, I think it would be very difficult to get Linux certified. A
big part of it is a verification of the development process, and you'd have an incredibly hard
time showing that Linux went through any sort of rigor (mainly because it didn't ;p). You
would probably have to take a snapshot of the kernel, rip out everything that isn't needed,
and then do code review and testing in-house, replacing or removing things that fail review or
don't conform to MISRA-C. By the time that's done whatever benefit there was to using Linux
over an in-house OS would likely be gone. 

Does the process serve to inflate the profits of a company like Wind River? Probably, but then
again, Aerospace has a culture of safety that doesn't exist in most other areas where software
is developed and used. The fact that Avionics have been remarkably free of people-killing bugs
is a pretty good validation of the process.

Software for airborne systems must be DO178B certified in the US, in 
Europe there's a similar standard, can't remember the abbrev. right now. 
There are different levels for that, depending on the criticality of the 
software. E.g. the inflight entertainment system requires a less strict 
level of certification than the fly-by-wire software, since this can kill 
people.

For the highest criticality levels you need things like testing with 100% 
code coverage, you need to track all requirements and you have to be able 
to document in which lines of code each requirement is implemented, you 
must not have code where you don't have a requirement for it, each line 
of code must be "signed" by at least two developers, etc.

I think the Linux kernel is just too big and moving too fast to do this. 
Or, as somebody else already said, if you snapshot a kernel, strip out 
unneeded drivers, then start the testing, documenting etc., you are 
probably not better off than with another solution (months or years 
behind Linus tree, patches don't apply, behaviour is different because 
you changed so much, etc.).

So for these systems really a small OS (in LOC) is a good choice, it is 
just easier to certifiy (there are also free RTOS). RTOS in general are 
not necessarily something very sophisticated or complex, often they are 
actually quite simple and stripped down compared to a general purpose OS. 
But this makes them easier predictable and also certifiable.

Alex

European "mirror" of the DO-178B is called ED-12B. It is just a copy, since the DO-178B was
developed by both RTCA and EUROCAE.
DO-178B "name" is widely used in EU. We used to name DO-178B instead ED-12B all the time :)

DO-178B is not only a certification, it is a process that starts the same day as the
development project itself. It will be very hard for an existing application get certified for
any DO-178B level and of course, definitively almost imposible to get Level A certification.

The interview lightly touches upon a thought that crossed my mind: with more people watching
Internet video and live feeds on their phones and Linux gaining ground in the mobile market
space, who knows, we finally might end up with widely used high quality open spec/source audio
& video formats.
Then again, that might just be me, over optimistic, wishful thinking :)

More than likely they'll implement a closed source player on top of the open bits, but as you
say it would be a good opportunity for open formats.

It's not at all unthinkable to have x264 video on those devices, decoding of which can be
offloaded to various media extensions some of these embedded platforms have.

As a recent owner of a Freerunner telephone (well, more like a "device" actually), I would
have been interested in hearing this guy's point of view on Openmoko and Freesmartphone (.org)
stuff.  The way it is, I'm left wondering whether he just doesn't have anything to say or
whether he's deliberately ignoring it as irrelevant or as in "first they ignore you".

Sorry that this might be off-topic, but I want to hear your opinion on OpenMoko's mobile
phone, because I am willing to buy it. Is it worth it? What could be improved? 

Please reply

Miladin

The hardware is very attractive: GSM/GPRS, GPS, Bluetooth, Wifi, USB (both ways),
accelerometers, excellent screen resolution, a CPU powerful enough, all that in a phone-sized
device.  Yummy.

The software part is not ready for the masses yet, and clearly announced as such.  There are
at least three different software stacks (GTK-based Openmoko 2007.2, QT-based April/August
software update, and Freesmartphone.org framework-based); I think there's a fourth one
(Qtopia-based), and people speak of an hypothetical fifth ("stable hybrid release").  And some
have installed Debian on it, too.  I have only tried the OM2007.2 one, which mostly works for
me as a phone+GPS (with a few manual tweaks).  And the open hardware and software is really
lovely to work with: many things can be controlled through Dbus and/or Gconf, so you can write
small Python scripts to, for instance, import/export your phone contacts to VCF, or tune the
power saving mode, or whatever.  This should be even more so with the generalised move to the
FSO stack.

I'm still quite impressed by the community though: there used to be a GPS problem, and the
wisdom went from "blargh, internal antenna connector isn't soldered properly, you'll have to
fix that yourself" to "hey, it works better when there's no SD card" to "okay, fix committed
in kernel, don't bother soldering anything" in about a week.  One opkg update && opkg upgrade
later, everyone was happily recording GPS tracks for Openstreetmap :-)

All in all, I'm quite pleased with it.  It is, to me, the (current) ultimate toy for smug
geeks: rare, versatile, hackable, and (for the smug part) restricted to those elitist bastards
who can handle a command-line for upgrades.

I have been told the GSM driver uses a non-free software BLOB, and maybe too the Bluetooth,
Wifi, GPS and GPU chips.

Do you know if that is real?

As far as I know, that was only true for the Neo 1973, and has been fixed for the Freerunner.
I'm no expert however, please consult the nice wiki.

It seems Openmoko has been forced to sign a _super_ NDA to be able to use the Ti Calypso GSM
chipset.   

You can read it at http://lists.openmoko.org/pipermail/community/2007-Januar...


I do not know yet about the Wifi, GPS an other Openmoko devices.