LWN.net Logo

Some views from Vision

By Jonathan Corbet
October 7, 2008
Your editor had the honor of speaking at MontaVista's Vision 2008 conference recently. This conference - a gathering of MontaVista's customers - provided an opportunity to observe how (part of) the embedded industry sees itself and its role in the larger Linux community. Relations between embedded systems and Linux as a whole have often been a little uneasy; a situation which probably will not change in the near future. That said, there are signs that embedded developers are starting to think about the value of engaging more directly with the development community that they depend on.

William Mills is the Chief Technologist for Open Linux Solutions at Texas Instruments; his brief presentation at Vision was an interesting demonstration of how attitudes in the industry are changing. According to Mr. Mills, TI's method for developing Linux drivers for its products involved doing the work behind closed doors, then distributing the result through MontaVista. That approach has changed, though. TI now does its driver work in a public git tree, with a focus on merging the code upstream as a first priority. Customers who want to work directly with upstream kernels can get the code directly.

In a sense, it would appear that TI has removed MontaVista as the intermediary which distributes drivers for TI hardware. But TI still distributes code through MontaVista, so customers looking for a supported, integrated offering can still get a distribution which suits their needs. There's no shortage of embedded systems vendors who lack the skills and the desire to support a Linux distribution themselves; for those vendors, buying a supported system makes a lot of sense. For everybody else, the software is free and part of the mainline kernel, as it should be.

MontaVista founder Jim Ready discussed "the state of embedded Linux," focusing on areas where there is a bit of a mismatch between what the Linux community is providing and what the embedded industry needs. Certain kinds of functionality are missing; the ability to do user-space interrupt synchronization was one example. The rate of change in the kernel is very high, presenting embedded vendors with the difficult choice of backporting fixes or upgrading to a more recent kernel. Tracing and profiling tools are not up to the level needed by the industry.

Jim also talked some about realtime functionality, which currently must be patched into the kernel separately. He complained that changes made to the mainline kernel often break the realtime patch sets, leaving developers scrambling to make things work again. Keeping these patches in a working state requires constant effort; it is a significant cost.

All of this may sound like whining from an industry which has earned a reputation for taking more from Linux than it is willing to put back in. But Jim put the blame directly on the embedded industry itself; embedded vendors, he says, still haven't quite gotten it. While taking some pride in MontaVista's position in the list of top contributors to the kernel, he suggested that MontaVista should be enjoying the company of more embedded systems firms. The embedded industry should be contributing more to the kernel than it is.

What it comes down to, says Jim, is that the center of gravity in the Linux development world can be found in enterprise computing. Vendors in that industry are contributing heavily to the kernel and, as a result, the kernel tends to fit their needs better. The embedded community needs to get together and figure out how it, too, can become a more prominent contributor and work to drive the kernel in directions which suit its needs.

Judging from the response in the room, many of those in the audience seem to agree with this point of view. Some see it differently, though. During your editor's talk, a member of the audience asked whether the embedded community should stop using a kernel developed by enterprise system vendors and, instead, make its own version of the kernel suited to its needs. Needless to say, your editor discouraged this approach; the cost of forking the kernel and fragmenting the development community would vastly exceed the value of any benefits gained. But the questioner seemed unconvinced.

The clear conclusion to be made from that exchange is that there are still people in the embedded industry who do not see the value of working with the larger Linux development community. It is easy to fault the embedded community for its failure to contribute back, but it also makes sense to look in the mirror and ask if we couldn't make a more persuasive case for joining in. There has been a sustained effort to encourage the embedded systems industry to become a full participant in our community; over the years, that work has yielded a steady stream of successes. By continuing and improving this work, we'll continue the process of bringing our community together. Then we'll truly have a single system that runs on everything from wrist watches to supercomputers.


(Log in to post comments)

Embedded world view of kernels...

Posted Oct 9, 2008 1:43 UTC (Thu) by smoogen (subscriber, #97) [Link]

My short time in embedded land seemed to show that it attracts a very high amount of the 'do it ourselves' personality. We were working on a software middleware that was going to 1 embedded company and ended up with multiple ports because they had 8 different RTOS for the hardware. I thought each RTOS would be for different hardware, but it was all for the same piece.. and many for the same problem sets. It seemed they had various lead developers who disagreed over various items and just decided to fork and do their own.

I thought this was very strange, but the MIT guy in our company assured us that it was quite common as the type of person who needed to have the focus to deal with limited power, cpu, etc was rarely the sort to let someone else tell them how to deal with an OS.

Forking the kernel pointless, because that doesn't gain cooperation

Posted Oct 9, 2008 4:06 UTC (Thu) by dwheeler (guest, #1216) [Link]

"... a member of the audience asked whether the embedded community should stop using a kernel developed by enterprise system vendors and, instead, make its own version of the kernel suited to its needs."

The questioner failed to understand the problem, and thus proposed a silly solution. If the problem was that the enterprise system vendors inhibited real-time development, then such a fork might be a good idea. But it's nonsense.

The problem is, fundamentally, that many of the real-time system developers do not work together to make a better kernel. This would not be improved by having a fork; any such fork would just make it more obvious without solving it. As noted in the article, "there are still people in the embedded industry who do not see the value of working with the larger Linux development community."

My hat's off to the many good guys at MontaVista, TI, and others, who are working to try to change things for the better. I think things are already much better than they used to be, and will continue to get better over time, but it's really hard to spread such a massive cultural change as this is.

On the other hand, I recently got a new HDTV from Sharp. It prominently displayed the GPL, specifically noting both the Linux kernel and busybox (and how to get source code). Routers with Linux are _everywhere_. I think that as embedded developers become more comfortable and confident with this stuff, they'll learn to adjust... either directly, or because they're getting their stuffing knocked out of them by competitors who DO get it. The organizations whose maintenance costs are borne by the entire development community will slowly pinch out the organizations who try to carry that cost all by themselves.

Forking the kernel pointless, because that doesn't gain cooperation

Posted Oct 9, 2008 8:58 UTC (Thu) by epa (subscriber, #39769) [Link]

Does your new Sharp HDTV include a means to upload a new kernel image or a
new version of busybox?

Forking the kernel pointless, because that doesn't gain cooperation

Posted Oct 9, 2008 18:45 UTC (Thu) by JoeBuck (subscriber, #2330) [Link]

The GPL doesn't require that. But I'd be surprised if there weren't a way to upgrade the firmware (and thus void your warranty).

Some views from Vision

Posted Oct 9, 2008 16:43 UTC (Thu) by Frej (subscriber, #4165) [Link]

Flamebait...., read this as a question!

I guess a different need of embedded people is that they ship to actual users. The 'Enterprise' have people (IT department) with special knowledge handling all issues related to IT, and that includes a constant changing kernel with arbitrary deprecation and release dates.

It's not just features that's aligned with enterprise needs, the linux development process is as well.

Some views from Vision

Posted Oct 9, 2008 19:39 UTC (Thu) by vonbrand (subscriber, #4458) [Link]

The 'Enterprise' have people (IT department) with special knowledge handling all issues related to IT, and that includes a constant changing kernel with arbitrary deprecation and release dates.

"Enterprise IT" people have enough on their plates as is. They definitely don't wellcome any "constant changing" kernel (or whatever). Just look at the Enterprise distributions (like Red Hat), they guarantee several years without version changes whatsoever

Some views from Vision

Posted Oct 9, 2008 23:04 UTC (Thu) by nix (subscriber, #2304) [Link]

Indeed. This can be very annoying.

Just in the last week I've had to compile from source the following to
work around nonpackaging of obvious crash fixes in one still-under-support
enterprise distro

- zsh, to work around a core dump in zsh 4.0.7, fixed in 4.0.8: 4.0.9 was
released in *2003*, but was never packaged for that distro
- coreutils, to work around a crash in du, fixed in coreutils 5.3.0 (!),
released in 2005; this distro has 5.2.1

This is getting annoying. I only want a bloody shell script (if a complex
one, complex enough that it uses zsh for cross-platform consistency
because no other shell works the same everywhere and is capable enough) to
work when called from a daemon without coredumping. It works everywhere
else, including every environment I actually do development on.

I am agitating to upgrade to non-stone-age software. Even Solaris is less
painful than this.

Some views from Vision

Posted Oct 10, 2008 4:40 UTC (Fri) by phip (subscriber, #1715) [Link]

Another way to look at this is that when an embedded system is done properly, the company that designs/sells/leases the hardware serves as the "IT department", and the actual user doesn't need to deal with IT issues at all (unless they want to, of course).

Cheers,
Phil

Copyright © 2008, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds