ELC: The embedded Linux nightmare

Posted Apr 17, 2007 23:57 UTC (Tue) by lutchann (subscriber, #8872) [Link]

In the process, a 7,000-line driver became a much better 1,300-line driver. Using the COCOMO model, Thomas estimates that about $180,000 was wasted in the creation of this vendor driver.

Not having attended the talk I don't have any background on this example, but a lot of times vendor drivers like this are the result of gluing the hardware component's DV test code to a kernel driver skeleton. Naturally, this doesn't produce the best driver in the world, but it does result in a shorter time to market and requires less initial effort on the part of the hardware vendor than writing a proper driver. It's not very fair to assume that providing bloated, ugly drivers means the vendor doesn't care about working with the community.

Posted Apr 18, 2007 2:05 UTC (Wed) by eli (guest, #11265) [Link]

Something not really addressed is the fact that it does take effort, and a good bit of time, to get large changes merged upstream. I've done it, and it's rewarding, but you need to have support for it in the organization or your developers won't bother.

Posted Apr 18, 2007 4:41 UTC (Wed) by sepreece (subscriber, #19270) [Link]

Gleixner's talk had a distinct implication that the vendors were at fault and the community had a right to expect more of them.

The vendors, of course, see things differently. Unlike computers, whose users have been taught to expect to have to replace the system software regularly, devices are generally expected to remain the same from the day they are shipped to the day they are discarded. Most users would not like to see that change.

Vendors, therefore, tend to like to get the software very stable before they ship the product, and to use the same software base to build a given product (and updates, refreshes, and successors of that product) for a long time.

So, vendors tend to be working on old releases, even though they want current features. Thus it's not surprising that he found a vendor release that was based on 2.6.10 but was actually closer to a later release, based on patch size.

The point is that this is not evil on the vendor's part, it's just a difference in needs and expectations. In many cases it's not that the vendors don't want to "give back", it's that they can't. When they find a bug or make an enhancement, their base is so far from the community's that the community has no interest in what they have to say.

The vendors aren't dumb. They know that community gives leverage and that having more eyes working on something will make it better, faster, and many of them would love to have their work mainstreamed so they don't have to maintain it. But they are often working on things that literally nobody else in the world cares about (like devices for custom hardware) or on old releases or just have different priorities than the community developers.

Posted Apr 18, 2007 6:16 UTC (Wed) by zipdisk (guest, #8589) [Link]

I think the point Thomas is trying to make is basically that vendors are
not embracing the full concept of using Open Source. I've been in both
sides of the fence (working for a vendor and being part of the community)
and I have to admit that in either case you tend to see to the other side
as the guilty part.
But in the end, I think the real problem is not who is guilty of what but
how to fix the "bad" relation. Is true that vendors many times do not
release source code back to the community (not only on Linux but in other
Open Source Operating System), and is true that sometimes the community is
not interested in what the vendor is doing. Is true that sometimes a
vendor finds problems and try to get help from the community and the
community does not help at all and is also true that sometimes the
community brings a lot of help but the vendor does not give a thing in
return.
This topic is more complex than what it seems, and for people outside the
embedded world is difficult to understand why this happens.

Posted Apr 18, 2007 8:28 UTC (Wed) by nim-nim (subscriber, #34454) [Link]

It's all a matter of control.

Private fork means you can do whatever short-term changes your PHB asks of you. It means you don't have to answer to community reviewers in addition to your boss (which very often you know how to distract). It means you're paid to reinvent the wheel instead of doing more difficult stuff. It means you can slip in pilfered code of dubious origin without detection. For those reasons many developpers are activelly hostile to working with mainline. They can easily convince their managers with bad arguments if management is not educated about the Linux ecosystem. Especially since management sometimes wants the "big picture" and ignores the little details that made Linux successful.

So far evangelization has focused on developpers. This is IMHO a big mistake. Working through mainline is a big win mid and long-term but this kind of prospective view is asked of management, not developpers.

It's all too easy to frighten management with the short-term drawbacks of working through mainline if there's no authoritative voice telling them it's worth the pain. Even if you manage to enlighten a few developpers in an organisation their peers can easily contain them.

What's needed is more successful big names like Intel, Dell, Red Hat... making presentations on the strategic benefits of working through mainline. That would give management of smaller companies the backbone necessary to resist short-term FUDing.

Posted Apr 18, 2007 8:40 UTC (Wed) by Hanno (guest, #41730) [Link]

Are the slides available?

Posted Apr 18, 2007 9:29 UTC (Wed) by simlo (subscriber, #10866) [Link]

Can we get the this article for "free"? I would like to send a link to some of my co-workers :-)

Esben

Posted Apr 18, 2007 10:43 UTC (Wed) by alex (subscriber, #1355) [Link]

It's not always that way. When I last worked on an embedded project I did make at active effort for submit patches into the mainline. I understood very well the advantages of being able to closely track the mainline.

Of course back in the those days I was trying to do it with an linuxsh CVS overlay ontop of the mainline kernel while trying to keep stuff not worth submitting (like BSP's for the board) from more general changes (like core processor changes). I suspect nowadays using GIT would have made my life a lot easier in juggling these things around.

Having said that this was driven by me as an engineer not by management. It's a shame that since I've left the company (when it was taken over) I've not seen any further activity on the developer lists. Since then the product has made it to market but they don't make the (GPL parts) of the source code available. Having said that I doubt any of their customers actually care about that.

Posted Apr 26, 2007 12:29 UTC (Thu) by endecotp (guest, #36428) [Link]

> 7,000-line driver became a much better 1,300-line driver

Let's consider why this happens. These people are not completely stupid. Yet rather than produce a 1300-line driver, they produced a 7000-line driver. It can only be that it wasn't obvious to them how to achieve what they wanted in 1300 lines: they have written 5700 lines of "unnecessary" stuff, probably because it was easier overall for them to do that than to understand how to do it "properly". I can think of two fundamental issues:

1. Kernel documentation needs to be better. (Or, the kernel design needs to be easier to understand so that it "works" with less documentation.)

2. Mailing lists need to be more attractive venues for discussion of this sort of thing. I'm not sure why they are currently not attractive to these developers, but it's clear that they aren't. If the fundamental issue is corporate privacy then there probably isn't much that can be done, but there may be other reasons. This could be a good subject for a research project....