The embedded Linux nightmare - an epilogue [LWN.net]

The embedded Linux nightmare - an epilogue

Posted May 2, 2007 23:14 UTC (Wed) by roelofs (guest, #2599)
Parent article: The embedded Linux nightmare - an epilogue

It is undisputed that kernel versions need to be frozen for product releases, but it can be observed that those freezes are typically done very early in the development cycle and are kept across multiple versions of the product or product family. These freezes, which are the vain attempt to keep the existing procedures alive ...

That's a tad simplistic--sometimes the reason is as innocuous as "minimize your variables." For example, when we encounted bugs on a project on which I worked several years ago, it was rarely obvious whether it was an application bug, a toolchain bug (GCC/binutils/glibc), a kernel bug, or a hardware bug (CPU itself or support chips/boards, each of which may be literally unique due to manufacturing flaws). Indeed, over the course of the project we found examples of each flavor--and at least one of the kernel bugs wasn't in driver code; it was in regular, mainline code (albeit infrequently used mainline code).

That said, we did upgrade kernels at least two or three times (I recall at least 2.4.7, 2.4.14, and 2.4.18), and for all I know, there were further updates before the product(s) shipped. But it's an overstatement to solely blame a desire to "keep the existing procedures alive"--unless, of course, one of the procedures in question is the scientific method.

Bigger players in the embedded market apparently have budgets large enough to ignore the benefits of working with the community and just concentrate on their private forks.

I suspect it's less an issue of enjoying large budgets than of suffering large bureaucracies. ;-) Barring a champion with enough authority to make his or her own decisions on behalf of the company (rare!), it can be extremely difficult even to find out whose department has such authority, much less to make contact and convince her/him/them. Even my implication that there's a single such department is questionable; it could easily involve multiple departments/divisions/management chains, not to mention lawyers and perhaps even one or more third parties. Ten iterations of review/revise before acceptance into the mainline kernel is nothing compared to that level of brutality. :-(

Greg

(Log in to post comments)

The embedded Linux nightmare - an epilogue

Posted May 3, 2007 1:14 UTC (Thu) by bene42 (subscriber, #32623) [Link]

> Ten iterations of review/revise before acceptance into the mainline kernel is nothing compared to that level of brutality. :-(

But we have to show the "kernel" way is the cheapest way. And every QA, exept the SM ones, is on our side. For a diligent QA is it easier to accept a new stable kernel version, then a ton of backports. They do the same tests on each kernel, so the difference is one question: What backing?

The embedded Linux nightmare - an epilogue

Posted May 3, 2007 4:19 UTC (Thu) by roelofs (guest, #2599) [Link]

But we have to show the "kernel" way is the cheapest way.

Oh, I get that, and I'm grateful for articles like this one and its predecessor--it should be an excellent tool to help embedded engineers (who are often favorably inclined toward FLOSS in the first place) convince their management chains to work in a more community-compatible way. I was just pointing out that one of Thomas's arguments was maybe not quite as strong as it might appear at first glance; sometimes there are good reasons for sticking with an older kernel for an extended period.

Also note that not all architectures get as much developer attention as i386 and x86_64, nor are they all as unified.

Greg

The embedded Linux nightmare - an epilogue

Posted May 3, 2007 6:44 UTC (Thu) by drag (subscriber, #31333) [Link]

How much of what you described above can be blamed on just stumbling along new bugs by introducing new software versions versus maybe just not having the ability to quickly diagnose bugs in the first place?

Say if this project described on the following article is successfull?:
http://lwn.net/Articles/232769/

So if that makes it much easier to dectect and isolate bugs (say make it more obvious were in the hardware->application toolchain the bug is occuring) would it have a positive affect on embedded developers to be able to track closer to mainline kernel?

Could it it be that lack of good debugging tools for Linux is going to hold people back from using newer Linux versions?

Or is it that what you have now is good enough for embedded developers and such things probably aren't going to make anybody's job any easier?

The embedded Linux nightmare - an epilogue

Posted May 4, 2007 3:50 UTC (Fri) by roelofs (guest, #2599) [Link]

How much of what you described above can be blamed on just stumbling along [upon?] new bugs by introducing new software versions versus maybe just not having the ability to quickly diagnose bugs in the first place?

Well, it was certainly true that, given the size of the problem space, we were more than a bit understaffed. And the Chronicle debugger (thanks for the link, btw--I hadn't gotten to this week's Development page yet), had it existed back then, probably would have helped, at least if we could have run it from the host system. But insofar as it runs on top of Valgrind, it appears to be limited to apps and libraries, and the really hard, time-consuming bugs were the ones in the kernel and hardware.

Could it it be that lack of good debugging tools for Linux is going to hold people back from using newer Linux versions? Or is it that what you have now is good enough for embedded developers and such things probably aren't going to make anybody's job any easier?

I'm no longer doing embedded work (quite the opposite, in fact!), but I don't know anyone who would claim the existing tools are "good enough." That is, they're sufficient to get most things done, but it's clear they could be much, much better. In fact, most reasonably unbiased developers with whom I've spoken seem to agree that development tools are one of the principal areas where Microsoft still holds a significant lead over the FLOSS community, and I fully believe them. After all, MS has been working on their toolchain and using it to woo developers (their own approach to World Domination) for a quarter-century, and even when they were spread the thinnest, they supported only three or four CPU architectures--and that for only a few years. GCC and gdb are truly marvels of portability, and it's really nice not to have to learn a new set of commands on every platform, but that very portability has been a huge drag on the pace of development, both in optimization and in debugging (and, no doubt, other areas).

That said, I continue to use GCC and gdb every day, both at work and at home. And as much as I love to eke the last little bit of performance out of my code, I'm far more excited about the possibility of major strides on the debugging end of things--especially for massively multithreaded applications. The sooner, the better...

Greg