The embedded Linux nightmare - an epilogue [LWN.net]

The embedded Linux nightmare - an epilogue

Posted May 1, 2007 21:23 UTC (Tue) by raven667 (guest, #5198)
Parent article: The embedded Linux nightmare - an epilogue

I can only can discourage companies to even think about such things. The time for backports and the maintenance of the resulting unique kernel fork is definitely better spent on adjusting the internal development and quality assurance procedures to the way how the Linux kernel development process works. Otherwise it would be just another great example for useless waste of resources.

So how does this relate to long-term supported vendor releases such as RHEL, SLES, Ubuntu LTS or Debian? Each of these vendors maintains one forked kernel version into which they backport fixes for the life of their product. It would seem that you are arguing that this is also the wrong way to approach the problem of maintenance unless you see a fundamental difference between the two cases.

to post comments

The embedded Linux nightmare - an epilogue

Posted May 1, 2007 22:14 UTC (Tue) by AdHoc (guest, #1115) [Link] (13 responses)

They backport fixes, which is much different from backporting features. A fix is generally localized, features may be touch many parts of the kernel. I'm guessing they occasionally backport new drivers if the driver framework hasn't changed much.

The embedded Linux nightmare - an epilogue

Posted May 2, 2007 0:10 UTC (Wed) by drag (guest, #31333) [Link] (9 responses)

Plus the kernel differences don't seem as huge.

Currently Debian Stable and Ubuntu are using 2.6.18 kernel variations. Redhat EL 5 uses 2.6.18 variation. All of them are released fairly recently.

Old Debian Stable shipped 2.6.8 and Redhat EL 4 used 2.6.9. Those systems are the longest supported distributions out of all mainline Linux versions and now they are basicly obsolete, except as legacy installations. Nobody would want to install either of those systems for new servers or workstations.

Adding new hardware support from backports would be exponentially easier from 2.6.21 to 2.6.18 then trying to go from 2.6.21 to 2.6.8

So in the author's example he was asked about backporting drivers to 2.6.10... probably that means that the device is not even into beta mode yet if they are still working out the details of hardware support. So that is a very odd kernel for anything still in relatively early stages of development.

That kernel is very old now. It will be ancient by the time it reaches market. So it seems very weird that you'd choose that version to work with.

I guess that a good approach would be that generally people will want to keep up with the latest stable releases for as long as they can, and 'freeze' the version as late into the development cycle as comfortably possible.

but ...

Posted May 2, 2007 1:45 UTC (Wed) by JoeBuck (subscriber, #2330) [Link] (2 responses)

Red Hat is still supporting RHEL 3 for their customers.

but ...

Posted May 2, 2007 20:01 UTC (Wed) by tglx (subscriber, #31301) [Link]

right, because it's a product and they have support contracts. But it is in maintainence mode. They won't add the new wireless stack or whatever big feature to it anymore.

but ...

Posted May 2, 2007 21:18 UTC (Wed) by riel (subscriber, #3142) [Link]

RHEL releases only get select new hardware and feature support for the first 2 to 3 years after release. The rest of the 7 year support cycle the product will only get fixes for security problems and other really severe bugs. RHEL 2.1 is in that latter stage of the support cycle and RHEL 3 will be there soon.

We also have a policy of making sure that bugs our customers find in RHEL also get fixed in the upstream kernel. Preferably, they get fixed in the upstream kernel before the fix is put in RHEL.

Not only do we get the benefit of upstream review of the code change, but the bug is also automatically fixed when we come out with our next RHEL release. We only need to fix each bug once, and the upstream community gets the benefits of our work.

When dealing with a long product cycle and the faster upstream Linux development cycle there are ways to ensure that both your product and the upstream Linux community benefit.

The embedded Linux nightmare - an epilogue

Posted May 2, 2007 3:48 UTC (Wed) by raven667 (guest, #5198) [Link] (2 responses)

Old Debian Stable shipped 2.6.8 and Redhat EL 4 used 2.6.9. Those systems are the longest supported distributions out of all mainline Linux versions and now they are basicly obsolete, except as legacy installations. Nobody would want to install either of those systems for new servers or workstations.

I disagree with this statement. I still install RHEL4 in my business and plan to do so for some time, heck I still install the occasional RHEL3 host. I'm not going to rebuild systems that currently work just to get the latest and greatest without some clear justification. It costs time to maintain another OS variant, re-checking our services and potentially porting them to the new release. Of course they are all similar (RHEL3-RHEL5) but each version has differences (or else it wouldn't be a new version) that I need to understand and account for. The long term support for RHEL, SLES and Ubuntu LTS exists for a reason, users which don't want a high rate of churn for their base OS.

The embedded Linux nightmare - an epilogue

Posted May 2, 2007 5:17 UTC (Wed) by dlang (guest, #313) [Link] (1 responses)

however, you won't be trying to install that old version on brand new hardware, you will be installing it on the hardware that it supports.

as such you just need bugfixes, you don't need new drivers and other major improvements from the new kernel.

so you are just fine. the maintinance nightmare is when you try to backport larger chunks of things, and they drag in more stuff that they depend on.

even bugfixes aren't going to be complete, many bugs get fixed by replacing the code that is buggy (frequently without the person writing the new code recognising all the implications of the bug, sometimes without them noticing that there is a bug) with new code, useually to add new features, or as part of a cleanup effort.

bugs that are fixed like this are extremely hard to backport

The embedded Linux nightmare - an epilogue

Posted May 3, 2007 11:59 UTC (Thu) by raven667 (guest, #5198) [Link]

<blockquote>however, you won't be trying to install that old version on brand new hardware,
you will be installing it on the hardware that it supports.

as such you just need bugfixes, you don't need new drivers and other major improvements
from the new kernel.
</blockquote>
I hate to say it again but this is also not true. RHEL3 supports modern hardware, the have
backported newer versions of drivers and other compatability fixes. RHEL3 is supported on the
latest Dell PowerEdge 9th generation systems for example and even comes with the Dell hardware
management kernel drivers that were integrated with upstream.

The embedded Linux nightmare - an epilogue

Posted May 2, 2007 4:47 UTC (Wed) by AJWM (guest, #15888) [Link] (1 responses)

> Redhat EL 4 used 2.6.9. [...] Nobody would want to install either of those systems for new servers or workstations.

Heh, I wish I could agree with you, but within the next couple of days I'll be doing a fresh Redhat 4 install. The (4x 3GHz Xeon) server is being repurposed, currently it's still running RHEL 2.1. Most of the servers in this particular environment are running RH 3, with just a handful at RH 4. (Mind, they do get updates). Once you've got an enterprise system stable (eg, a four-node Oracle cluster with 8 CPUs/node (16 virtual with hyperthreading) and a SAN backend) you don't want to mess with it very often. (When that's upgraded it'll also be to new boxes, 2 each 32-way IA-64s, and we can kiss hugemem goodbye - but it'll be to RH4, not RH5).

The embedded Linux nightmare - an epilogue

Posted May 2, 2007 11:51 UTC (Wed) by drag (guest, #31333) [Link]

I said _want_.

You don't want to use it, do you? I'd expect you'd rather just install the newest of what was aviable and leave it at that.

You use it because you _need_ to, right?

Also I suppose you'd like all new hardware, too.

:-)

The embedded Linux nightmare - an epilogue

Posted May 3, 2007 5:28 UTC (Thu) by msmeissn (subscriber, #13641) [Link]

SUSE is still supporting 2.4.21 kernels (SLES 8)...

And it really only sees security updates at this time... Very
small self contained fixes.

Ciao, Marcus

The embedded Linux nightmare - an epilogue

Posted May 2, 2007 3:54 UTC (Wed) by sobdk (guest, #38278) [Link] (1 responses)

RHEL 4 with a 2.6.9 kernel has the mutex patches backported from 2.6.16. I would hardly call this a fix, nor is it a new driver. Perhaps they do this to actually closer track the mainline while pretending to keep the kernel at 2.6.9, but they certainly do not just backport fixes.

The embedded Linux nightmare - an epilogue

Posted May 3, 2007 23:05 UTC (Thu) by ronaldcole (guest, #1462) [Link]

I believe that would be because RHEL4 still has one year before it enters maintenance mode, and would therefore only get bug fixes.

The embedded Linux nightmare - an epilogue

Posted May 2, 2007 7:14 UTC (Wed) by bos (guest, #6154) [Link]

The big vendors do *not* limit themselves to backporting fixes into their enterprise kernels. Each of the five extant RHEL 4 kernels (original and updates 1 through 4) is quite different from the others. For example, the core device model code changed TWICE during updates between RHEL 4 kernels.

Backporting drivers to older kernels absolutely sucks as a way to spend your time because of changes like this, and worse, it feels like a waste of effort while you do it. Dispiriting stuff.

The embedded Linux nightmare - an epilogue

Posted May 1, 2007 22:26 UTC (Tue) by dmarti (subscriber, #11625) [Link]

It seems like there's a difference between backporting fixes (the kind of changes that happen between 2.6.x.0 and 2.6.x.y in mainline) and backporting whole new features such as the new wireless code.

The embedded Linux nightmare - an epilogue

Posted May 2, 2007 15:34 UTC (Wed) by vmole (guest, #111) [Link]

One other thing that makes RedHat different: they have actual full-time kernel developers working for them. That makes it *far* more likely that they'll be able to successfully backport a feature/fix than J. Random Embedded. Not because Ms. Embedded is a bad programmer, but because she doesn't spend all her time working with and understanding the kernel.