LWN.net Logo

Old kernels and new compilers

Under the long-lasting maintainership of Marcelo Tosatti, the 2.4 kernel went into a deep maintenance mode, with only important fixes being considered for merging. For some people, perhaps, it was a little too deep - Marcelo clearly had other tasks besides 2.4 maintenance keeping him busy. Even so, few expected major changes when Willy Tarreau took over 2.4 maintenance after the 2.4.33 release. Why mess with 2.4 at this point?

So Willy's 2.4.34-pre1 announcement raised a few eyebrows. The prepatch itself contains a relatively small number of patches of the type one would expect. But the announcement itself notes that Willy is considering merging a set of patches to allow 2.4 kernels to be built with current gcc 4.x compilers. This is not a trivial set of changes; gcc 4.x is sufficiently different that a fairly wide-ranging set of fixes is required. The gcc 4.x transition for 2.6 was not an overnight affair.

A clear question comes immediately to mind: why would somebody who is not interested in running a current kernel be bothering with contemporary compilers? One answer is to be found in the announcement itself: there are administrators who deploy 2.4 kernels on ultra-stable systems, but who build those kernels on their desktops. It is getting increasingly hard to find a current distribution with a compiler old enough to build 2.4 kernels, so these administrators are finding themselves in a bit of a bind. A 2.4 kernel which could be compiled with a current gcc would allow current systems to be used to build kernels for deployment on stable, production systems, many of which may not have their own compilers installed at all.

Solar Designer has also noted that Openwall GNU/*/Linux is planning to upgrade to gcc 4.x and would really rather not have to change to the 2.6 kernel at the same time.

For an interesting read, see Willy's description of the user base, as he sees it, for the 2.4 kernel. In his view, the major users are those setting up very high-reliability sites. These people prefer 2.4 kernels for this job:

Simply because we already know from collective experience that these versions can achieve very long uptimes (while we don't know this yet for a fresh new version which got 5700 patches in the last 3 months), and because with the addition of very few patches, you can make a bet on security: as long as newly discovered vulnerabilities don't affect you or are covered by your additional patches, you win. If you need to update and induce excessive downtime, you lose and pay penalties.

The idea is to keep these people happy - by enabling the use of current compilers, among other things - until a 2.6 kernel comes along which is able to provide the same sort of stability guarantees. The 2.6 development model makes that sort of guarantee harder, however, because older 2.6.x kernels go out of general maintenance relatively quickly (though distributors can and do maintain them for longer). It is hard to find a 2.6 kernel with a multi-year track record of reliability, security, and ongoing fixes.

Willy's hope is that the current 2.6.16 kernel, which Adrian Bunk has stepped forward to maintain for the long term, will help in this regard. Once 2.6.16 has received a year or two of fixes (and nothing else), it might reach a point where high-reliability people might trust it in deployed systems. Time will tell if this kernel is able to reach that point.

As an aside, it's worth mentioning that a small number of developers (well, OK, one developer) have expressed some discontent about the 2.6.16 long-term process. This developer has said that it would have been better to elect an extra-stable tree maintainer through some sort of popular vote, and, perhaps, to move on to a 2.7 development series as well. This complaint ignores the fact that volunteers to maintain 2.6 kernels over the long term have been in relatively short supply; in fact, Adrian would appear to be about the only one. It does not appear that Adrian's appointment as the long-term 2.6.16 maintainer has deprived anybody else of their lifetime dreams. So maintainer elections - other than those of the "vote with your feet" variety - seem unlikely to happen in the near future.


(Log in to post comments)

Old kernels and new compilers

Posted Aug 24, 2006 6:22 UTC (Thu) by eru (subscriber, #2753) [Link]

It is getting increasingly hard to find a current distribution with a compiler old enough to build 2.4 kernels, so these administrators are finding themselves in a bit of a bind. A 2.4 kernel which could be compiled with a current gcc would allow current systems to be used to build kernels for deployment on stable, production systems, many of which may not have their own compilers installed at all.

A bit odd argument: It is quite easy to install an older GCC sufficient for building a kernel with it, especially since you only have to build the C part, not C++ etc. Building the C support of, say, GCC 2.95.3 takes less time than konfiguring and building a 2.4 kernel...

Old kernels and new compilers

Posted Aug 24, 2006 16:24 UTC (Thu) by landley (guest, #6789) [Link]

The BusyBox website has a Red Hat 9 image we test-build with under QEMU.
(It also has a patch to build qemu under gcc 4.x if you want to do it
from source.)

Once you've got qemu installed, it's as simple as:

wget http://busybox.net/downloads/qemu/rh-9-shrike.img.bz2
bunzip2 rh-9-shrike.img.bz2
qemu rh-9-shrike.img

And voila, it pops up a new window with RH9 running in it.

Login as either user "busybox" or as root, password "busybox" in both
cases. If you prefer a gui to text mode, I think you can run "startx".
The emulated system should have an emulated (masqueraded) network
connection, so you can scp files in and out. Grab your source, build it,
tar it up, and scp the result back out.

Rob

Old kernels and new compilers

Posted Aug 24, 2006 9:44 UTC (Thu) by nix (subscriber, #2304) [Link]

Very long uptimes, yep: I just had a forced powerdown of one machine due to a power failure (running, I think, 2.4.29) with an uptime of (497 + 140) = 637 days. It'd been running without trouble so long that when a two-year-old bug surfaced and knocked it off the net I had trouble remembering the root password to fix it...

Linux 2.4: for systems that Just Work Dammit.

Old kernels and new compilers

Posted Aug 24, 2006 15:33 UTC (Thu) by jbailey (subscriber, #16890) [Link]

$ uname -a
Linux ia64 2.6.8-3-mckinley-smp #1 SMP Mon Mar 14 18:23:12 MST 2005 ia64 GNU/Linux
$ uptime
11:23:49 up 459 days, 21:52, 1 user, load average: 0.00, 0.00, 0.00

While it's admittedly not being *heavily* used, I do use it for compiler and glibc builds and testing.

I don't know where people have complaints about 2.6 stability. I've had almost no problems, including on heavily used public machines. The public machines don't have this sort of uptime only because of the once-every-week-security-upgrade-reboot.

Tks,
Jeff Bailey

Old kernels and new compilers

Posted Aug 24, 2006 22:29 UTC (Thu) by cventers (guest, #31465) [Link]

Agreed. It's not that I haven't had 2.6 bugs, but I've never seen 2.6
just decide to panic out of the blue. If something's wrong, it's going to
bite almost immediately.

In fact, you can abuse the piss out of 2.6 (repeatedly insert buggy
kernel modules you're working on and then rmmod -f them, oopsing the
kernel over and over again) and it will still pretend like there's
nothing wrong afterwards. I've seen a 2.6 system that oopsed due to some
ATI proprietary driver nonsense that continued to work for weeks
afterwards (albeit not with the console).

Old kernels and new compilers

Posted Aug 24, 2006 12:59 UTC (Thu) by ewan (subscriber, #5533) [Link]

No-one's servers run source code - if the ultra-reliability folks are only
going to trust kernels with a good long track record, why would they trust
a shiny new, and rather internally different, GCC to build them correctly?

Old kernels and new compilers

Posted Aug 25, 2006 6:26 UTC (Fri) by bryanr (guest, #25324) [Link]

gcc has an excellent automated testsuite.

linux does not have an official testsuite, and the existing
test projects are only moderately successful at covering the
set of strange, unexpected, and intentionally malicious
scenarios that may be faced by an OS kernel

Old kernels and new compilers

Posted Sep 2, 2006 22:34 UTC (Sat) by renox (subscriber, #23785) [Link]

> gcc has an excellent automated testsuite.

For regular code, this is probably ok, but for kernel code?

When there is a major revision of gcc, if memory serves, the Linux kernel breaks in various ways..

Embedded systems

Posted Aug 24, 2006 13:50 UTC (Thu) by abatters (✭ supporter ✭, #6932) [Link]

I maintain the firmware for some embedded devices that still run 2.4 kernels. I will eventually move them to the 2.6 kernels, but that is a pretty big job, and I have lots of other things to do. I do development under Ubuntu 5.10, which has packages for both gcc 4.0 and 3.4 which can be installed simultaneously. I find that it is only slightly inconvenient to use an older compiler for the kernel.

Volatility vs Stability?

Posted Aug 24, 2006 18:59 UTC (Thu) by cdmiller (subscriber, #2813) [Link]

Perhaps stability is not the correct word, how about Volatility? We had an issue with several Dell servers and the Adaptec Raid controller driver present in the 2.6 series of kernels. Solution? We ran 2.4 kernels for 2.5 years until Kernel 2.6.15 appeared with a fixed (completely rewritten) driver. There were a few different versions of the Adaptec Raid driver in 2.6, none of which worked until the rewrite in 2.6.15.

We have live CD firewall/routers running on 2.4, (Devil linux), compiled with grsec etc.. They achieve very long uptimes and have very little volatility. No forced reboots due to the latest 2.6 kernel security patch.

For the first half or kernel 2.4, I ran many systems on kernel 2.2, guess which one was more volatile, more forced reboots, etc.?

SSDD...

Another reason for 2.4: customized systems

Posted Aug 26, 2006 21:37 UTC (Sat) by giraffedata (subscriber, #1954) [Link]

I don't contest Willy's description of who most his users are, but I know another class of 2.4 users, of which I am one: People who have built customized systems out of it. It would be quite costly for me to adapt my kernel modifications, drivers, user space programs, and maintenance procedures to any 2.6 kernel. Since the kernel still does fine what it was tasked to do 5 years ago, it wouldn't be worth it.

On the other hand, lots of other parts of these systems have kept up with the times and I can definitely see value in having modern Gcc for them while not juggling two versions of Gcc.

For me, since stability isn't the issue, any breakage caused by a Gcc 4 upgrade, assuming it gets fixed eventually, wouldn't bother me.

Old kernels and new compilers

Posted Sep 9, 2006 10:29 UTC (Sat) by Klavs (subscriber, #10563) [Link]

I have a hard time believing the only reason to make 2.4 compat. with gcc-4.x is for those high-uptime users who use 2.4, and yet still can't figure out how to compile an older gcc, for their 2.4 compile needs.
Having an old (or just a chroot'ed) system around for this is no problem (and people with old systems using 2.4, often don't follow the latest distro's anyhow) or just using something like Gentoo, which allows you to have as many compilers as you wish, and just choose which one to use, before issuing make bzImage (or whatever ;)

Copyright © 2006, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds