RHEL 5.4 released

Posted Sep 2, 2009 16:09 UTC (Wed) by qg6te2 (guest, #52587)
Parent article: RHEL 5.4 released

Looks like the version number in RH's kernels bears little resemblance to what the original "2.6.18" actually means. Why does RH persist on backporting so much stuff instead of just using a newer kernel? I can see a perceived need to keep the ABI stable, but so much backporting can and does increase the risk of internal inconsistencies within such a mongrel kernel.

to post comments

RHEL 5.4 released

Posted Sep 2, 2009 16:57 UTC (Wed) by smoogen (subscriber, #97) [Link] (9 responses)

Simple answer... because people who pay them request it heavily.

RHEL 5.4 released

Posted Sep 2, 2009 17:37 UTC (Wed) by jgg (subscriber, #55211) [Link] (8 responses)

The only customers that request this are ones that are trapped with some kind of important proprietary driver. Everyone else would rather have a supported newer kernel option at this point. Its been almost 3 years since 2.6.18 was released now!

2.6.30 runs circles around the RH kernel on certain benchmarks, the RH kernel is no longer a good representation of Linux. Whats worse, API improvements in the mainline take forever to get back to the RH user base, to the point that app developers can't make use of new performance features simply because RH ships that ancient kernel.

I wish RH would give the choice to use either their back ported mess, or something newer. Officially supported, with full user space support. I suppose if they did that then the users would demand of the vendors that it work with their drivers too and the vendors would hate that :)

RHEL 5.4 released

Posted Sep 2, 2009 18:14 UTC (Wed) by jengelh (subscriber, #33263) [Link] (3 responses)

Time for ripping 2.6.18 support out of all programs that do happen to still have it. That should give them the hint!

RHEL 5.4 released

Posted Sep 2, 2009 18:35 UTC (Wed) by jhubbard (guest, #5513) [Link] (2 responses)

You don't understand. RHEL is made up of versions of applications that were around when RHEL 5.0 was released. New versions of software are rarely introduced and when they are it's older versions. For example, boost 1.33.1 was released with RHEL 5.3. At that time, I believe that boost 1.37 was the newest.

Backporting drivers and other new features of the kernel takes up an enormous amount of time. It's been covered many times before in articles here. There's a reason that they do it. It because customers want it. I work on systems where they expect leave them deployed for 5 years on more with a minimal amount of change. They only want critical bug fixes and security patches. They have neither the budget or time to update the system every year.

A good example of the churn that customers want to miss is all of the effort going into fixing X. There are many good articles on getting X fixed here. If you'll notice, it's been painful. The ABI for the server has changed multiple times and the various modules that need to be used have changed.

Since the opensource drivers don't support the full functionality that customers need, they have to use something that does. Yes I know that we need to support the effort to the open drivers. However, you have to be pragmatic and do what's needed to get the job done.

Examples: I still can't suspend my laptop with the nouveau driver. What about CUDA, StreamSDK, or OpenCL drivers for apps that need the parallelism? Don't forget the general high 3D applications.

RHEL 5.4 released

Posted Sep 3, 2009 1:12 UTC (Thu) by qg6te2 (guest, #52587) [Link] (1 responses)

There's a reason that they do it. It because customers want it. I work on systems where they expect leave them deployed for 5 years on more with a minimal amount of change.

While I'm stating the obvious by pointing out that the above is an instance of "we want stuff that's not going to break", it also implies that the real problem lies with kernel development (or more precisely, the testing part of development).

Regressions are bad, but this problem would be far less prevalent if the kernel developers (which includes many RH folks) were more thorough in their testing before forcing a release every 3 months. I'd wager that most users would be far happier with a 6 month release cycle if they were assured an upgrade wouldn't break their existing setup.

On the other hand, it could be argued that sticking to an ancient upstream release and patching the crap out of it is a way of keeping regressions in check. At the same time, I really do wonder about the stability of such mongrel kernels, given that far less eyes have looked at the final source and there was far less exposure to various environments. Maintaining a huge patchset also destroys the huge leverage RH gets from collaborative development in the upstream kernel.

(and yes, I do submit bug reports).

RHEL 5.4 released

Posted Sep 3, 2009 1:23 UTC (Thu) by dlang (guest, #313) [Link]

with a 3-month release cycle, the amount of new things that are found tapers off towards the end of the three months. delaying the release longer doesn't gain as much as you would like to think.

there is usually a 2-3 week time period where there are not many known regressions (there are very seldom none, and it's not unusual for a reported regression to actually be related to a change that was introduced several releases ago), the reason these regressions don't get fixed is that they are hard or impossible for the kernel developers to duplicate. what they can't duplicate they can't fix. if what caused the problem can be identified, but not fixed easily, it's usually reverted (specificly to avoid regressions), the one exception to this is if the fix is deemed 'important' enough (usually a security issue)

note that arguments along the lines of 'but it fixes far more boxes than it breaks' are _not_ allowed.

RHEL 5.4 released

Posted Sep 2, 2009 23:09 UTC (Wed) by Ed_L. (guest, #24287) [Link]

2.6.30 runs circles around the RH kernel on certain benchmarks, the RH kernel is no longer a good representation of Linux.

Lets qualify that: 2.6.30 runs circles around the RH 2.6.18 kernel on certain benchmarks on the machines it runs on, the RH kernel is no longer a good representation of Linux unless your particular workstation won't boot anything newer.

Three and a half years ago I built me a new Opteron workstation based on the latest lower-power AMD/ULi 1575 chipset in a really nice Abit AT8 motherboard. It serves me very well. But about four months after my purchase, Nvidia bought ULi and put their AMD-compatible south-bridges on the spike. So my AT8, while very reliable and cool-running, is a bit of an orphan. It initially ran Fedora 6, then CentOS 5.0 when that became available, that being the distro used by the shop I then worked. A year ago I moved the machine to my home office and again involved myself with Fedora.

Fedora 10 consistently hosed my md raid.

Fedora 11 (2.6.29), when it boots at all, consistently hoses my md raid. But 2.6.29's radeon driver doesn't appear to do dual-monitor spanning, so there's liitle point. Neither does it host para-virtual Xen, which is also a requirement.

Fedora 12 (Rawhide, 2.6.31), when it boots at all, appears to do dual-monitors with radeon just fine, and supposedly will host paravirtual Xen as well, so I'm optimistically in the process of preparing a bug-report against its boot process. After I can get 2.6.31 to boot reliably from an IDE disk, I can start to worry about whether it can play well with the ULi SATA controller. Then onward to Xen.

Lotsa testing. Meantime I'm writing this, and continuing development, from the machine's main CentOS 5.3 (RH 2.6.18-128.7.1.el5xen) md raid 1 partition, which also hosts a CentOS 4.6 Xen client for development/support compilation, all of which works Just Fine, and I'm very glad to have it.

Kudos to those responsible.

RHEL 5.4 released

Posted Sep 3, 2009 10:34 UTC (Thu) by NAR (subscriber, #1313) [Link]

2.6.30 runs circles around the RH kernel on certain benchmarks, the RH kernel is no longer a good representation of Linux.

Well, I saw sometime ago (probably here) a link to benchmarks comparing subsequent Fedora versions. Some benchmarks had better values on later version, some had worse. Generally there were less benchmarks with worse values, still, it was far from evident that a newer version performs better.

Anyway, in the real world people usually run applications, not benchmarks and they really don't care what is a "good representation of Linux". It doesn't really help if some benchmarks runs 5% faster if e.g. the ClearCase kernel module doesn't load anymore and the developers can't access the source code of the project they are working on... I use SLED at work and I really don't want a new kernel version, because I don't trust neither the kernel developers nor the NVidia developers that my two-monitor setup will work with a newer version. I really don't care about a 10% performance improvement in the kernel, because 99.99% of the time my CPU does nothing.

RHEL 5.4 released

Posted Sep 5, 2009 11:40 UTC (Sat) by dag- (guest, #30207) [Link] (1 responses)

> The only customers that request this are ones that are trapped with some kind of important proprietary driver. Everyone else would rather have a supported newer kernel option at this point. Its been almost 3 years since 2.6.18 was released now!

That is not true. Companies want to be able to install and run the same baseline operating system that behaves the same way and was tested from the start as long as possible.

When we did acceptance tests for RHEL 5.2 (few companies start with RHEL 5.0) we want to be sure that when we install or upgrade RHEL 5.3 and RHEL 5.4 that the system behaves exactly the same way. The risk of a kernel behaving different under a certain situation (load, hardware defect, etc...) is reduced a lot by staying with 2.6.18 rather than move gradually to 2.6.35 (?) over its 7 year support period.

For certain applications (think: firefox or lftp) re-basing them to a newer version is possible and sometimes needed, but doing the same with the kernel is simply too dangerous.

If you take 3rd party software into account (think: Oracle, SAP, CA software), it becomes even a more problematic as it often takes one year to 18 months before the first applications are being supported on a new RHEL releases. Imagine what would happen if Red Hat would change the kernel underneath every 2 years ? Vendors would want to re-certify their applications.

One problem that is becoming apparent in big companies is that Red Hat's move to a 3 year release cycle is making 7 years of support insufficient. With a 4 year hardware life cycle it is becoming hard to install systems that have been tested properly and are supported for the next 4 years.

I recently blogged about this here:

http://dag.wieers.com/blog/is-7-years-of-rhel-support-sti...

RHEL 5.4 released

Posted Sep 5, 2009 18:14 UTC (Sat) by quotemstr (subscriber, #45331) [Link]

Hey. It's good to see you comment here. I've been using your packages for years. Thanks!

RHEL 5.4 released

Posted Sep 2, 2009 18:51 UTC (Wed) by dowdle (subscriber, #659) [Link] (1 responses)

Starting with kernel X and staying with kernel X for the duration of the release and updates has been Red Hat's thing since RHEL 2.1, 3, 4, and now 5. Please see any of Jon Corbet's articles about "Enterprise" distros and their kernels and why they backport patches not just for the kernel but for most everything else too.

Novell, as I understand it, is the same way too... because they also have an "Enterpise" release.

I'm not sure about Ubuntu's LTS release. I'd guess it is the same.

Debian has also been known for just backporting things.

RHEL 5.4 released

Posted Sep 2, 2009 19:22 UTC (Wed) by foom (subscriber, #14868) [Link]

> Debian has also been known for just backporting things.

Generally not major new subsystems, though, as RedHat does.

And in Debian's last release (etch), they in fact did introduce a new kernel in an update. It
was an optional install -- you had to specifically ask for the new kernel if you wanted it,
and the old kernel remained supported.

http://www.debian.org/News/2008/20080726

RHEL 5.4 released

Posted Sep 2, 2009 23:54 UTC (Wed) by smadu2 (guest, #54943) [Link] (3 responses)

Is there any analysis somewhere of how different RH/CentOS .18 kernel from the stable .18 kernel tree ? If so, why did not they make into upstream ?

RHEL 5.4 released

Posted Sep 3, 2009 0:51 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

it's a combination of things that

1. redhat has choosen not to submit upstream

2. features/drivers that were introduced after 2.6.18 that redhat has backported to their kernel (with the nessasary changes to make it work with older infrastructure)

the redhat kernel maintainers are _very_ aware of what patches they maintain, and have been making a significant effort to reduce the size of that patch set

RHEL 5.4 released

Posted Sep 3, 2009 2:31 UTC (Thu) by qg6te2 (guest, #52587) [Link] (1 responses)

the redhat kernel maintainers are _very_ aware of what patches they maintain, and have been making a significant effort to reduce the size of that patch set

In what sense? The set of patches for RHEL 5 (2.6.18) has increased, no? I'm sure the kernel used for RHEL 6 will be at first far closer to upstream, but overtime we'll end up with another huge patch set. (I'm not counting Fedora here, given the 1 year lifetime of each release.)

RHEL 5.4 released

Posted Sep 3, 2009 3:21 UTC (Thu) by dlang (guest, #313) [Link]

well yes, the list of patches to 2.6.18 will keep growing as long as redhat maintains it.

but (other than the initial set (which is what I was referring to above) almost all of the patches are patches _from_ the upstream kernel, so there is nothing to try to push upstream.

you must not remember the bad old days (in the 2.4/2.5 kernel era) where redhat had so many patches to their kernel that they were becoming incompatible with other distros due to all the 'extra' stuff that they included.

RHEL 5.4 released

Posted Sep 3, 2009 17:01 UTC (Thu) by blitzkrieg3 (guest, #57873) [Link] (2 responses)

While the other comments are not necessarily wrong, they do leave out kernel ABI. When Red Hat says it's using -18, they mean that the kernel ABI has _not_ changed since vanilla -18. This is required for proprietary drivers and even some software when syscalls are added and removed

RHEL 5.4 released

Posted Sep 4, 2009 10:53 UTC (Fri) by nix (subscriber, #2304) [Link] (1 responses)

Syscalls are removed? When?

(Apps are broken by syscalls being added? How?)

RHEL 5.4 released

Posted Sep 4, 2009 14:26 UTC (Fri) by blitzkrieg3 (guest, #57873) [Link]

Okay, maybe I should have said _changed_, but I think there are syscalls that get removed from time to time, I just can't think of them.

http://lwn.net/Articles/350219/

There is also the question of which semantics older applications should get. Jamie Lokier argued that applications requesting O_SYNC behavior wanted full metadata synchronization, even if the kernel has been cheating them out of the full experience. So, when running under a future kernel with a proper O_SYNC implementation, an old, binary application should get O_SYNC behavior.