Kernel competition in the enterprise space

Consider Red Hat Enterprise Linux 6; its kernel is ostensibly based on the 2.6.32 release. The actual kernel, as shipped by Red Hat, differs from 2.6.32 by around 7,700 patches, though. Many of those are fixes, but others are major new features, often backported from more recent releases. Thus, the RHEL "2.6.32" kernel includes features like per-session group scheduling, receive packet/flow steering, transparent huge pages, pstore, and, of course, support for a wide range of hardware that was not available when 2.6.32 shipped. Throw in a few out-of-tree features (SystemTap, for example), and the end result is a kernel far removed from anything shipped by kernel.org. That is why Red Hat has had no real use for the 2.6.32 stable kernel series for some years.

Red Hat's motivation for creating these kernels is not hard to understand; the company is trying to provide its customers with a combination of the stability that comes from well-aged software and the features, fixes, and performance improvements from the leading edge. This process, when it goes well, can give those customers the best of both worlds. On the other hand, the resulting kernels differ widely from the community's product, have not been tested by the community, and exclude recent features that have not been chosen for backporting. They are also quite expensive to create; behind Red Hat's many high-profile kernel hackers is an army of developers tasked with backporting features and keeping the resulting kernel stable and secure.

When developers grumble about enterprise kernels, what they are really saying is that enterprise distributions might be better served by simply updating to more current kernels. In the process they would get all those features, improvements, and bug fixes from the community, in the form that they were developed and tested by that community. Enterprise distributors shipping current kernels could dispense with much of their support expense and could better benefit from shared maintenance of stable kernel releases. The response that typically comes back is that enterprise customers worry about kernel version bumps (though massive changes hidden behind a minor number change are apparently not a problem) and that new kernels bring new bugs with them. The cost of stabilizing a new kernel release, it is suggested, could exceed that of backporting desired features into an older release.

Given that, it is interesting to see two other enterprise distributors pushing forward with newer kernels. Both SUSE Linux Enterprise Server 11 Service Pack 2 and Oracle's Unbreakable Enterprise Kernel Release 2 feature much more recent kernels - 3.0.10 and 3.0.16, respectively. In each case, the shift to a newer kernel is a clear attempt to create a more attractive distribution; we may be seeing the beginning of a change in the longstanding enterprise mindset.

SUSE seems firmly stuck in a second-place market position relative to Red Hat. As a result, the company will be searching for ways to differentiate its distribution from RHEL. SUSE almost certainly also lacks the kind of resources that Red Hat is able to apply to its enterprise kernels, so it will be looking for cheaper ways to provide a competitive set of features. Taking better advantage of the community's work by shipping more current kernels is one obvious way to do that. By shipping recent releases, SUSE does not have to backport fixes and features, and it is able to take advantage of the long-term stable support planned for the 3.0 kernel. In that context, it is not entirely surprising that SUSE has repeatedly pulled its customers forward, jumping from 2.6.27 to 2.6.32 in the Service Pack 1 release, then to 3.0.

Oracle, too, has a need to differentiate its distribution - even more so, given that said distribution is really just a rebranded RHEL. To that end, Oracle would like to push some of its in-house features like btrfs, which is optimistically labeled "production-ready" in a recent press release. If btrfs is indeed ready for production use, it certainly has only gotten there in very recent releases; moving to the 3.0 kernel allows Oracle to push this feature while minimizing the amount of work required to backport the most recent fixes. Oracle is offering this kernel with releases 5 and 6 of Oracle Linux; had Oracle stuck with Red Hat's RHEL 5 kernel, Oracle Linux 5 users would still be running something based on 2.6.18. For a company trying to provide a more feature-rich distribution on a budget, dropping in a current kernel must seem like a bargain.

What about the down side of new kernels - all those new bugs? Both companies have clearly tried to mitigate that risk by letting 3.0 stabilize for six months or so before shipping it to customers. There have been over 1,500 fixes applied in the 24 updates to 3.0 released so far. The real proof, though, is in users' experience. If SLES or Oracle Linux users experience bugs or performance regressions as a result of the kernel version change, they may soon start looking for alternatives. In the Oracle case, the original Red Hat kernel remains an option for customers; SUSE, instead, seems committed to the newer version.

Between these two distributions there should be enough users to eventually establish whether moving to newer kernels in the middle of an enterprise distribution's support period is a smart move or not. If it works out, SUSE and Oracle may benefit from an influx of customers who are tired of Red Hat's hybrid kernels. If the new kernels prove not to be enterprise-ready, instead, Red Hat's position may become even stronger. Learning which way things will go may take a while. Should Red Hat show up one day with a newer kernel for RHEL customers, though, we'll know that the issue has been decided at last.