By Jonathan Corbet
March 7, 2011
The "enterprise distribution" business is a bit of a funny one; it often
seems like an attempt - by vendors and customers both - to fit free
software back into the business model long used by proprietary operating systems
vendors. It will necessarily create some tensions between the values of
the free software community (including free access, rapid development, and
collaboration) and those of the business, which include rigid
stability and the preservation of competitive advantage. Recent changes
made by Red Hat show the effects that this tension can cause, especially
when combined with strong, arguably unfair, competition.
The core Red Hat Enterprise Linux (RHEL) distribution is built entirely on
free software (the supplementary repository includes a few third-party
proprietary bits like Flash). The
source for the distribution is freely available and freely distributable
by any other party as long as trademark-relevant bits are stripped out.
This freedom is exploited by free distributions like CentOS and Scientific
Linux; it is also exploited by commercial competitors who are happy to
charge money to support the distribution that Red Hat has made.
Naturally, it is the commercial competitors that are creating the most
trouble for Red Hat, which invests a considerable amount of money into the
development of both the software on which RHEL is based and the creation of
RHEL itself. Companies like Oracle also contribute to upstream projects,
but on a rather smaller scale, and they contribute little indeed to the
development and maintenance of the RHEL distribution. Freed of that expense, these
companies are able to offer support for Red Hat's distribution (or
derivatives thereof) while charging much less. They are said to be
actively courting Red Hat's existing customers, with a certain amount of
success. It seems like a clear path to a successful business, even without
the darker allusions, heard occasionally, that the real intent is simply to
drive Red Hat out of business.
Given this situation, it is not entirely surprising that Red Hat has
decided that it needs to respond; a company which makes life too easy for
its competitors tends not to stay around for long. The company
could try to make its distribution more proprietary to keep others
from redistributing it. The use of proprietary kernel modules to this end
is not unheard of, but it would be surprising indeed to see Red Hat take
that approach. Proprietary user space components, like those used by
Google to keep some control over Android, would have fewer licensing
difficulties, but it still would be a major change for a (previously)
free distribution. The good news is that Red Hat has resisted any
such pressures - so far.
What Red Hat is doing is still seen by many as being something other
than good news, though. Essentially, Red Hat is asserting stronger
proprietary control over the metadata it possesses about the RHEL
distribution. That metadata includes its "knowledge base" of problems and
solutions, bugs and fixes, etc. What Red Hat wants to do is to exploit the
advantage gained from having built and supported RHEL over all these years,
and to make it harder for others to exploit that knowledge. The company is
trying to compete by providing better support, and one way to have better
support is to keep your competitors from making their own support offerings
better through the use of your own information and experience.
Kernel source packaging
The move that brought all this to the community's attention was a change in
the way that the source RPM file for the RHEL 6 kernel is packaged.
Traditional source-packaging practice is to combine an unmodified upstream
tarball, a set of
distributor patches, and a script (the "spec file") which applies the
patches and builds the software. The RHEL kernel package, instead,
consists of a single tarball reflecting the kernel in its fully-patched
form. There is a long list of patch titles in the spec file, but there is
no other information which can be used to recover the specific patches
which have been applied to the kernel.
The purpose of this change, evidently, is to make it harder for competitors
to figure out why specific patches were made. Many changes applied to the
RHEL kernel will have been done in response to specific customer problems;
the understanding of those problems and how they were fixed will, at times,
be useful in future support situations. By mixing all of its patches into
a single, lossy tarball with no changelogs, Red Hat hopes to deprive others
of this useful support information.
Will it work? The restriction of information in general should certainly
deprive competitors of a useful resource, though the cynical might note
that access to that information could be restored via an inexpensive
low-end RHEL subscription. Whether broken-out patches for the kernel, in
particular, are useful to competitors is not entirely clear. It may well
be that Red Hat has irritated - and possibly made life harder for - the
development community without bothering anybody else in any appreciable
way.
Red Hat claims that the packaging change should not bother the development
community at all, noting that the change had been in effect for four months
before complaints were heard. One of the foundations of this claim is the
fact that the RHEL "2.6.32" is far removed from
anything called "2.6.32" anywhere else; the company once described
it this way:
The Red Hat Enterprise Linux 6 kernel includes numerous subsystems
and enhancements from 2.6.34, as well as its predecessor
versions. As a result, the Red Hat Enterprise Linux 6 kernel cannot
be simply labeled as any particular upstream version. Rather, the
Red Hat Enterprise Linux 6 kernel is a hybrid of the latest several
kernel versions. And, as Red Hat provides regular updates over the
lifecycle of the product, we expect that the Red Hat Enterprise
Linux 6 kernel will incorporate selected features from future
upstream kernels that have yet to be developed.
This kernel is a sort of Frankenstein's monster of fixes and backports; it
is a distant relation to any sort of upstream kernel. Given that, Red Hat
has said, most patches to this kernel do not apply in any useful way to the
kernels the rest of us are working with. In other words, they claim, there
is no value to the community in most of the RHEL kernel patches.
More importantly, though, Red Hat insists that it is a strong believer in
working upstream. Any patches developed by Red Hat which have any
relevance to upstream kernels are promptly sent upstream for merging. This
line of reasoning suggests that there will not be anything of interest to
the community in RHEL kernels; everything of value will have already been
submitted for merging. The company's history backs up this claim; its
history of contributions is matched by few others, and, by all accounts,
the practice of contributing upstream is wired deeply into the company's
culture.
That said, maintainers of the community stable kernels (and some other
developers as well) have been heard to
complain that Red Hat's practice is making work harder for them. Beyond
that, though, this practice leaves the community with a bit of a bad taste
in its mouth. The development of a significant fork of the Linux kernel
has become more closed in a possibly ineffective attempt to support
corporate interests. The fact that those corporate interests are in the
personal interests of everybody whose salary is paid by that company and by
everybody who benefits from that company's contributions perhaps softens
the blow, but it does not make everything better. Many people are left
wondering what the next step will be.
The enterprise distribution business
They may be right to wonder. The "enterprise distribution" business model,
as it is currently implemented, is a strange distortion of the development
community that supports it. It is said by some to violate the GPL, or, at
least, to stretch its spirit to the breaking point; that is a subject which
will be examined in another article. Of interest here is the fact that
this model also led to the creation of kernels so
divergent that its simple fixes no longer apply to the mainline. Your
editor can't help thinking of Andrew
Morton's classic keynote at the 2004 Ottawa Linux Symposium:
I have a few words for the Linux vendors here. It's understandably
tempting for Linux vendors of various forms to seek to
differentiate their products by adding extra features to their
kernels. OK. But moving down this path does mean that there will be
incompatible versions of Linux released to the public. This will,
by design, lock some of our users into a particular vendor's
implementation of Linux. And this practice exposes the entire Linux
industry to charges of being fragmented. And it exposes us to the
charge that we are headed along the same path as that down which
the proprietary Unixes are deservedly vanishing... [A]s a
person who has some responsibility for Linux as a whole, I see the
perfectly understandable vendor strategy of offering product
differentiation as being in direct conflict with the long-term
interests of Linux.
The RHEL kernel, which is certainly differentiated from any other kernel
out there, would seem to be well described by Andrew's words. Red Hat's
move to lock down information about this kernel is an admitted attempt to
increase customer lock-in; they are trying to make it harder to move to
another provider of support. Is this good for Linux?
Corporate moves do not happen in a vacuum; Red Hat's competitors will
likely come up with responses of their own. Perhaps they will just develop
other sources for the information they need. If, instead, they adopt a
strategy like
Red Hat's, we could end up with a fragmented range of incompatible Linux
distributions, each of which tries to lock in customers with its own
combination of unique features and proprietary information. Veterans of
the Unix wars can be forgiven for shuddering a bit at that thought.
An alternative might be to move away from the idea of "frozen" kernels in
enterprise distributions. The truth of the matter is that these kernels
are far from frozen; RHEL's "2.6.32" kernel has features from 2.6.38 - and
probably beyond. Nobody wants a kernel which was set in stone years
before, so these "stable" kernels get no end of features backported into
them. The result is a combination which has never been seen before in
nature and which may have stability problems of its own. It is also expensive
to create and expensive to maintain.
Perhaps enterprise distributions should stick closer to mainline kernels
and even move to new releases occasionally? Oracle's "Unbreakable Enterprise Kernel" looks like a
move in that direction. Novell did something similar with its SP1 update
to SUSE Linux Enterprise 11. This approach would seem to offer some
real advantages: users would get new features and fixes from upstream, and
distributors would not have to pay the cost of creating and maintaining a
highly divergent kernel. There would also be less need to treat related
information as proprietary; companies would compete on their ability to
support current mainline software, rather than a vendor-specific kernel.
That seems like an environment in which Red Hat, which does so much to
create mainline kernels, could compete most effectively.
Of course, anybody who listens to business advice from your editor has a
strong chance of learning all about the bankruptcy process in short order.
Beyond that, this is not a new idea; it was discussed here in 2007. The ultra-stable
enterprise distribution model seems to be in no danger of going away, in
any case, even though some vendors seem to be experimenting with some
changes. There are customers, it seems, so there will be vendors.
In the end, companies need to make money or they disappear. Red Hat has,
so far, found ways to keep its stockholders happy while supporting the
development community which makes it all possible. One can only hope that,
as this market becomes more competitive, the companies involved will
continue to find a way to balance those important interests; the
alternative, certainly, will be destructive to both.
(
Log in to post comments)