The Android 2.3 "Gingerbread" release happened in late 2010, but that
version of the operating system was initially only available to those who
obtained a Nexus S handset. Your editor, not being one of those
people, has had to wait a little longer. Nexus One owners running stock
firmware have been receiving updates for a few weeks as of this writing.
Those running CyanogenMod
have yet to
see a stable Gingerbread-based distribution, but their wait, too, is coming
to an end; CyanogenMod
was announced on March 8. Your editor, never afraid to
brick an expensive handset, decided to give this release candidate a try.
Talking on the phone is overrated anyway.
The initial experience says that there is little need to worry; this
release candidate seems to be stable indeed on the Nexus One. Your
editor's careful Nandroid backup, it seems, will not be needed; there is no
evident reason to go back to the older code. That said, there are reports
of a "wonkiness issue" which affects phone calls in rc2; your editor has
not yet seen this behavior but others certainly have. Smarter CyanogenMod
users may want to wait for the official 7.0 release before updating.
Naturally, CyanogenMod 7 brings all of the enhancements that Google put
into the Gingerbread release. The new color scheme is nice - though some
applications (WeatherBug, for example) have not yet caught up to a world
where writing black text into the (now black) notification bar fails to
yield the desired results. The improved on-screen keyboard is most
welcome; it seems easier to avoid typing mistakes and the number of needed
mode-shifts is reduced. Most welcome are the ability to apply the spelling
corrector anywhere in a message and a comma key which does not require a mode
shift to reach.
Some other 2.3 features look nice, but your editor has not, yet, had the
ability to try them out. Near-field communications are not supported by
the Nexus One, and there is no nearby place to use that capability yet in
any case. For the time being, perhaps, being unable to spend money
directly with the handset is not an entirely bad thing. This release also
supports Internet telephony via an outside SIP provider, a feature which
certainly merits exploration at some point.
Nexus One users wanting the features found in the 2.3 release can, of
course, just use the stock firmware which is nicely delivered via an
over-the-air update. Owners of other handsets may not get any such
service. One of the most valuable features of CyanogenMod is the large
number of handsets supported; for many, it may be about the only way to get
a current Android distribution on their phones.
But, naturally, there is more to CyanogenMod than ports; it also brings a
number of features of its own. At the top of the list must be simple
configurability. There is (once again) an apparent trend in parts of the
free software community to remove configurability from systems in the name
of "user friendliness"; sometimes distributions like Android are held out
as examples of successful application of this approach. The CyanogenMod
developers have not bought into that idea.
With CyanogenMod, one can set the color of the notification LED based on
which application is trying to communicate something - email updates can be
pink, missed calls yellow, and text messages a deep blue. There are three
variants on the basic lock screen, with options for music player controls
and access to a user-chosen application without unlocking the screen. The
home screen/application launcher application has options controlling screen
layout, whether the wallpaper scrolls, various gesture actions, where the
trash can sits, and a vast number of other things. There is also a theme
mechanism and, happily, a means to
back up and restore all those settings. Other options control what happens
when the search button is pressed, how hard the phone vibrates in response
to various types of key events, which hours of the day the phone should be
silent, and more. If you truly want to change which CPU frequency governor is
used, or if you want to render the home screen in green only, CyanogenMod
makes it possible.
The "power control" widget follows the same theme. The contents of this
widget are under user control, with far more options (WiFi hotspot, mobile
data, 2G/3G, etc.) than stock Android provides. One can even control the
all-important flashlight capability through the power widget. An
especially nice feature is the ability to make connections between some of
the settings; for example, application synchronization can be enabled only
when WiFi is enabled. That allows the cautious use of data in
international-roaming environments without the danger of being bankrupted
by a surprise application update.
All told, CyanogenMod's configuration options can seem a bit overwhelming,
especially since their effect is not always obvious. But for somebody who
makes a lot of use of their handset, a half-hour or so spent doing a
depth-first traversal of the settings menus can go a long way toward
optimizing the device's behavior.
Other CyanogenMod features include an "incognito" mode for the web browser,
support for FLAC files in the music player (which has been "restyled" in a
number of ways), SMS templates available via gestures, "phone goggles" (an
outgoing call blocking mechanism), openVPN support, a dynamic equalizer for
audio output, various hardware enablement and performance improvements, and
Perhaps the real value of CyanogenMod, though, is that it turns Android
into something resembling a real open source project. They take Google's
code dumps and turn them into a working system which can be built and
enhanced by anybody. The community around the distribution appears to be
growing, and it is slowly developing the sort of infrastructure that a
community-oriented project should have (even if they still insist on using
obnoxious forums for their communications). Many people repackage Android
binaries; CyanogenMod seems nearly alone in working at the source level.
The result is a phone which works better for its users and which improves
over time. It's hard to be grumpy about that.
Comments (19 posted)
The "enterprise distribution" business is a bit of a funny one; it often
seems like an attempt - by vendors and customers both - to fit free
software back into the business model long used by proprietary operating systems
vendors. It will necessarily create some tensions between the values of
the free software community (including free access, rapid development, and
collaboration) and those of the business, which include rigid
stability and the preservation of competitive advantage. Recent changes
made by Red Hat show the effects that this tension can cause, especially
when combined with strong, arguably unfair, competition.
The core Red Hat Enterprise Linux (RHEL) distribution is built entirely on
free software (the supplementary repository includes a few third-party
proprietary bits like Flash). The
source for the distribution is freely available and freely distributable
by any other party as long as trademark-relevant bits are stripped out.
This freedom is exploited by free distributions like CentOS and Scientific
Linux; it is also exploited by commercial competitors who are happy to
charge money to support the distribution that Red Hat has made.
Naturally, it is the commercial competitors that are creating the most
trouble for Red Hat, which invests a considerable amount of money into the
development of both the software on which RHEL is based and the creation of
RHEL itself. Companies like Oracle also contribute to upstream projects,
but on a rather smaller scale, and they contribute little indeed to the
development and maintenance of the RHEL distribution. Freed of that expense, these
companies are able to offer support for Red Hat's distribution (or
derivatives thereof) while charging much less. They are said to be
actively courting Red Hat's existing customers, with a certain amount of
success. It seems like a clear path to a successful business, even without
the darker allusions, heard occasionally, that the real intent is simply to
drive Red Hat out of business.
Given this situation, it is not entirely surprising that Red Hat has
decided that it needs to respond; a company which makes life too easy for
its competitors tends not to stay around for long. The company
could try to make its distribution more proprietary to keep others
from redistributing it. The use of proprietary kernel modules to this end
is not unheard of, but it would be surprising indeed to see Red Hat take
that approach. Proprietary user space components, like those used by
Google to keep some control over Android, would have fewer licensing
difficulties, but it still would be a major change for a (previously)
free distribution. The good news is that Red Hat has resisted any
such pressures - so far.
What Red Hat is doing is still seen by many as being something other
than good news, though. Essentially, Red Hat is asserting stronger
proprietary control over the metadata it possesses about the RHEL
distribution. That metadata includes its "knowledge base" of problems and
solutions, bugs and fixes, etc. What Red Hat wants to do is to exploit the
advantage gained from having built and supported RHEL over all these years,
and to make it harder for others to exploit that knowledge. The company is
trying to compete by providing better support, and one way to have better
support is to keep your competitors from making their own support offerings
better through the use of your own information and experience.
Kernel source packaging
The move that brought all this to the community's attention was a change in
the way that the source RPM file for the RHEL 6 kernel is packaged.
Traditional source-packaging practice is to combine an unmodified upstream
tarball, a set of
distributor patches, and a script (the "spec file") which applies the
patches and builds the software. The RHEL kernel package, instead,
consists of a single tarball reflecting the kernel in its fully-patched
form. There is a long list of patch titles in the spec file, but there is
no other information which can be used to recover the specific patches
which have been applied to the kernel.
The purpose of this change, evidently, is to make it harder for competitors
to figure out why specific patches were made. Many changes applied to the
RHEL kernel will have been done in response to specific customer problems;
the understanding of those problems and how they were fixed will, at times,
be useful in future support situations. By mixing all of its patches into
a single, lossy tarball with no changelogs, Red Hat hopes to deprive others
of this useful support information.
Will it work? The restriction of information in general should certainly
deprive competitors of a useful resource, though the cynical might note
that access to that information could be restored via an inexpensive
low-end RHEL subscription. Whether broken-out patches for the kernel, in
particular, are useful to competitors is not entirely clear. It may well
be that Red Hat has irritated - and possibly made life harder for - the
development community without bothering anybody else in any appreciable
Red Hat claims that the packaging change should not bother the development
community at all, noting that the change had been in effect for four months
before complaints were heard. One of the foundations of this claim is the
fact that the RHEL "2.6.32" is far removed from
anything called "2.6.32" anywhere else; the company once described
it this way:
The Red Hat Enterprise Linux 6 kernel includes numerous subsystems
and enhancements from 2.6.34, as well as its predecessor
versions. As a result, the Red Hat Enterprise Linux 6 kernel cannot
be simply labeled as any particular upstream version. Rather, the
Red Hat Enterprise Linux 6 kernel is a hybrid of the latest several
kernel versions. And, as Red Hat provides regular updates over the
lifecycle of the product, we expect that the Red Hat Enterprise
Linux 6 kernel will incorporate selected features from future
upstream kernels that have yet to be developed.
This kernel is a sort of Frankenstein's monster of fixes and backports; it
is a distant relation to any sort of upstream kernel. Given that, Red Hat
has said, most patches to this kernel do not apply in any useful way to the
kernels the rest of us are working with. In other words, they claim, there
is no value to the community in most of the RHEL kernel patches.
More importantly, though, Red Hat insists that it is a strong believer in
working upstream. Any patches developed by Red Hat which have any
relevance to upstream kernels are promptly sent upstream for merging. This
line of reasoning suggests that there will not be anything of interest to
the community in RHEL kernels; everything of value will have already been
submitted for merging. The company's history backs up this claim; its
history of contributions is matched by few others, and, by all accounts,
the practice of contributing upstream is wired deeply into the company's
That said, maintainers of the community stable kernels (and some other
developers as well) have been heard to
complain that Red Hat's practice is making work harder for them. Beyond
that, though, this practice leaves the community with a bit of a bad taste
in its mouth. The development of a significant fork of the Linux kernel
has become more closed in a possibly ineffective attempt to support
corporate interests. The fact that those corporate interests are in the
personal interests of everybody whose salary is paid by that company and by
everybody who benefits from that company's contributions perhaps softens
the blow, but it does not make everything better. Many people are left
wondering what the next step will be.
The enterprise distribution business
They may be right to wonder. The "enterprise distribution" business model,
as it is currently implemented, is a strange distortion of the development
community that supports it. It is said by some to violate the GPL, or, at
least, to stretch its spirit to the breaking point; that is a subject which
will be examined in another article. Of interest here is the fact that
this model also led to the creation of kernels so
divergent that its simple fixes no longer apply to the mainline. Your
editor can't help thinking of Andrew
Morton's classic keynote at the 2004 Ottawa Linux Symposium:
I have a few words for the Linux vendors here. It's understandably
tempting for Linux vendors of various forms to seek to
differentiate their products by adding extra features to their
kernels. OK. But moving down this path does mean that there will be
incompatible versions of Linux released to the public. This will,
by design, lock some of our users into a particular vendor's
implementation of Linux. And this practice exposes the entire Linux
industry to charges of being fragmented. And it exposes us to the
charge that we are headed along the same path as that down which
the proprietary Unixes are deservedly vanishing... [A]s a
person who has some responsibility for Linux as a whole, I see the
perfectly understandable vendor strategy of offering product
differentiation as being in direct conflict with the long-term
interests of Linux.
The RHEL kernel, which is certainly differentiated from any other kernel
out there, would seem to be well described by Andrew's words. Red Hat's
move to lock down information about this kernel is an admitted attempt to
increase customer lock-in; they are trying to make it harder to move to
another provider of support. Is this good for Linux?
Corporate moves do not happen in a vacuum; Red Hat's competitors will
likely come up with responses of their own. Perhaps they will just develop
other sources for the information they need. If, instead, they adopt a
Red Hat's, we could end up with a fragmented range of incompatible Linux
distributions, each of which tries to lock in customers with its own
combination of unique features and proprietary information. Veterans of
the Unix wars can be forgiven for shuddering a bit at that thought.
An alternative might be to move away from the idea of "frozen" kernels in
enterprise distributions. The truth of the matter is that these kernels
are far from frozen; RHEL's "2.6.32" kernel has features from 2.6.38 - and
probably beyond. Nobody wants a kernel which was set in stone years
before, so these "stable" kernels get no end of features backported into
them. The result is a combination which has never been seen before in
nature and which may have stability problems of its own. It is also expensive
to create and expensive to maintain.
Perhaps enterprise distributions should stick closer to mainline kernels
and even move to new releases occasionally? Oracle's "Unbreakable Enterprise Kernel" looks like a
move in that direction. Novell did something similar with its SP1 update
to SUSE Linux Enterprise 11. This approach would seem to offer some
real advantages: users would get new features and fixes from upstream, and
distributors would not have to pay the cost of creating and maintaining a
highly divergent kernel. There would also be less need to treat related
information as proprietary; companies would compete on their ability to
support current mainline software, rather than a vendor-specific kernel.
That seems like an environment in which Red Hat, which does so much to
create mainline kernels, could compete most effectively.
Of course, anybody who listens to business advice from your editor has a
strong chance of learning all about the bankruptcy process in short order.
Beyond that, this is not a new idea; it was discussed here in 2007. The ultra-stable
enterprise distribution model seems to be in no danger of going away, in
any case, even though some vendors seem to be experimenting with some
changes. There are customers, it seems, so there will be vendors.
In the end, companies need to make money or they disappear. Red Hat has,
so far, found ways to keep its stockholders happy while supporting the
development community which makes it all possible. One can only hope that,
as this market becomes more competitive, the companies involved will
continue to find a way to balance those important interests; the
alternative, certainly, will be destructive to both.
Comments (77 posted)
The changes that Red Hat has made in the way it releases kernel source for RHEL
6 have stirred up some controversy. There is a certain tension between the needs of an enterprise
distribution and the interests of the free software projects on which it is
the changes that Red Hat has made do not seem very popular, there has also
been a fair amount of speculation that the move somehow violates the GPL,
which is a rather serious charge. But there is good reason to believe that
Red Hat is not crossing into GPL-violation territory, though it may well be
sitting close to that line.
First, the obvious: Red Hat has a top-notch legal team, with a lot of GPL
experience, so it's a little hard to believe that those lawyers, at least,
haven't examined Red Hat's position and believe it is defensible in the
unlikely event of a lawsuit. While Red Hat has a legal team, LWN seems to
be lacking in the legal budget department, so nothing in this article should
be considered legal advice of any sort (I am not a lawyer, nor do I play
The arguments being made
about GPL compliance seem to fall into two categories. The first is that
what is effectively a tarball of the kernel source does not qualify as the
"preferred form of the work for making modifications to it"
(as is specified in the GPL). While the second is that making the
individual patches to
the kernel source available to customers, but not allowing those customers
to redistribute the patches to others, puts a "further
GPL-covered code (which the patches clearly are). While neither complaint is
completely without merit, both would seem to fall short of GPL violations.
While there can certainly be arguments about what "preferred
form" means for source code distributions, Red Hat's actions here
don't seem very close to the line. The basic idea is that the code be
released in digital form, along with any build scripts, so that downstream
developers can reproduce the GPL-covered binary that is being shipped. It
is partly meant to prevent recalcitrant vendors from releasing the source on
paper—or stamped into clay tablets in cuneiform. One could argue
that those methods do provide the source code, but it certainly isn't in a
form preferred by anyone—nor does it adhere to the "medium
customarily used for software interchange" requirement.
Obfuscated source code, where all the variables and function names get
turned into unintelligible gibberish, or the source is deliberately split
up in ways that are harder to work with, are a bit more of a gray
area—but not much. Those kinds of actions should be seen as clear
attempts to circumvent the GPL requirements. But that's not at all what
Red Hat is doing.
The tarball that Red Hat is releasing may not be the preferred form for the
Red Hat kernel developers to use, but that is not part of the requirement.
Tarball releases of GPL code are a longstanding tradition in the free
software world. Many projects
still do it that way and the FSF itself has made releases that way in the
past. While it is nice to get some visibility into the version control
system of a project, it certainly isn't required for GPL compliance. No
one complained that Red Hat didn't provide access to its Git repositories
back when it was still providing the broken-out patches, for example. A
the preferred form for Red Hat developers to use, but there are lots of
reasons that the company might want to keep its repositories to
itself—something that is obviously within its rights.
There is an oft-ignored GPL clause about modifications to consider here as
well: "You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change."
Red Hat may well be technically violating that clause, but it is hardly
alone in that as it is pretty much completely ignored by everyone, without
any real consequences. But this GPL requirement is of little interest to most.
What everyone really wants is the reason
that the change was made (along with which changes to other files go along with
it), and the GPL is silent on any requirement to provide
that level of detail.
For the RHEL kernel source, what Red Hat is providing is well within the
GPL boundaries. It is also well within community norms. The community
gets kernel tarballs from lots of companies (often those in the embedded
Linux world), sometimes after a bit of a
struggle, and is happy to get them. Red Hat shouldn't be treated any
differently than those companies are with respect to the distribution of
Tarballs may be
something of a barrier to easier collaboration, debugging, and so on, but
distributing them doesn't rise to anything near a GPL violation. The other
complaint, on the other hand, is a bit less clear cut.
Trading GPL rights for support
While it isn't anything particularly new, some aspects of the RHEL support
model are raising eyebrows in light of the changes to the kernel source
distribution. In particular, the broken-out patches that used to come with
the kernel source RPMs are evidently being made available to Red Hat
customers, but only if they don't distribute them further on pain of losing
support going forward. This sounds
like a restriction on the GPL rights of the recipients of the
patches—and it is—but that restriction is enforced via an
agreement between Red Hat and its customers.
This is part of Red Hat's business model that Bradley Kuhn calls
the "'if you like copyleft, your money is no good here' business
model" (scroll down a ways to get to that part of his post). It is,
he says, GPL-compliant "merely because the GPL is silent on whether
or not you must keep someone as your customer". He also says that
Red Hat was the first to make use of that particular business model, but he
may be forgetting about Sveasoft.
The history is fading into link rot a bit at this point, so it is possible
that Red Hat
was the first to get there. But Sveasoft's subscription model, which
essentially required subscribers to give up their distribution right on
GPL-covered code provided by Sveasoft, was roundly criticized in the early
2000s. In 2004,
though, Sveasoft announced
that the FSF had reviewed and, somewhat surprisingly, at least at the time,
approved that arrangement. Essentially, the support (or subscription)
agreement is a contract between two entities and those parties can agree to
pretty much anything they want. The GPL cannot protect one from agreeing
from certain behavior.
In the distant past, there were also complaints that Red Hat will terminate
contract for users that run their own kernel, rather than what is supplied
with RHEL. While that is another restriction on exercising GPL rights, it
is one that is rather easier to understand. Trying to support any random
kernel that a customer wants to run is likely to be a quick way to drive
Red Hat's kernel engineers mad. Complete termination of support may be an
excessive reaction—and is probably rarely if ever used—but
certainly not supporting those systems running other kernels is not
The inability to redistribute the kernel patches is clearly seen by some as
the most egregious "violation" of the GPL rights of its customers. In
some, possibly many, cases those patches are not even owned by Red Hat, but
were instead submitted by others to the upstream kernel and then integrated
or backported into the RHEL 6 kernel. But the results of that effort
are being distributed in source form. If Red Hat's customers are
willing to trade their GPL rights for support, there is little anyone else can
do but complain.
On the other hand, a sufficiently motivated group could probably
reverse-engineer the patches from the RHEL 6 kernel source. It would be a
tedious and error-prone process, but Red Hat has said that
all of the RHEL 6 features have been submitted upstream, so it should be doable.
Anyone with a RHEL support contract would want to think long and hard
before participating in such a project, however. Whether there is
sufficient value in doing so remains to be seen.
Comments (164 posted)
Page editor: Jonathan Corbet
Next page: Security>>