Leading items

The grumpy editor answers a call from CyanogenMod 7

The Android 2.3 "Gingerbread" release happened in late 2010, but that version of the operating system was initially only available to those who obtained a Nexus S handset. Your editor, not being one of those people, has had to wait a little longer. Nexus One owners running stock firmware have been receiving updates for a few weeks as of this writing. Those running CyanogenMod have yet to see a stable Gingerbread-based distribution, but their wait, too, is coming to an end; CyanogenMod 7.0.0-rc2 was announced on March 8. Your editor, never afraid to brick an expensive handset, decided to give this release candidate a try. Talking on the phone is overrated anyway.

The initial experience says that there is little need to worry; this release candidate seems to be stable indeed on the Nexus One. Your editor's careful Nandroid backup, it seems, will not be needed; there is no evident reason to go back to the older code. That said, there are reports of a "wonkiness issue" which affects phone calls in rc2; your editor has not yet seen this behavior but others certainly have. Smarter CyanogenMod users may want to wait for the official 7.0 release before updating.

Naturally, CyanogenMod 7 brings all of the enhancements that Google put into the Gingerbread release. The new color scheme is nice - though some applications (WeatherBug, for example) have not yet caught up to a world where writing black text into the (now black) notification bar fails to yield the desired results. The improved on-screen keyboard is most welcome; it seems easier to avoid typing mistakes and the number of needed mode-shifts is reduced. Most welcome are the ability to apply the spelling corrector anywhere in a message and a comma key which does not require a mode shift to reach.

Some other 2.3 features look nice, but your editor has not, yet, had the ability to try them out. Near-field communications are not supported by the Nexus One, and there is no nearby place to use that capability yet in any case. For the time being, perhaps, being unable to spend money directly with the handset is not an entirely bad thing. This release also supports Internet telephony via an outside SIP provider, a feature which certainly merits exploration at some point.

Nexus One users wanting the features found in the 2.3 release can, of course, just use the stock firmware which is nicely delivered via an over-the-air update. Owners of other handsets may not get any such service. One of the most valuable features of CyanogenMod is the large number of handsets supported; for many, it may be about the only way to get a current Android distribution on their phones.

But, naturally, there is more to CyanogenMod than ports; it also brings a number of features of its own. At the top of the list must be simple configurability. There is (once again) an apparent trend in parts of the free software community to remove configurability from systems in the name of "user friendliness"; sometimes distributions like Android are held out as examples of successful application of this approach. The CyanogenMod developers have not bought into that idea.

With CyanogenMod, one can set the color of the notification LED based on which application is trying to communicate something - email updates can be pink, missed calls yellow, and text messages a deep blue. There are three variants on the basic lock screen, with options for music player controls and access to a user-chosen application without unlocking the screen. The home screen/application launcher application has options controlling screen layout, whether the wallpaper scrolls, various gesture actions, where the trash can sits, and a vast number of other things. There is also a theme mechanism and, happily, a means to back up and restore all those settings. Other options control what happens when the search button is pressed, how hard the phone vibrates in response to various types of key events, which hours of the day the phone should be silent, and more. If you truly want to change which CPU frequency governor is used, or if you want to render the home screen in green only, CyanogenMod makes it possible.

The "power control" widget follows the same theme. The contents of this widget are under user control, with far more options (WiFi hotspot, mobile data, 2G/3G, etc.) than stock Android provides. One can even control the all-important flashlight capability through the power widget. An especially nice feature is the ability to make connections between some of the settings; for example, application synchronization can be enabled only when WiFi is enabled. That allows the cautious use of data in international-roaming environments without the danger of being bankrupted by a surprise application update.

All told, CyanogenMod's configuration options can seem a bit overwhelming, especially since their effect is not always obvious. But for somebody who makes a lot of use of their handset, a half-hour or so spent doing a depth-first traversal of the settings menus can go a long way toward optimizing the device's behavior.

Other CyanogenMod features include an "incognito" mode for the web browser, support for FLAC files in the music player (which has been "restyled" in a number of ways), SMS templates available via gestures, "phone goggles" (an outgoing call blocking mechanism), openVPN support, a dynamic equalizer for audio output, various hardware enablement and performance improvements, and much more.

Perhaps the real value of CyanogenMod, though, is that it turns Android into something resembling a real open source project. They take Google's code dumps and turn them into a working system which can be built and enhanced by anybody. The community around the distribution appears to be growing, and it is slowly developing the sort of infrastructure that a community-oriented project should have (even if they still insist on using obnoxious forums for their communications). Many people repackage Android binaries; CyanogenMod seems nearly alone in working at the source level. The result is a phone which works better for its users and which improves over time. It's hard to be grumpy about that.

Comments (19 posted)

Enterprise distributions and free software

The "enterprise distribution" business is a bit of a funny one; it often seems like an attempt - by vendors and customers both - to fit free software back into the business model long used by proprietary operating systems vendors. It will necessarily create some tensions between the values of the free software community (including free access, rapid development, and collaboration) and those of the business, which include rigid stability and the preservation of competitive advantage. Recent changes made by Red Hat show the effects that this tension can cause, especially when combined with strong, arguably unfair, competition.

The core Red Hat Enterprise Linux (RHEL) distribution is built entirely on free software (the supplementary repository includes a few third-party proprietary bits like Flash). The source for the distribution is freely available and freely distributable by any other party as long as trademark-relevant bits are stripped out. This freedom is exploited by free distributions like CentOS and Scientific Linux; it is also exploited by commercial competitors who are happy to charge money to support the distribution that Red Hat has made.

Naturally, it is the commercial competitors that are creating the most trouble for Red Hat, which invests a considerable amount of money into the development of both the software on which RHEL is based and the creation of RHEL itself. Companies like Oracle also contribute to upstream projects, but on a rather smaller scale, and they contribute little indeed to the development and maintenance of the RHEL distribution. Freed of that expense, these companies are able to offer support for Red Hat's distribution (or derivatives thereof) while charging much less. They are said to be actively courting Red Hat's existing customers, with a certain amount of success. It seems like a clear path to a successful business, even without the darker allusions, heard occasionally, that the real intent is simply to drive Red Hat out of business.

Given this situation, it is not entirely surprising that Red Hat has decided that it needs to respond; a company which makes life too easy for its competitors tends not to stay around for long. The company could try to make its distribution more proprietary to keep others from redistributing it. The use of proprietary kernel modules to this end is not unheard of, but it would be surprising indeed to see Red Hat take that approach. Proprietary user space components, like those used by Google to keep some control over Android, would have fewer licensing difficulties, but it still would be a major change for a (previously) free distribution. The good news is that Red Hat has resisted any such pressures - so far.

What Red Hat is doing is still seen by many as being something other than good news, though. Essentially, Red Hat is asserting stronger proprietary control over the metadata it possesses about the RHEL distribution. That metadata includes its "knowledge base" of problems and solutions, bugs and fixes, etc. What Red Hat wants to do is to exploit the advantage gained from having built and supported RHEL over all these years, and to make it harder for others to exploit that knowledge. The company is trying to compete by providing better support, and one way to have better support is to keep your competitors from making their own support offerings better through the use of your own information and experience.

Kernel source packaging

The move that brought all this to the community's attention was a change in the way that the source RPM file for the RHEL 6 kernel is packaged. Traditional source-packaging practice is to combine an unmodified upstream tarball, a set of distributor patches, and a script (the "spec file") which applies the patches and builds the software. The RHEL kernel package, instead, consists of a single tarball reflecting the kernel in its fully-patched form. There is a long list of patch titles in the spec file, but there is no other information which can be used to recover the specific patches which have been applied to the kernel.

The purpose of this change, evidently, is to make it harder for competitors to figure out why specific patches were made. Many changes applied to the RHEL kernel will have been done in response to specific customer problems; the understanding of those problems and how they were fixed will, at times, be useful in future support situations. By mixing all of its patches into a single, lossy tarball with no changelogs, Red Hat hopes to deprive others of this useful support information.

Will it work? The restriction of information in general should certainly deprive competitors of a useful resource, though the cynical might note that access to that information could be restored via an inexpensive low-end RHEL subscription. Whether broken-out patches for the kernel, in particular, are useful to competitors is not entirely clear. It may well be that Red Hat has irritated - and possibly made life harder for - the development community without bothering anybody else in any appreciable way.

Red Hat claims that the packaging change should not bother the development community at all, noting that the change had been in effect for four months before complaints were heard. One of the foundations of this claim is the fact that the RHEL "2.6.32" is far removed from anything called "2.6.32" anywhere else; the company once described it this way:

This kernel is a sort of Frankenstein's monster of fixes and backports; it is a distant relation to any sort of upstream kernel. Given that, Red Hat has said, most patches to this kernel do not apply in any useful way to the kernels the rest of us are working with. In other words, they claim, there is no value to the community in most of the RHEL kernel patches.

More importantly, though, Red Hat insists that it is a strong believer in working upstream. Any patches developed by Red Hat which have any relevance to upstream kernels are promptly sent upstream for merging. This line of reasoning suggests that there will not be anything of interest to the community in RHEL kernels; everything of value will have already been submitted for merging. The company's history backs up this claim; its history of contributions is matched by few others, and, by all accounts, the practice of contributing upstream is wired deeply into the company's culture.

That said, maintainers of the community stable kernels (and some other developers as well) have been heard to complain that Red Hat's practice is making work harder for them. Beyond that, though, this practice leaves the community with a bit of a bad taste in its mouth. The development of a significant fork of the Linux kernel has become more closed in a possibly ineffective attempt to support corporate interests. The fact that those corporate interests are in the personal interests of everybody whose salary is paid by that company and by everybody who benefits from that company's contributions perhaps softens the blow, but it does not make everything better. Many people are left wondering what the next step will be.

The enterprise distribution business

They may be right to wonder. The "enterprise distribution" business model, as it is currently implemented, is a strange distortion of the development community that supports it. It is said by some to violate the GPL, or, at least, to stretch its spirit to the breaking point; that is a subject which will be examined in another article. Of interest here is the fact that this model also led to the creation of kernels so divergent that its simple fixes no longer apply to the mainline. Your editor can't help thinking of Andrew Morton's classic keynote at the 2004 Ottawa Linux Symposium:

The RHEL kernel, which is certainly differentiated from any other kernel out there, would seem to be well described by Andrew's words. Red Hat's move to lock down information about this kernel is an admitted attempt to increase customer lock-in; they are trying to make it harder to move to another provider of support. Is this good for Linux?

Corporate moves do not happen in a vacuum; Red Hat's competitors will likely come up with responses of their own. Perhaps they will just develop other sources for the information they need. If, instead, they adopt a strategy like Red Hat's, we could end up with a fragmented range of incompatible Linux distributions, each of which tries to lock in customers with its own combination of unique features and proprietary information. Veterans of the Unix wars can be forgiven for shuddering a bit at that thought.

An alternative might be to move away from the idea of "frozen" kernels in enterprise distributions. The truth of the matter is that these kernels are far from frozen; RHEL's "2.6.32" kernel has features from 2.6.38 - and probably beyond. Nobody wants a kernel which was set in stone years before, so these "stable" kernels get no end of features backported into them. The result is a combination which has never been seen before in nature and which may have stability problems of its own. It is also expensive to create and expensive to maintain.

Perhaps enterprise distributions should stick closer to mainline kernels and even move to new releases occasionally? Oracle's "Unbreakable Enterprise Kernel" looks like a move in that direction. Novell did something similar with its SP1 update to SUSE Linux Enterprise 11. This approach would seem to offer some real advantages: users would get new features and fixes from upstream, and distributors would not have to pay the cost of creating and maintaining a highly divergent kernel. There would also be less need to treat related information as proprietary; companies would compete on their ability to support current mainline software, rather than a vendor-specific kernel. That seems like an environment in which Red Hat, which does so much to create mainline kernels, could compete most effectively.

Of course, anybody who listens to business advice from your editor has a strong chance of learning all about the bankruptcy process in short order. Beyond that, this is not a new idea; it was discussed here in 2007. The ultra-stable enterprise distribution model seems to be in no danger of going away, in any case, even though some vendors seem to be experimenting with some changes. There are customers, it seems, so there will be vendors.

In the end, companies need to make money or they disappear. Red Hat has, so far, found ways to keep its stockholders happy while supporting the development community which makes it all possible. One can only hope that, as this market becomes more competitive, the companies involved will continue to find a way to balance those important interests; the alternative, certainly, will be destructive to both.

Comments (77 posted)

Red Hat and the GPL

The changes that Red Hat has made in the way it releases kernel source for RHEL 6 have stirred up some controversy. There is a certain tension between the needs of an enterprise distribution and the interests of the free software projects on which it is based. While the changes that Red Hat has made do not seem very popular, there has also been a fair amount of speculation that the move somehow violates the GPL, which is a rather serious charge. But there is good reason to believe that Red Hat is not crossing into GPL-violation territory, though it may well be sitting close to that line.

First, the obvious: Red Hat has a top-notch legal team, with a lot of GPL experience, so it's a little hard to believe that those lawyers, at least, haven't examined Red Hat's position and believe it is defensible in the unlikely event of a lawsuit. While Red Hat has a legal team, LWN seems to be lacking in the legal budget department, so nothing in this article should be considered legal advice of any sort (I am not a lawyer, nor do I play one anywhere).

The arguments being made about GPL compliance seem to fall into two categories. The first is that distributing what is effectively a tarball of the kernel source does not qualify as the "preferred form of the work for making modifications to it" (as is specified in the GPL). While the second is that making the individual patches to the kernel source available to customers, but not allowing those customers to redistribute the patches to others, puts a "further restriction" on GPL-covered code (which the patches clearly are). While neither complaint is completely without merit, both would seem to fall short of GPL violations.

Preferred form

While there can certainly be arguments about what "preferred form" means for source code distributions, Red Hat's actions here don't seem very close to the line. The basic idea is that the code be released in digital form, along with any build scripts, so that downstream developers can reproduce the GPL-covered binary that is being shipped. It is partly meant to prevent recalcitrant vendors from releasing the source on paper—or stamped into clay tablets in cuneiform. One could argue that those methods do provide the source code, but it certainly isn't in a form preferred by anyone—nor does it adhere to the "medium customarily used for software interchange" requirement.

Obfuscated source code, where all the variables and function names get turned into unintelligible gibberish, or the source is deliberately split up in ways that are harder to work with, are a bit more of a gray area—but not much. Those kinds of actions should be seen as clear attempts to circumvent the GPL requirements. But that's not at all what Red Hat is doing.

The tarball that Red Hat is releasing may not be the preferred form for the Red Hat kernel developers to use, but that is not part of the requirement. Tarball releases of GPL code are a longstanding tradition in the free software world. Many projects still do it that way and the FSF itself has made releases that way in the past. While it is nice to get some visibility into the version control system of a project, it certainly isn't required for GPL compliance. No one complained that Red Hat didn't provide access to its Git repositories back when it was still providing the broken-out patches, for example. A Git repository is definitely the preferred form for Red Hat developers to use, but there are lots of reasons that the company might want to keep its repositories to itself—something that is obviously within its rights.

There is an oft-ignored GPL clause about modifications to consider here as well: "You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change." Red Hat may well be technically violating that clause, but it is hardly alone in that as it is pretty much completely ignored by everyone, without any real consequences. But this GPL requirement is of little interest to most. What everyone really wants is the reason that the change was made (along with which changes to other files go along with it), and the GPL is silent on any requirement to provide that level of detail.

For the RHEL kernel source, what Red Hat is providing is well within the GPL boundaries. It is also well within community norms. The community gets kernel tarballs from lots of companies (often those in the embedded Linux world), sometimes after a bit of a struggle, and is happy to get them. Red Hat shouldn't be treated any differently than those companies are with respect to the distribution of source code. Tarballs may be something of a barrier to easier collaboration, debugging, and so on, but distributing them doesn't rise to anything near a GPL violation. The other major complaint, on the other hand, is a bit less clear cut.

Trading GPL rights for support

While it isn't anything particularly new, some aspects of the RHEL support model are raising eyebrows in light of the changes to the kernel source distribution. In particular, the broken-out patches that used to come with the kernel source RPMs are evidently being made available to Red Hat customers, but only if they don't distribute them further on pain of losing support going forward. This sounds like a restriction on the GPL rights of the recipients of the patches—and it is—but that restriction is enforced via an agreement between Red Hat and its customers.

This is part of Red Hat's business model that Bradley Kuhn calls the "'if you like copyleft, your money is no good here' business model" (scroll down a ways to get to that part of his post). It is, he says, GPL-compliant "merely because the GPL is silent on whether or not you must keep someone as your customer". He also says that Red Hat was the first to make use of that particular business model, but he may be forgetting about Sveasoft.

The history is fading into link rot a bit at this point, so it is possible that Red Hat was the first to get there. But Sveasoft's subscription model, which essentially required subscribers to give up their distribution right on GPL-covered code provided by Sveasoft, was roundly criticized in the early 2000s. In 2004, though, Sveasoft announced that the FSF had reviewed and, somewhat surprisingly, at least at the time, approved that arrangement. Essentially, the support (or subscription) agreement is a contract between two entities and those parties can agree to pretty much anything they want. The GPL cannot protect one from agreeing to refrain from certain behavior.

In the distant past, there were also complaints that Red Hat will terminate its support contract for users that run their own kernel, rather than what is supplied with RHEL. While that is another restriction on exercising GPL rights, it is one that is rather easier to understand. Trying to support any random kernel that a customer wants to run is likely to be a quick way to drive Red Hat's kernel engineers mad. Complete termination of support may be an excessive reaction—and is probably rarely if ever used—but certainly not supporting those systems running other kernels is not an unreasonable position.

The inability to redistribute the kernel patches is clearly seen by some as the most egregious "violation" of the GPL rights of its customers. In some, possibly many, cases those patches are not even owned by Red Hat, but were instead submitted by others to the upstream kernel and then integrated or backported into the RHEL 6 kernel. But the results of that effort are being distributed in source form. If Red Hat's customers are willing to trade their GPL rights for support, there is little anyone else can do but complain.

On the other hand, a sufficiently motivated group could probably reverse-engineer the patches from the RHEL 6 kernel source. It would be a tedious and error-prone process, but Red Hat has said that all of the RHEL 6 features have been submitted upstream, so it should be doable. Anyone with a RHEL support contract would want to think long and hard before participating in such a project, however. Whether there is sufficient value in doing so remains to be seen.

Comments (164 posted)

Page editor: Jonathan Corbet
Next page: Security>>