LWN.net Weekly Edition for June 28, 2012

LinuxCon Japan: The business of contribution

Bdale Garbee has been involved in free software, particularly Debian, for a long time, but he's also been involved in the "corporate side" of open source at HP for quite some time as well. That gives him a good perspective on the interface between companies and free software projects. In a bit of a reprise of a talk that goes back to 2003 or 2004, he spoke at LinuxCon Japan on "The Business of Contribution"—how companies and free software projects can benefit from each other.

There are multiple ways that someone can become involved in the free and open source software (FOSS) ecosystem. They range from those who take the code and use it in some way all the way up to those that lead and manage a project or subsystem. Garbee said that he hoped to convince attendees to move their way up to higher levels of engagement in FOSS.

In general FOSS projects have certain things in common. They are developed and supported by the community, there is no single company in charge, and there is a wide range of contributions that come from people with a variety of interests, abilities, and motivations. The users of the software have flexibility in how they acquire support; they can become a developer or pay someone to do development for them. This puts them in control of their own destiny, which stands in contrast to license arrangements where users just get binaries.

The key to FOSS licensing is that "people we don't even know are empowered to build things we can't even imagine", Garbee said. Everyone can take advantage of those newly built things, both individuals and companies.

Business and community

Something that is probably immediately obvious, he said, are that the expectations are somewhat different for FOSS developers and companies. It is a bit difficult to characterize what FOSS developers are looking for because each project and developer is different, but there are some common elements.

Developers often want to scratch an itch and solve a problem that they are having. They also are generally not working to a particular schedule, something that is particularly true with Debian for example, he said. FOSS developers are looking for the fun that comes from a collaborative development project. Lastly, FOSS development is a means to develop and maintain a personal reputation, which is something that can be hard for companies to understand when hiring them.

On the other side of the coin, companies need to make money. In fact, in the US, companies are legally required to make decisions based on their revenue generation possibilities. Companies also must grow and expand to provide the financial return that investors expect. That means that companies want to be able to differentiate their products from their competitors' products. Corporate reputation is also important, but there are so many ways to lower a company's reputation, and seemingly few ways to increase it, which makes doing new things risky, he said.

But there is good news in that there is something that the community and companies can agree on: positive user experience. For the community, that will increase the number of people who want to use and benefit from what it has made. For companies, a great user experience will help motivate customers to want to give them money.

Licenses and business models

FOSS licenses have their basis in copyright law, which differs somewhat in various jurisdictions, but there is a lot of commonality. The choice of which license to use for a given project is made by the founder(s) and those that join the project later are explicitly accepting those terms for their contributions. That "very special and unique right" to set the license is an important decision as it can have a "really profound impact" on the nature of the community that forms. In addition, a particular choice of license can sometimes cause new projects to form using different license terms.

There are lots of different open source software licenses available—60 or so—which is a bit unfortunate. But, they break down into two basic types: reciprocal or permissive. Reciprocal licenses require that one pass on the same rights to the code to those you distribute it to, while permissive licenses do not require that. The reality is that each license type has its place, Garbee said. It is important for project founders to carefully consider the license they choose, while it is equally important for contributors and users to understand the license that is being used.

It is vital to recognize that "open source" is not a business model, it is, instead, a process for development. One can use open source to enable the sale of hardware—this is what HP does—by ensuring that Linux runs well on that hardware. That is one possible business model, but there are others.

Selling services, like training or support, is a common example of a business model around FOSS. One could also give the source away, but charge for access to the binaries. That may sound strange, but it is more or less what the enterprise Linux distributions do, he said. Another model is to open up the core of some large software system, perhaps a framework and a few sample modules, and build a business around selling add-ons. That model can work, but it sometimes becomes something of an arms race as FOSS developers create and contribute add-ons that compete with the ones that are sold.

Working with FOSS gives a company the "opportunity to harness the creativity of a very diverse community". There is no company on earth, or even a collection of companies, that can bring to bear the talents that exist in a community like that surrounding the Linux kernel. Finding a way to work with FOSS communities can be an enormous boon for a company.

Deciding to contribute

There are multiple benefits to a company for participating in open source. For one, the software has "zero marginal cost", which means that a second (or third ...) copy costs nothing extra. The web provides a low-cost collaboration and distribution mechanism, which also reduces costs. Most of tools are also available as free software, so building a product is less costly. There is an enormous amount of existing code in the form of libraries and tools that can be used to build a product more cheaply and with a faster time to market.

There are multiple levels of engagement with FOSS, starting with taking the code and running it. Those who do so are valid members of the FOSS community. Going beyond that and monitoring the development or user mailing lists makes you a bit more engaged. The next step might be to find and fix a bug or to answer a question on the list. Each step up gives a person more influence in the direction of the project.

There are more and less efficient ways to use open source in a product, he said. It is common that one or more OSS components will require some change before they can be used, and that is "completely legally OK". If there is a reciprocal license, the changes must be distributed, but that's not a big deal. The problem occurs when it is time to put out a new version of the product. If the changes have not gone upstream, they will need to be ported to any new version of that upstream project.

On the other hand, if the changes have been contributed upstream—and shepherded through the process of getting them merged—the features and bug fixes needed will be there in the next versions. That means that the work that was done once won't have to be repeated each time a new version of the product (which may require a new version of the upstream project) is built. That doesn't mean there is no work that needs to be done, as the new version may have new bugs that need to be addressed, but it is much more efficient to get changes upstream.

For the kernel, things like device drivers will be kept up to date as the internals of the kernel change over time. The kernel is a good example because it is such a fast moving and dynamic body of code, but the situation exists in other projects as well.

Choosing to use FOSS is a "first step to making the best use of resources", Garbee said. That allows a company to focus on differentiating its product. But maximizing community involvement helps to grow the body of technology that's available to be used in future products, without having to reinvest over and over again.

Using FOSS does not mean that a company has to give everything away for free. But it doesn't mean that the company gets everything it wants for free either. There will always be a small piece that the company will need to invest in, but in spite of that, working with FOSS projects is an "incredibly powerful model".

Getting more involved with a project can bring other advantages too. One can help influence the project's direction and learn about enhancements that others are working on. Learning about security vulnerabilities and other problems in the code earlier will also be useful. It will give insight into who is contributing to the project, which may lead to interesting information on what competitors are up to. Because of that, Garbee is surprised by the email he sometimes gets from people working at other companies who didn't expect some announcement that HP has made. Had they been paying attention to the projects that HP was working on, the announcement would probably have been less surprising.

One way to experiment with FOSS is to work with partners on a project without really committing the company to doing the work, but that will not build "FOSS equity". The technical accomplishments and the reputation that the project builds will not accrue to the company (and instead to the partner). A better way is to have company employees who work directly on FOSS projects. The expertise that they gain can be reapplied to other projects, and the company gets to help set the direction for the project. In addition, the company can take credit for the work it is doing.

One important thing to recognize, however, is that reputation is built by individuals. Getting code accepted involves a combination of technical prowess and reputation within the community. In that case, the reputation that matters is that of the individual. Linus Torvalds has said that he doesn't really know the corporate affiliation of most kernel developers. In order for the company to be successful in the community, it must allow individuals to gain reputation in that community. It is important for managers to understand this, he said, and to make it a part of the career development of the employee.

Garbee noted that he once was talking to a peer at a competitor who asked about how he should review a particular employee. That person had been tasked with getting some code upstream, but ran into a competing proposal. The employee then made sure that all of the company's needs were met in the alternative, which eventually was merged. The peer was wondering whether the employee should be rewarded for that, and Garbee's response was "give the guy a raise". He recognized that the alternative proposal was better, dealt with his ego, and ensured that the company's requirements were met. He achieved the desired result, rather than get his own code upstream, which fully met the objective. That is something that many managers have a hard time understanding, he said.

Does this really work?

Garbee then described HP's experiences with FOSS as something of a justification for what he had been saying. In the last ten years or so, HP has accomplished a great deal by trying to engage with FOSS projects "as best we can". That is one of the things that has led HP to be the world's largest IT company by many measures, he said.

One of the more significant recent decisions that HP made with regard to FOSS was for the HP Public Cloud, which is based on OpenStack and Ubuntu LTS. The company has made a "significant commitment" of HP resources to OpenStack. It is now one of the largest technical contributors to the project and hosts its build farm. In addition, HP is helping with the transition of OpenStack to a "foundation model", similar to that of Eclipse or Apache.

HP has recently completed an analysis of all of its products and found that the majority contain at least some FOSS. Any of those that do have FOSS components are increasing that percentage rapidly. It is the most efficient use of HP's software engineering effort, he said. All across HP, there is more and more direct engagement with FOSS projects, which is helping to make the company more successful.

As a consequence of the talk, Garbee said that he hoped the audience would find that becoming more directly engaged with FOSS projects made sense for them and their companies. He ended with a famous quote from Margaret Mead:

That is, he said, a strong statement of how things work in open source. If someday you look around and wonder why it is that something isn't fixed, remember that you are someone.

Garbee's talk was clearly aimed at the mostly Asian audience, and tried to assist those present with arguments to make to their management about the benefits of working more closely with FOSS projects. The arguments are useful for anyone trying to get their company more involved in FOSS projects, of course, but Asian companies have traditionally been less engaged with upstream projects. He also slipped some rocket slides into the talk, which, beyond bringing some of his hobby into view, also gave him an opportunity to show a more personal connection to open source.

One of those slides showed him carrying a rather large rocket to the launching pad (shown at right). That rocket was subsequently lost after reaching Mach 1.3 and attaining 5000m above the ground because of a malfunctioning closed-source altimeter. That led him and Keith Packard into a collaboration to create open hardware and software for rocket telemetry and control, which ultimately helps him "not lose rockets". So, not only is open source good for companies, it's good for rockets—and other hobbies too.

Comments (3 posted)

CloudLinux busted — twice

Proprietary kernel modules have always had an unclear status in the kernel development community. Most (but not all) developers seem to agree that it is possible to make a module that is not a derived product of the kernel itself (and, thus, not subject to the requirements of the GPL); the Andrew filesystem, ported from Unix in the early days, has been cited as an example of this kind of module. Most developers also seem to agree that many other types of modules cannot be written as an independent work. There is far less agreement on where the line should be drawn, but there is a consensus around the idea that modules should not lie about their licensing status to gain closer access to kernel internals. The story of a company that was recently caught shipping that sort of dishonest module shows the kind of reaction that can be expected—and why closed-source modules are a bad idea in general.

Loadable kernel modules can only access kernel symbols that have been explicitly exported for that use. Two types of declarations are used to export these symbols. EXPORT_SYMBOL_GPL() is an indication that any module using the symbol is reaching so deeply into the kernel that it is, by necessity, a derived product of the kernel. Instead, EXPORT_SYMBOL() is used for symbols that only might indicate a derived-product status; it is also used for symbols that were already exported before the GPL-only mechanism was added.

The kernel's module loader includes an enforcement mechanism that prevents any module without a GPL-compatible license from accessing explicitly GPL-only symbols. To function, this mechanism clearly must know what the license for a specific module is. That is the weak point of the entire scheme: the kernel uses the honor system. Loadable modules include a special declaration that tell the kernel which license applies; the module loader looks at that declaration and decides whether the module will have access to GPL-only symbols or not. If the module lies about its license, the kernel will normally trust it.

That is where CloudLinux comes in. This company offers a CentOS-like distribution aimed at the needs of cloud computing providers. Essentially, CloudLinux has taken a RHEL clone, added its own container mechanism, and created a business around providing updates and support. The container mechanism is at least partially implemented by the company's "lve" kernel module; lve includes an access control and resource limit mechanism that, seemingly, is implemented independently of the kernel's control group subsystem. It can place limits on CPU usage, I/O bandwidth usage, and memory consumption, among other things; all this is managed through an administrator-friendly XML file.

Matthew Garrett recently became aware that, while the lve module claims to be GPL-licensed, there is not any source available for it; he responded with a patch causing the kernel to recognize and blacklist this module. Since lve uses GPL-only symbols, this change will prevent it from loading, essentially breaking it. This change has yet to be pushed into the mainline, but it has been accepted by the modules maintainer (Rusty Russell) and marked for stable updates as well. Unless something changes, it will soon become impossible to load the lve module into any current kernels.

The immediate effect of this change on CloudLinux is likely to be limited. Chances are that few users download the binary lve module and load it into their own kernels; almost all lve users are probably running the entire CloudLinux distribution. CloudLinux could easily patch its own kernel to remove the check for the lve module and restore the previous functionality. But such a move would almost certainly be noticed, and there is a distinct possibility that one or more kernel developers would react in an unpleasant way. It would not be an advisable course of action.

What CloudLinux did instead was to apologize and plead for time:

This response was not entirely well received; it is not at all clear why CloudLinux is unable to immediately ship the source corresponding to the binary module it is distributing. That said, the community will probably limit itself to grumbling as long as CloudLinux follows through and comes back into compliance with the kernel's licensing. Nobody likes a delay in a source release, but short delays are usually tolerated, grudgingly.

So the licensing part of this episode is, probably, resolved, but it is also interesting to look at why the lve module needs GPL-only symbols in the first place. The specific symbols it needs relate to the creation of binary attributes (essentially unstructured sysfs files) in the sysfs virtual filesystem. Much of the internal sysfs interface is GPL-only, and the functions for the management of binary attributes (which are generally discouraged) are certainly so. It turns out that lve creates binary attributes for a reason that would certainly never get through a proper review:

As a comprehensive security solution, this implementation leaves just a little bit to be desired.

CloudLinux is trying to reduce the amount of damage that could be done should an attacker successfully compromise a daemon running on the system. Kernel developers have been working for years to mitigate just this sort of threat; the results of that work take the form of security modules, namespaces, and more. Had CloudLinux done its work in the open, it would certainly have been directed toward solutions that have support from others and years of experience in actual use. Solutions, in other words, that might actually deliver some additional security. Instead, they got a one-off "not pretty hack" that could never get upstream.

If CloudLinux follows through on its plan to close the module entirely, it will have to cease use of GPL-only symbols and, thus, will not be able to implement this particular mechanism. That may be enough to convince kernel developers to leave it alone. But it should leave others wondering what other surprises may be lurking inside—surprises that cannot be investigated, since the source for the module is not available.

There may well be a viable business in the creation of a well-supported distribution aimed at the needs of cloud providers. With solid security support and some top-tier user-space management tools, a company like CloudLinux could well produce an offering that others are willing to pay for. But the most robust and trustworthy server distribution will come from using (and improving) the work that many developers have done for years to create a useful containers implementation. It is hard to imagine that a proprietary solution put together by a small company can work as well. Compared to that, the risk of licensing troubles seems like a case of adding insult to injury.

Comments (17 posted)

GNOME and input method integration

Those of us who type in Latin characters may easily overlook what it takes to get text into windows or command lines in other writing systems. Entry of characters not found on one's keyboard requires the use of an input method (IM) which turns multiple keystrokes into characters. There are plenty of capable projects, but they often lack deep integration into the desktop environment or widget toolkit. In April, GNOME developer Rui Matos proposed a feature for the upcoming GNOME 3.6 release that would integrate the IBus framework into the core GNOME desktop, tackling this precise challenge. IBus is a framework that allows the user to select — and switch between — multiple IMs. The plan spawned considerable debate, not only on the merits of IBus, but on the wisdom of tightly integrating a single component into the desktop environment. Complicating matters is the divide between the bulk of the GNOME developer community and those users who depend on input methods, primarily from the Chinese-Japanese-Korean (CJK) language communities.

At the heart of the issue is how CJK users input text. In alphabetic or syllabic writing systems (including European, North Indic, and Middle Eastern), there is a known mapping between the pronunciation of a word and its written representation. In the logographic writing systems of CJK, where the strokes of the character do not represent sounds or other subdivisions of the word, users make use of an IM instead. There are phonetic IMs such as Pinyin (in which the user types in the romanization of a word on the keyboard and the IM substitutes the correct character, or logogram), shape-based IMs like Cangjie (which decompose the logogram in a strict order), and many hybrids. In most cases, good dictionaries or tables are required, plus word-prediction, spell-checking, and other add-on features to cope with homophones and other tricky bits. Mobile phone users got a taste of the IM experience through T9 and other numeric-keypad predictive text systems, which have largely been replaced.

For everyday typing, the challenge is greater because no one input method is inherently superior: if you do not know how to pronounce an unfamiliar word, a phonetic method is no use, and switching to shape-based method makes sense. On the other hand, typing a word that you hear but have not seen written down demands a phonetic method. Likewise, regional accents can have very different pronunciations for the same logogram, but there is often more than one way to decompose a logogram by shape — not to mention the problem of writing reforms like Simplified Chinese, which is not simplified uniformly between mainland Chinese hanzi and Japanese kanji. Thus, almost no IM can be relied upon to the exclusion of all others. Throw in open source developers' penchant for reinventing the wheel to scratch their own itches, and the result is multiple IM frameworks, each of which can load individual IMs as the user chooses.

Supporting all of these frameworks is a challenge, one that Matos (who works mostly on gnome-control-center, GNOME Shell, and other desktop components) felt detracted from GNOME's ability to provide a consistent desktop experience and left CJK users a step behind those users whose writing system works out-of-the-box. As he explained on the desktop-devel list, IM framework proliferation reduces the odds that any of the frameworks will get enough testing to be robust, and it greatly complicates bug triage for GNOME. Plus, many of the popular IM frameworks also attempt to be cross-desktop, which makes integrating them seamlessly into GNOME difficult. Their visual appearance does not match, they conflict with core GNOME settings like XKB configuration and key bindings, and they cannot be automatically started (relying either on user interaction or shell scripts to launch). The fix he proposed is to take one IM framework, tie it in more directly to GNOME (including adding or extending GNOME APIs where needed), and make sure that it works for the majority of users.

Specifically, Matos said, GNOME has three requirements to properly integrate an IM framework with the desktop: a GTK+ module, a D-Bus API that can enumerate, activate, and configure installed IMs, and a D-Bus API on GNOME's side that the framework can use to draw predictive-text pop-up windows. Right now, he said, only IBus meets all of the requirements. Owen Taylor added some additional requirements to integrate the framework with GNOME accessibility and configuration standards, and a quality set of existing IMs. The IBus team expressed a willingness to work with GNOME, which was also critical.

Feedback from the CJK community

However, the plan elicited strong reactions from some members of the CJK user community. Marguerite Su replied that few if any CJK users like IBus; with most preferring rival framework Fcitx out of the existing options available on desktop Linux. Others chimed in with criticism of IBus. The complaints included several overlapping issues, including customization features, IM engine quality, and speed. Su elaborated on the feature disparity, saying that some users want to customize shortcuts, rearrange the word-completion suggestions, or fetch new dictionary words from Internet servers. But trading IBus for Fcitx is not the simple solution. IBus works well for Simplified Chinese, but not Traditional Chinese, while Fcitx is focused largely on Chinese and has less robust support for other languages, including Japanese (although it has plans to improve its Japanese support). Chinese users are greater in number, which might argue in favor of Fcitx, but surely the best solution would be to find a framework that serves everyone. Support for writing systems unrelated to the CJK family (such as Tibetan or Thai) is weaker in general across the IM frameworks.

But over the course of the sometimes-heated list discussion, the IBus critics proposed not simply adopting Fcitx outright, but keeping IM frameworks a user-controlled, pluggable option. For example, Liang Suilong, who argued that IBus is laggy, still advocated building an "IM framework framework" for GNOME, thus allowing users to choose the framework they preferred. But that idea was not well-received by GNOME developers. Tomas Frydrych summed up the two sides of the divide:

As others pointed out on the list, the latter point was not accurate; IBus does support CJK, but not all users are happy with its IM implementations or auxiliary features. Complicating matters more is the cultural divide between the groups. Weng Xuetian pointed out that the core GNOME developers are not IM users (most speak European languages), which makes them unqualified to select the "best" IM framework to then be used by a large number GNOME users who were not able to participate in the discussion.

Frameworks, not engines

But the groundswell of IBus criticism was not the final word. Tommy He said that the debate was too focused on CJK input, to the exclusion of other writing systems, and moreover that much of the lobbying in favor of Fcitx focused on the quality of the various IM engines, rather than the frameworks themselves. After all, if Fcitx can add a new and improved Japanese IM, then IBus could add one for Traditional Chinese.

Admittedly, the framework-versus-IM debate is a difficult one to pin down. On the one hand, Fcitx proponents enjoy its more flexible plugin system, which allows user-defined macros and other daily-use features. But that same flexibility could make it more difficult to integrate with GNOME's existing language settings and preferences, when providing a better first-run experience is explicitly one of the goals. As Owen Taylor observed, Fcitx generates its configuration GUI on-the-fly, which is a technique GNOME finds problematic and tries to avoid.

On May 14, Taylor attempted to re-center the discussion to the framework question, arguing that GNOME needs to pick one framework, then make it work "as well as we can possibly make it for all users. Multiple partially working frameworks are not a substitute." When the IBus-versus-Fcitx debate briefly resurfaced, Bastien Nocera advocated starting the IBus integration work as soon as possible, then inviting Fcitx to bring its own code demonstrating that it could fit the role better. "A lot of the code should be reusable, and you can show how much more awesome and complete your favourite IMF is."

When it looked like the core GNOME developers had made their final decision to integrate IBus, a few IBus detractors asked the team to postpone the decision for further discussion, but without success. Nocera reiterated his opinion that it would be better to pick an imperfect IM framework and replace it in later releases than to do nothing:

But Jasper St. Pierre compared the decision to other components, like audio. GNOME does not attempt to support ESD and OSS in addition to PulseAudio due to limited resources, he said, and GNOME's choice of IM frameworks will not force distributions to follow suit. "I doubt we have the resources to support both IBus and FCITX, and provide a good experience for both. Individual distributions may, but that's their call, not GNOME's."

The Fcitx lobby may not be satisfied with that response, but it looks like IBus is here to stay for the 3.6 development cycle at the very least. The GNOME wiki page lists seven tracker bugs to follow the integration with GNOME Shell, GNOME control center, and other components. There is still plenty of integration and debugging work to be done, including improving IBus's support for Hong Kong localization and reducing interference with other applications.

In an email, Matos said that he thought many of the user concerns about losing Fcitx boiled down to UI/UX issues, particularly attachment to "things like skins and themes for the UI popups. In this regard I just can say that as far as gnome-shell is (somewhat) themable people will be able to do themes for gnome-shell's IM popups." On the other hand, he added, there are Fcitx features like spell-checking that should be handled desktop-wide, while features like retrieving word lists from an online service "is orthogonal and there's no reason it can't be implemented [in] IBus." It may not be an easy journey, but clearly pulling CJK users into the fold with first-order input support has potential benefits that far outweigh the costs.

Comments (43 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

• Security: Ubuntu details its UEFI secure boot plans; New vulnerabilities in apache, dhcpd, mantis, mediawiki, ...
• Kernel: Kernel Komedians; QR codes for kernel crashes; printk() problems; Tightening security: not for the impatient.
• Distributions: DNF, which may or may not replace Yum; CyanogenMod, Red Hat, UEFI, ...
• Development: Beautiful Soup; KDE 4.9rc1, Firefox for Android, the Linux graphics stack, ...
• Announcements: GNOME Foundation board, Outsource vs open source, Why We're Fighting for an Open Cloud, ...

Next page: Security>>