Bdale Garbee has been involved in free software, particularly Debian, for a
long time, but he's
also been involved in the "corporate side" of open source at HP for quite some
time as well. That
gives him a good perspective on the interface between companies and free
software projects. In a bit of a reprise of a talk that goes back to 2003
he spoke at LinuxCon Japan on "The Business of Contribution"—how
companies and free software projects can benefit from each other.
There are multiple ways that someone can become involved in the free and
open source software (FOSS) ecosystem. They range from those who take the
code and use it in some way all the way up to those that lead and manage a
project or subsystem. Garbee said that he hoped to convince attendees to
move their way up to higher levels of engagement in FOSS.
In general FOSS projects have certain things in common. They are developed
and supported by the community, there is no single company in charge, and
there is a wide range of contributions that come from people with a variety
of interests, abilities, and motivations. The users of the software have
flexibility in how they acquire support; they can become a developer or pay
someone to do development for them. This puts them in control of their own
destiny, which stands in contrast to license arrangements where users just
The key to FOSS licensing is that "people we don't even know are empowered
to build things we can't even imagine", Garbee said. Everyone can take
advantage of those newly built things, both individuals and companies.
Business and community
Something that is probably immediately obvious, he said, are that the
expectations are somewhat different for FOSS developers and
companies. It is a bit difficult to characterize what FOSS developers are
looking for because each project and developer is different, but there are
some common elements.
Developers often want to scratch an itch and
solve a problem that they are having. They also are generally not working
to a particular schedule, something that is particularly true with Debian
for example, he said. FOSS developers are looking for the fun that comes
from a collaborative development project. Lastly, FOSS development is a
means to develop and maintain a personal reputation, which is something
that can be hard for companies to understand when hiring them.
On the other side of the coin, companies need to make money. In fact, in
the US, companies are legally required to make decisions based on their
revenue generation possibilities. Companies also must grow and expand to
provide the financial return that investors expect. That means that
companies want to be able to differentiate their products from their
competitors' products. Corporate reputation is also important, but there
are so many ways to lower a company's reputation, and seemingly few ways to
increase it, which makes doing new things risky, he said.
But there is good news in that there is something that the community and
companies can agree on: positive user experience. For the community, that
will increase the number of people who want to use and benefit from what it
has made. For companies, a great user
experience will help motivate customers to want to give them money.
Licenses and business models
FOSS licenses have their basis in copyright law, which differs somewhat in
various jurisdictions, but there is a lot of commonality. The choice of which
license to use for a given project is made by the founder(s) and those that
join the project later are explicitly accepting those terms for their
contributions. That "very special and unique right" to set the license is
an important decision as it can have a "really profound impact" on the
nature of the community that forms. In addition, a particular choice of
license can sometimes cause new projects to form using different license terms.
There are lots of different open source software licenses
available—60 or so—which
is a bit unfortunate. But, they break down into two basic types:
reciprocal or permissive. Reciprocal licenses require that one pass on the
same rights to the code to those you distribute it to, while permissive
licenses do not require that. The reality is that each license type has
its place, Garbee said. It is important for project founders to carefully
consider the license they choose, while it is equally important for
contributors and users to understand the license that is being used.
It is vital to recognize that "open source" is not a business model, it
is, instead, a process for development. One can use open source to enable
the sale of hardware—this is what HP does—by ensuring that
Linux runs well on that hardware. That is one possible business model, but
there are others.
Selling services, like training or support, is a common example of a
business model around FOSS. One could also give the source away, but
charge for access to the binaries. That may sound strange, but it is more
or less what the enterprise Linux distributions do, he said. Another model
is to open up the core of some large software system, perhaps a framework
and a few sample modules, and build a business around selling add-ons.
That model can work, but it sometimes becomes something of an arms race as
FOSS developers create and contribute add-ons that compete with the ones
that are sold.
Working with FOSS gives a company the "opportunity to harness the
creativity of a very diverse community". There is no company on earth, or
even a collection of companies, that can bring to bear the talents that
exist in a community like that surrounding the Linux kernel. Finding a way
to work with FOSS communities can be an enormous boon for a company.
Deciding to contribute
There are multiple benefits to a company for participating in open source.
For one, the software has "zero marginal cost", which means that a second
(or third ...)
copy costs nothing extra. The web provides a low-cost collaboration and
distribution mechanism, which also reduces costs. Most of tools are also
available as free software, so building a product is less costly. There is
an enormous amount of existing code in the form of libraries and tools that
can be used to build a product more cheaply and with a faster time to market.
There are multiple levels of engagement with FOSS, starting with taking the
code and running it. Those who do so are valid members of the FOSS
community. Going beyond that and monitoring the development or user
mailing lists makes you a bit more engaged. The next step might be to find
and fix a bug or to answer a question on the list. Each step up gives a
person more influence in the direction of the project.
There are more and less efficient ways to use open source in a product, he
said. It is common that one or more OSS components will require some change
before they can be used, and that is "completely legally OK". If there is a
reciprocal license, the changes must be distributed, but that's not a big
deal. The problem occurs when it is time to put out a new version of the
product. If the changes have not gone upstream,
they will need to be ported to any new version of that upstream project.
On the other hand, if the changes have been contributed upstream—and
shepherded through the process of getting them merged—the features
and bug fixes needed will be there in the next versions. That means that
the work that was done once won't have to be repeated each time a new
version of the product (which may require a new version of the upstream
project) is built. That doesn't mean there is no work that needs to be
done, as the new version may have new bugs that need to be addressed, but
it is much more efficient to get changes upstream.
For the kernel, things like device drivers will be kept up to date as the
internals of the kernel change over time. The kernel is a good example
because it is such a fast moving and dynamic body of code, but the
situation exists in other projects as well.
Choosing to use FOSS is a "first step to making the best use of resources",
Garbee said. That allows a company to focus on differentiating its
product. But maximizing community involvement helps to grow the body of
technology that's available to be used in future products, without having
to reinvest over and over again.
Using FOSS does not mean that a company
has to give everything away for free. But it doesn't mean that the
company gets everything it wants for free either. There will always be a
small piece that the company will need to invest in, but in spite of that,
working with FOSS projects is an "incredibly powerful model".
Getting more involved with a project can bring other advantages too. One
can help influence the project's direction and learn about enhancements
that others are working on. Learning about security vulnerabilities and other problems in
the code earlier will also be useful. It will give insight into who is
contributing to the project, which may lead to interesting information on
what competitors are up to. Because of that, Garbee is surprised by the
email he sometimes gets from people working at other companies who didn't
expect some announcement that HP has made. Had they been paying attention
to the projects that HP was working on, the announcement would probably
have been less surprising.
One way to experiment with FOSS is to work with partners on a project
without really committing the company to doing the work, but that will not
equity". The technical accomplishments and the reputation that the project builds
will not accrue to the company (and instead to the partner). A better way
is to have company employees
who work directly on FOSS projects. The expertise that they gain can be
reapplied to other projects, and the company gets to help set the direction
for the project. In addition, the company can take credit for the work it
One important thing to recognize, however, is that reputation is built by
individuals. Getting code accepted involves a
combination of technical prowess and reputation within the community. In
that case, the reputation that matters is that of the individual. Linus
Torvalds has said that he doesn't really know the corporate
affiliation of most kernel developers. In order for the company to be
successful in the community, it must allow individuals to gain reputation
in that community. It is important for managers to understand this, he
said, and to make it a part of the career development of the employee.
Garbee noted that he once was talking to a peer at a competitor who asked
about how he should review a particular employee. That person had been tasked
with getting some code upstream, but ran into a competing proposal. The
employee then made sure that all of the company's needs were met in the
alternative, which eventually was merged. The peer was wondering
whether the employee should be rewarded for that, and Garbee's response was
"give the guy a raise". He recognized that the alternative proposal was
better, dealt with his ego, and ensured that the company's requirements
were met. He achieved the desired result, rather than get his own code
upstream, which fully met the objective. That is something that many
managers have a hard time understanding, he said.
Does this really work?
Garbee then described HP's experiences with FOSS as something of a
justification for what he had been saying. In the last ten years or so, HP
has accomplished a great deal by trying to engage with FOSS projects "as
best we can". That is one of the things that has led HP to be the world's
largest IT company by many
measures, he said.
One of the more significant recent decisions that HP made with regard to
FOSS was for the HP Public Cloud, which is based on OpenStack and Ubuntu
LTS. The company has made a "significant commitment" of HP
resources to OpenStack. It is now one of the largest technical
contributors to the project and hosts its build farm. In addition, HP is
helping with the transition of OpenStack to a "foundation model", similar
to that of Eclipse or Apache.
HP has recently completed an analysis of all of its products and found that
the majority contain at least some FOSS. Any of those that do have FOSS
components are increasing that percentage rapidly. It is the most
efficient use of HP's software engineering effort, he said. All across HP,
more and more direct engagement with FOSS projects, which is helping to
make the company more successful.
As a consequence of the talk, Garbee said that he hoped the audience would
find that becoming more directly engaged with FOSS projects made sense for
them and their companies. He ended with a famous quote from Margaret Mead:
Never doubt that a small group of thoughtful,
committed citizens can change the world.
Indeed, it is the only thing that ever has.
That is, he said, a strong statement of how things work in open source. If
someday you look around and wonder why it is that something isn't fixed,
remember that you are someone.
Garbee's talk was clearly aimed at the mostly Asian audience, and tried to
assist those present with arguments to make to their management about the
benefits of working more closely with FOSS projects. The arguments are
useful for anyone trying to get their company more involved in FOSS
projects, of course, but Asian companies have traditionally been less
upstream projects. He also slipped some
rocket slides into the talk, which, beyond bringing some of his hobby into
view, also gave him an opportunity to show a more personal connection to
One of those slides showed him carrying a rather large rocket to the
launching pad (shown at right). That rocket was subsequently lost after reaching Mach 1.3
and attaining 5000m above the ground because of a
malfunctioning closed-source altimeter. That led him and Keith Packard into
a collaboration to create open hardware and software for rocket telemetry
and control, which ultimately helps him "not lose rockets". So, not only
is open source good for companies, it's good for rockets—and other hobbies
Comments (3 posted)
Proprietary kernel modules have always had an unclear status in the kernel
development community. Most (but not all) developers seem to agree that it is
possible to make a module that is not a derived product of the kernel
itself (and, thus, not subject to the requirements of the GPL); the Andrew
filesystem, ported from Unix in the early days, has been cited as an example
of this kind of module. Most developers also seem to agree that many other
types of modules cannot be written as an independent work. There is far
less agreement on where the line should be drawn, but there is a consensus
around the idea that modules should not lie about their licensing status to
gain closer access to kernel internals. The story of a company that was
recently caught shipping that sort of dishonest module shows the kind of
reaction that can be expected—and why closed-source modules are a bad idea
Loadable kernel modules can only access kernel symbols that have been
explicitly exported for that use. Two types of declarations are used to
export these symbols. EXPORT_SYMBOL_GPL() is an indication that
any module using the symbol is reaching so deeply into the kernel that it
is, by necessity, a derived product of the kernel. Instead,
EXPORT_SYMBOL() is used for symbols that only might
indicate a derived-product status; it is also used for symbols that were
already exported before the GPL-only mechanism was added.
The kernel's module loader includes an enforcement mechanism that prevents
any module without a GPL-compatible license from accessing explicitly
GPL-only symbols. To function, this mechanism clearly must know what the
license for a specific module is. That is the weak point of the entire
scheme: the kernel uses the honor system. Loadable modules include a
special declaration that tell the kernel which license applies; the module
loader looks at that declaration and decides whether the module will have
access to GPL-only symbols or not. If the module lies about its license,
the kernel will normally trust it.
That is where CloudLinux comes in.
This company offers a CentOS-like distribution aimed at the needs of cloud
computing providers. Essentially, CloudLinux has taken a RHEL clone,
added its own container mechanism, and created a business around providing
updates and support. The container mechanism is at least partially
implemented by the company's "lve" kernel module; lve includes an access
control and resource limit mechanism that, seemingly, is implemented
independently of the kernel's control group subsystem. It can place limits
on CPU usage, I/O bandwidth usage, and memory consumption, among other
things; all this is managed through an
administrator-friendly XML file.
Matthew Garrett recently became aware that, while the lve module claims to
be GPL-licensed, there is not any source available for it; he responded
with a patch causing the kernel to
recognize and blacklist this module. Since lve uses GPL-only symbols, this
change will prevent it from loading, essentially breaking it. This change
has yet to be pushed into the mainline, but it has been accepted by the modules maintainer (Rusty
Russell) and marked for stable updates as well. Unless something changes,
it will soon become impossible to load the lve module into any current
The immediate effect of this change on CloudLinux is likely to be limited.
Chances are that few users download the binary lve module and load it into
their own kernels; almost all lve users are probably running the entire
CloudLinux distribution. CloudLinux could easily patch its own kernel to
remove the check for the lve module and restore the previous
functionality. But such a move would almost certainly be noticed, and
there is a distinct possibility that one or more kernel developers would
react in an unpleasant way. It would not be an advisable course of action.
What CloudLinux did instead was to apologize and plead for time:
We planned to close source the module, and we will do it later
on. Yet, it looks like one of our developers missed the point --
and did things incorrectly.
Please, give us two-three weeks to straighten things out. By the
end of three weeks I plan to have source RPMs with the GPLed
version of the modules available in our source repositories.
Later on we will have new module that is not GPL released.
This response was not entirely well received; it is not at all clear why
CloudLinux is unable to immediately ship the source corresponding to the
binary module it is distributing. That said, the community will
probably limit itself to grumbling as long as CloudLinux follows
through and comes back into compliance with the kernel's licensing. Nobody
likes a delay in a source release, but short delays are usually tolerated,
So the licensing part of this episode is, probably, resolved, but
it is also interesting to look at why the lve module needs GPL-only symbols
in the first place.
The specific symbols it needs relate to the creation of binary attributes
(essentially unstructured sysfs files) in the sysfs virtual filesystem.
Much of the internal sysfs interface is GPL-only, and the functions for the
management of binary attributes (which are generally discouraged) are
certainly so. It turns out that lve creates binary attributes for a reason that would certainly never get
through a proper review:
We do a "hack", which is not a pretty one, populating /sys with
.htaccess files. This is really needed only by shared hosters,
where one of the end users on the server, could be a hacker and
could create symlinks that would later be followed by apache to
read privileged information.
As a comprehensive security solution, this implementation leaves just a
little bit to be desired.
CloudLinux is trying to reduce the amount of damage that could be done
should an attacker successfully compromise a daemon running on the system.
Kernel developers have been working for years to
mitigate just this sort of threat; the results of that work take the form
of security modules, namespaces, and more. Had CloudLinux done its work in
the open, it would certainly have been directed toward solutions that have
support from others and years of experience in actual use. Solutions, in
other words, that might actually deliver some additional security.
Instead, they got a one-off "not pretty hack" that could never get upstream.
If CloudLinux follows through on its plan to close the module entirely, it
will have to cease use of GPL-only symbols and, thus, will not be able to
implement this particular mechanism. That may be enough to convince kernel
developers to leave it alone. But it should leave others wondering what
other surprises may be lurking inside—surprises that cannot be
investigated, since the source for the module is not available.
There may well be a viable business in the creation of a well-supported
distribution aimed at the needs of cloud providers. With solid security
support and some top-tier user-space management tools, a company like
CloudLinux could well produce an offering that others are willing to pay
for. But the most robust and trustworthy server distribution will come
from using (and improving) the work that many developers have done for
years to create a useful containers implementation. It is hard to imagine
that a proprietary solution put together by a small company can work as
well. Compared to that, the risk of licensing troubles seems like a case
of adding insult to injury.
Comments (17 posted)
Those of us who type in Latin characters may easily
overlook what it takes to get text into windows or command lines in
other writing systems. Entry of characters not found on one's keyboard
requires the use of an input method (IM)
which turns multiple keystrokes into characters. There are plenty of capable projects, but they
often lack deep integration into the desktop environment or widget
toolkit. In April, GNOME developer Rui Matos proposed
a feature for the upcoming GNOME 3.6 release that would integrate the
IBus framework into the
core GNOME desktop, tackling this precise challenge. IBus is a
framework that allows the user to select — and switch between — multiple IMs. The plan spawned considerable debate, not
only on the merits of IBus, but on the wisdom of tightly integrating a
single component into the desktop environment. Complicating matters
is the divide between the bulk of the GNOME developer community and
those users who depend on input methods, primarily from the
Chinese-Japanese-Korean (CJK) language communities.
At the heart of the issue is how CJK users input text. In alphabetic
or syllabic writing systems (including European, North Indic, and Middle
Eastern), there is a known mapping between the pronunciation of a
word and its written representation. In the logographic writing
systems of CJK, where the strokes of the character do not represent sounds or other subdivisions of the word, users make use of an
IM instead. There are phonetic IMs such as Pinyin (in which the
user types in the romanization of a word on the keyboard and the IM
substitutes the correct character, or logogram), shape-based IMs like Cangjie (which
decompose the logogram in a strict order), and many hybrids. In most
cases, good dictionaries or tables are required, plus word-prediction,
spell-checking, and other add-on features to cope with homophones and other
tricky bits. Mobile phone users got a taste of the IM experience
and other numeric-keypad predictive text systems, which have largely
For everyday typing, the challenge is greater because no one input
method is inherently superior: if you do not know how to pronounce an
unfamiliar word, a phonetic method is no use, and switching to
shape-based method makes sense. On the other hand, typing a word
that you hear but have not seen written down demands a phonetic
method. Likewise, regional accents can have very different
pronunciations for the same logogram, but there is often more than one
way to decompose a logogram by shape — not to mention the
problem of writing reforms like Simplified Chinese, which is not
simplified uniformly between mainland Chinese hanzi and Japanese
kanji. Thus, almost no IM can be relied upon to the
exclusion of all others. Throw in open source developers' penchant
for reinventing the wheel to scratch their own itches, and the result
is multiple IM frameworks, each of which
can load individual IMs as the user chooses.
Supporting all of these frameworks is a challenge, one that Matos
(who works mostly on gnome-control-center, GNOME Shell, and other
desktop components) felt detracted from GNOME's ability to provide a
consistent desktop experience and left CJK users a step behind those
users whose writing system works out-of-the-box. As he explained
on the desktop-devel list, IM framework proliferation reduces the odds
that any of the frameworks will get enough testing to be robust, and
it greatly complicates bug triage for GNOME. Plus, many of the
popular IM frameworks also attempt to be cross-desktop, which makes
integrating them seamlessly into GNOME difficult. Their visual
appearance does not match, they conflict with core GNOME settings like
XKB configuration and key bindings, and they cannot be automatically
started (relying either on user interaction or shell scripts to
launch). The fix he proposed is to take one IM framework, tie it in
more directly to GNOME (including adding or extending GNOME APIs where
needed), and make sure that it works for the majority of users.
Specifically, Matos said, GNOME has three requirements to properly
integrate an IM framework with the desktop: a GTK+ module, a D-Bus API
that can enumerate, activate, and configure installed IMs, and a D-Bus
API on GNOME's side that the framework can use to draw predictive-text
pop-up windows. Right now, he said, only IBus meets all of the
requirements. Owen Taylor added
some additional requirements to integrate the framework with GNOME
accessibility and configuration standards, and a quality set of
existing IMs. The IBus team expressed a willingness to work with
GNOME, which was also critical.
Feedback from the CJK community
However, the plan elicited strong reactions from some members of the
CJK user community. Marguerite Su replied
that few if any CJK users like IBus; with most preferring rival
framework Fcitx out of the existing
options available on
desktop Linux. Others chimed in with criticism of IBus. The
complaints included several overlapping issues, including
customization features, IM engine quality, and speed. Su elaborated
on the feature disparity, saying that some users want to customize
shortcuts, rearrange the word-completion suggestions, or fetch new
dictionary words from Internet servers. But trading IBus for Fcitx is
not the simple solution. IBus works well for Simplified Chinese, but not
Traditional Chinese, while Fcitx is focused largely on Chinese and has less robust
support for other languages, including Japanese (although it has plans
to improve its Japanese support). Chinese users are greater in
number, which might argue in favor of Fcitx, but surely the best
solution would be to find a framework that serves everyone. Support
for writing systems unrelated to the CJK family (such as Tibetan or
Thai) is weaker in general across the IM frameworks.
But over the course of the sometimes-heated list discussion, the IBus critics
proposed not simply adopting Fcitx outright, but keeping IM frameworks
a user-controlled, pluggable option. For example, Liang Suilong, who argued
that IBus is laggy, still advocated building an "IM framework
framework" for GNOME, thus allowing users to choose the framework they
preferred. But that idea was not well-received by GNOME developers.
Tomas Frydrych summed
up the two sides of the divide:
Rather a long discussion over IBus, but it seems
to more or less boil down to two voices and this:
Gnome developers: we want tighter IM integration and simpler UI in the
name of better UX, and are looking at IBus as the underlying technology,
Users: IBus has poor support for CJK input and a history of not
addressing these problems.
As others pointed out on the list, the latter point was not
accurate; IBus does support CJK, but not all users are happy with its
IM implementations or auxiliary features. Complicating matters more is
the cultural divide between the groups. Weng Xuetian pointed
out that the core GNOME developers are not IM users (most speak
European languages), which makes them unqualified to select the "best"
IM framework to then be used by a large number GNOME users who were
not able to participate in the discussion.
Frameworks, not engines
But the groundswell of IBus criticism was not the final word. Tommy He said
that the debate was too focused on CJK input, to the exclusion of
other writing systems, and moreover that much of the lobbying in favor
of Fcitx focused on the quality of the various IM engines, rather than
the frameworks themselves. After all, if Fcitx can add a new and
improved Japanese IM, then IBus could add one for Traditional Chinese.
Admittedly, the framework-versus-IM debate is a difficult one to pin
down. On the one hand, Fcitx proponents enjoy its more flexible
plugin system, which allows user-defined macros and other daily-use
features. But that same flexibility could make it more difficult to
integrate with GNOME's existing language settings and preferences,
when providing a better first-run experience is explicitly one of the
goals. As Owen Taylor observed,
Fcitx generates its configuration GUI on-the-fly, which is a technique
GNOME finds problematic and tries to avoid.
On May 14, Taylor attempted to re-center
the discussion to the framework question, arguing
that GNOME needs to pick one framework, then make it work "as
well as we can possibly make it for all users. Multiple partially
working frameworks are not a substitute." When the
IBus-versus-Fcitx debate briefly resurfaced, Bastien Nocera advocated
starting the IBus integration work as soon as possible, then inviting
Fcitx to bring its own code demonstrating that it could fit the role
better. "A lot of the code should be reusable, and you can
show how much more awesome and complete your favourite IMF is."
When it looked like the core GNOME developers had made their final
decision to integrate IBus, a few IBus detractors asked the team to
postpone the decision for further discussion, but without success.
his opinion that it would be better to pick an imperfect IM framework
and replace it in later releases than to do nothing:
If we choose to merge integration based on IBus (because of a variety of
reasons), then two things can happen:
- Developers of other Input Frameworks can start creating patches to the
upstream GNOME to provide a better integration than the default choice.
- They choose to start working on the selected IMF because it's the
- They choose to concentrate on other desktops
In all cases, the implementation will evolve, and the integration will
get better. I don't want to have the choice between 2 equally badly
integrated IMFs for GNOME.
But Jasper St. Pierre compared
the decision to other components, like audio. GNOME does not attempt
to support ESD and OSS in addition to PulseAudio due to limited
resources, he said, and GNOME's choice of IM frameworks will not force
distributions to follow suit. "I doubt we have the resources to
support both IBus and FCITX, and provide a good experience for
both. Individual distributions may, but that's their call, not
The Fcitx lobby may not be satisfied with that response, but it looks
like IBus is here to stay for the 3.6 development cycle at the very
least. The GNOME wiki page
lists seven tracker bugs to follow the integration with GNOME Shell,
GNOME control center, and other components. There is still plenty of
integration and debugging work to be done, including improving IBus's
for Hong Kong localization and reducing interference
with other applications.
In an email, Matos said that he thought many of the user concerns
about losing Fcitx boiled down to UI/UX issues, particularly
attachment to "things like skins and themes for the UI
popups. In this regard I just can say that as far as gnome-shell is
(somewhat) themable people will be able to do themes for gnome-shell's
IM popups." On the other hand, he added, there are Fcitx
features like spell-checking that should be handled desktop-wide,
while features like retrieving word lists from an online service
"is orthogonal and there's no reason it can't be implemented [in]
It may not be an easy journey, but clearly
pulling CJK users into the fold with first-order input support has
potential benefits that far outweigh the costs.
Comments (43 posted)
Page editor: Jonathan Corbet
Next page: Security>>