As has seemingly become traditional, your editor moderated a panel of kernel developers (James Bottomley, Christoph Hellwig, Greg Kroah-Hartman, and Andrew Morton, this year). We discussed a wide range of topics, but the subject that caught the attention of the wider world was the "graybeards" question. As a result, we've been treated to a number of lengthy discussions on just why the kernel is no longer attracting energetic young developers.
The only problem is: the kernel doesn't really have that problem. Every three-month development cycle involves well over 1000 developers, a substantial portion of whom are first-time contributors. Nearly 20% of these contributors are working on their own time, because they want to. There does not appear to be a problem attracting developers to the kernel project; indeed, if anything, the problem could be the reverse: the kernel is such an attractive project that it gets an order of magnitude more developers than just about anything else. Your editor has heard it said in the past that Linux as a whole might be better off if some of those developers were to devote a bit more time to user-space projects, most of which would be delighted to have them.
The question the panel discussed, instead, had to do with the graying of the ranks of the kernel's primary subsystem maintainers. Many of those developers wandered into kernel development in the 1990's, when there were opportunities everywhere. A decade or more later, those developers are still there; James Bottomley suggested they may stay in place until they start dying off. Entrenched graybeards at high levels could, someday, start to act as a brake on kernel development as a whole. That does not appear to be a problem now; it's just something to watch out for in the future.
Andrew Morton raised an interesting related issue: as the core developers gain experience and confidence, they are happy to put code of increasing complexity into the kernel. That could indeed make it harder for new developers to come up to speed; it might also not bode well for the long-term maintainability of the kernel in general.
Dan Frye's keynote reflecting on IBM's 10+ years of experience with Linux was easily one of the best of the day. IBM's experience has certainly not been 100% smooth sailing; there were a lot of mistakes made along the way. As Dan put it, it is relatively easy for a company to form a community around itself, but it's much harder - and more valuable - to join an established community under somebody else's control.
A number of lessons learned were offered, starting with an encouragement to get projects out into the community early and to avoid closed-door communications. IBM discovered the hard way that dumping large blocks of completed code into the kernel community was not going to be successful. The community must be involved earlier than that. To help in that direction, IBM prohibited the use of internal communications for many projects, forcing developers to have their discussions in public forums.
Beyond that, companies need to demonstrate an ongoing commitment to the community - no drive-by submissions. Developers need to remain immersed in the community in order to build the expertise and respect needed to get things done. Companies, Dan says, should manage their developers, but they should not attempt to manage the maintainers those developers are working with. In general, influence in the project should be earned through expertise, skills, and a history of engagement - and not through attempts to control things directly.
One interesting final point is that what matters is results, not whose code gets merged. To that end, IBM has reworked things internally to reward developers who have managed to push things forward, even if somebody else's code gets merged in the end. This is an important attitude which should become more widely adopted; it certainly leads to a more team-oriented and productive kernel community in the long term.
Chris DiBona's talk on Google and the community was always going to be a bit of a hard sell; Greg Kroah-Hartman's discussion of Android and the community had happened just two days before. Additionally, Chris was scheduled last in the day, immediately after Josh Berkus gave a high-energy version of his how to destroy your community talk. So perhaps Chris cannot be blamed for his decision to dump his own slides and, instead, give a talk to Josh's slides - running backward.
Chris did eventually move on to his real talk. There was discussion of Google's contributions to the community, including 915 projects which have been released so far and something like 200 people creating patches at any given time. There are over 300,000 projects on code.google.com now. Google has also supported the community by providing infrastructure to sites like kernel.org and, of course, the Summer of Code program.
In short: Chris doesn't think that Google has much to apologize for; indeed, the contrary is true. This extends to the Android situation, which, Chris says, he's not really unhappy with. The targeted users for Android are different, and, in any case, it always takes a long time to get a real community going. That said, more of the Android code should indeed get into the mainline, and Google should be doing a better job. Part of the problem, it seems, is finding "masochists" who are willing to do the work of trying to upstream the code.
The truth of the matter is that Chris's talk failed to satisfy a lot of people; much of it came off as "Google is doing good stuff, we know best, we're successful, and we're going to continue doing things the same way." Starting with his playing with Josh's slides, Chris gave the impression that he wasn't taking the community's concerns entirely seriously.
On the other hand, the announcement at the end of his talk that Google was giving a Nexus One phone to every attendee almost certainly served to mute the criticism somewhat.
Outside of the sessions, your editor had the opportunity to talk with some of Google's Android kernel developers; the folks actually doing the work have a bit of a different take on things. They are working flat-out to create some very nice Linux-based products, and they are being successful at it, but they are in the bind that is familiar to so many embedded systems developers: the product cycles are so short and the deadlines are so tight that there just isn't time to spend on getting code upstream. That said, they are trying and intend to try harder. We are starting to see some results; for example, the NVIDIA Tegra architecture code has just been posted for review and merging.
The Android developers seem to feel that they have been singled out for a level of criticism which few other embedded vendors - including those demonstrating much worse community behavior - have to deal with. It can be a dismaying and demotivating thing. Your editor would like to suggest that, at this point, the Android developers have heard and understood the message that the community has tried to communicate to them. There is a good chance that things will start to get better in the near future. Perhaps it's time to back off a little bit and focus on helping them to get their code merged when they get a chance to submit it.
As promised, this article has barely scratched the surface of what happened at the 2010 Collaboration Summit. In particular, the large presence of MeeGo (which had a separate, day-long session dedicated to it) has been passed over, though some of that will be made up for in a separate article. In general, it was a good collection of people who do not always get a chance to mingle in the same hallway, all backed by the Linux Foundation's seamless "it all just works" organization. Improved collaboration within our community should indeed be the result of this event.
The Linux Foundation's Collaboration Summit (LFCS) is focused on, well, collaboration, so it is no surprise that a recent high-profile collaborative effort in the Linux world, MeeGo, had a strong presence at the conference. Both sides of the merger of Moblin and Maemo, Intel and Nokia, had representatives giving keynote speeches about MeeGo and how it intends to interact with the community. Since the project is hosted by the foundation, it makes sense that it would devote a good portion of LFCS—a day-long MeeGo track in addition to the keynotes—to the mobile distribution. The focus of both speakers was on developing MeeGo in the open and working closely with upstream projects, rather than targeting MeeGo itself.
Ari Jaaksi, Nokia's VP for Maemo devices and MeeGo operations, spoke first, which he saw as an advantage because Intel's Imad Sousou would be sure to correct anything he said "wrong". The goal of the MeeGo project is to "provide industry with an open platform" for various kinds of devices. Both companies have been working on mobile distributions, which means that they "integrate the same components multiple times", and that is "stupid", Jaaksi said. That is one of the main ideas behind the merger.
Nokia has been working with Linux and free software for a number of years, since 2002 or 2003, and that it has been a "learning exercise". It "made a lot of mistakes" but tried to work within the community by participating with many different projects. Its early realization that it needed to be part of the community, and not "just use the code", was important.
But the integration process, where the various components that made up Maemo were built and collected into a release, was not open to community involvement. That is something that will change for MeeGo: "We are going to build the MeeGo platform in the open". It is a "huge change" that is going on "right now, as I speak". The idea of doing that "may seem trivial" to LFCS attendees, he said, "but it is a big deal with us".
Sousou, who is the director of Intel's Open Source Technology Center, echoed that idea. Working in the open will make collaboration easier, but "you will see the messes, and we are OK with that". One of the keys to making that work will be to focus on the upstream projects, he said. It took Intel "some time to figure it out", but downstream projects must "contribute and use the open source model".
There are "hundreds" of Intel engineers working on MeeGo, Sousou said, but most of the work is not actually in MeeGo itself. "It's happening upstream", at kernel.org, freedesktop.org, and others. He doesn't want to see kernel patches, for example, submitted to MeeGo, "submit it upstream". It's all about "working with upstream and contributing upstream — there is nothing more".
Both speakers talked about governing MeeGo in the open, with steering committee meetings on IRC. Jaaksi notes that there is still some adjustment that Nokia needs to make. He gets email from other employees about seeing MeeGo roadmap plans on the Internet; they are worried about competitors getting that kind of formerly secret information. He tells them: "Yes, that's how we do things".
Jaaksi notes that Palm had gotten products out earlier than Nokia, "with our code", and that was "not their fault, [but] our fault" by being too slow to market. Google has also used Maemo code, and "we hope to use theirs". A concern is that MeeGo will give competitors an advantage, but he believes that it is the companies which participate in the project that will see the most benefit. That concern may not be relevant for most of the people in attendance, he said, but within Nokia, there is a question on how to differentiate itself.
Sousou listed oFono and ConnMan as two projects where the two companies had already worked together. For MeeGo, they complement each other well, Jaaksi said. Nokia brings experience working with mobile handsets, ARM, and the phone companies, while Intel has "so much knowledge about the Linux kernel". Both have good teams that "know how to work in open source and combine open source with business". Choosing the "best of breed" components from Moblin and Maemo—or elsewhere—for MeeGo is something that both speakers stressed—there is a general sense that the project is trying to avoid "turf wars".
But it's not just Intel and Nokia, as the MeeGo project is looking for more contributors, which is another thing that both speakers emphasized. Because of their close working relationship, it was relatively straightforward to merge Maemo and Moblin into one project. They didn't bring in other companies at the start, because, in Jaaksi's opinion, "it would have taken too much time" to get agreement with more companies. He said that MeeGo wants to "demonstrate that it is an open source project" that others can participate in. He listed multiple companies that have become involved since the MeeGo announcement, including hardware vendors, Linux distributors, embedded Linux vendors, and so on.
The two main participants have decided on a blueprint of the architecture, which includes Qt as a platform for application developers, but the design of the system is an "ongoing process that we invite people to participate in", Jaaksi said. "Now is the time to join MeeGo", he said, and there is much to be done, but there is little risk for others because they have made a commitment to do things in the open. Both stressed that there is a simple, open governance model.
But, as an audience member pointed out, there is a veto power that Intel and Nokia have over the project. The audience member wondered if a community can still be built around that veto. Jaaksi responded that things "will be fixed if we need to fix them" and that changes will be made for anything that becomes an obstacle. Currently, MeeGo is focused on getting more people involved and having "simple governance". Earlier in his talk, he said that the governance needed to stay out of the way to maintain the speed of development.
There are 200-300 participants in the IRC meetings, Sousou said, and anyone can get involved. "If you contribute, you can help make decisions", Jaaksi said, but MeeGo will "make some mistakes going forward".
The veto power for Nokia and Intel was one of the things that LFCS participants grumbled about in the "hallway track". There is concern that community input will be ignored. One area that is particularly sensitive is the choice of RPM as the mandatory package format for MeeGo-branded devices. Debian/Ubuntu-oriented developers felt slighted by that requirement, and there seems to be no room to change that decision, which gave rise to concerns about governance. Assuming it wants only one package format, there is no "good" choice for the project as either plausible choice would irritate some.
Beyond that, attendees seemed interested, some even excited, by the prospects of MeeGo. Some consolidation in the mobile Linux arena is to be welcomed. In the end, though, it will be the MeeGo devices that will largely decide its fate. While Moblin and Maemo are available, neither has gained the widespread device availability that Android is starting to enjoy. Most participants seemed to be taking a "wait and see" approach, with the sense that many will be watching developments fairly closely.
Marketing isn't the first word that one associates with the Linux community, but it is a necessary activity for those who wish to bring new users into the fold (and perhaps make a buck at the same time). The recent Ubuntu rebrand, and the subsequent media attention surrounding it, provides an opportunity to consider the larger question of branding for Linux distributions and open source projects.
The brand of a project includes the name, theming, artwork, fonts, logos, and even audio cues for a project. Community projects, especially those with a primary sponsor, have a unique set of challenges and considerations when it comes to branding. Companies promoting proprietary projects can pour millions into building cohesive brands, and have the ability to dictate branding decisions from the top down. No one at Microsoft need worry about whether branding decisions will affect the community distributions of Microsoft Office, because there aren't any. Nor do the decision makers in Redmond need to worry about offending a community art team when imposing brand changes, or worry about a lack of talent when it comes time to develop and implement branding.
Not so for community distributions and open projects. The tradeoff for strong community involvement is a diverse set of stakeholders that will want to weigh in on branding, but come with considerably less money and talent on tap to develop branding.
Canonical has created a somewhat confusing situation with the Ubuntu brand, because it is the corporate and community brand simultaneously. The company has been criticized for using its community brand, specifically the Ubuntu trademark, for proprietary offerings as well as its community distribution. It has also been a slight source of problems in the past when the community team didn't produce work that lived up to Mark Shuttleworth's standards.
This time around, Canonical brought members of different sub-projects together to work on the rebrand effort for 10.04. This hasn't satisfied everyone, and the decision to re-arrange the window buttons has generated quite a lot of grumbling. Overall, though, Canonical's handling of the rebrand seems to have gone well with its community.
While Canonical pursues a unified brand for Ubuntu, other companies attempt an approach of using a derivative of the corporate brand for the community project, such as was done with openSUSE and with OpenSolaris. This strategy has its merits, but can lead to a number of problems. Coordinating with a corporate marketing communications team can be a bit challenging for a community project. As with a single brand, it means that the commercial and community efforts will occasionally generate friction when the community wishes to go in one direction and the company wishes to go in another.
Coordinating community-generated materials with a corporate art department also leads to unexpected challenges. Community members tend to use, not surprisingly, free tools to generate art. Tools like Inkscape, GIMP, and Evince have come a long way and are capable of generating and displaying professional-looking artwork. They still, however, pose challenges when trying to send artwork to a professional art department that works with proprietary tools — and vice-versa. While Evince, for example, is perfectly capable of displaying most PDFs with reasonable fidelity, it does not handle all output perfectly. When developing art for the openSUSE retail box sets and t-shirts, for example, I still needed to rely on Adobe's reader to ensure the output matched what would be sent to the printer.
Further, tying the community brand and name to the corporate brand has side-effects. The upside is that a successful and healthy project will reflect well on the corporate brand. If the community project is well-known and popular, it has a "halo" effect on the commercial brand and has a marketing benefit for the corporate parent. The downside is that the association runs both ways, and if either the corporate parent or community project takes a drubbing, the other suffers as well.
The downside is that this means the primary sponsor has a vested interest in the community's handling of the brand, in particular trademarks and logos, that are derivative of the primary brand. This means a community may wind up with a slightly more restrictive trademark policy, and requirements to coordinate with the marketing communications team rather than leaving decisions to the community. This is also true of a shared brand, of course. Even when the corporate backer has been relatively hands-off with the community branding, as was the case when I worked with openSUSE, it still introduced some minor problems. For example, SUSE chose a proprietary font (Cholla) for its logo and associated marks, a practice that continued under Novell.
This meant that community members interested in producing art for shows, swag, and derivative projects were out of step with the "official" openSUSE brand. Ultimately it was necessary to create a look-alike called FifthLeg, which has worked out relatively well.
The other end of the spectrum is the Debian Project. With no primary sponsor, Debian's community can develop the brand as it sees fit without interference. But Debian has done relatively little with this freedom. It has an art team and has created a site for community contributed artwork, as well as official and unofficial logos with clear guidelines on how they may be used. But there's little in the way of an organized effort to produce a coherent Debian "brand" beyond logos and related artwork.
A more practical and community friendly approach for sponsored projects may be the one taken by Red Hat with Fedora. By providing a separate brand, but providing at least some corporate support for the development of the branding in conjunction with the community teams, it's possible to avoid the conflicts between commercial and community brands. Fedora has a relatively free hand in creating artwork for the distribution, and most of the guidelines are oriented around practical issues rather than sponsor issues — excepting the prohibition on artwork including hats.
Red Hat's trademark guidelines around the Fedora mark, however, have generated some friction with the community. The company has shown interest in trying to satisfy reasonable complaints with the trademark guidelines, but it is rarely possible to please everyone with a trademark licensing agreement.
When a project rebrands, it also has an impact on all the affiliated downstream projects that make use of its branding. All of the major distributions have one or more derivative projects or "respins" that make use of the core distribution but include significant changes from the standard offering.
This means that when an upstream, like Ubuntu, makes changes they roll downhill to Kubuntu, Xubuntu, and other assorted Ubuntu derivatives. Fedora has its respins, and openSUSE now has quite a few offshoots like the Education project or custom spins of openSUSE generated via SUSE Studio.
Derivatives have technical and legal issues to consider. When a distribution actively seeks spinoffs, it has to ensure that it's reasonably easy to rebrand the distribution. This means providing clear guidelines — as Fedora has — on what is allowed (and what's not) with regards to using trademarks and branding for remixed distributions.
Distributions also need to provide tools and instructions on rebranding. For instance, Fedora provides guidelines on replacing the branding packages for Fedora to ensure that Fedora and Red Hat trademarks are stripped from Spins. The openSUSE Project provides instructions and a tool called Rembrand to create derivatives — though prospective respinners may find it easier to use SUSE Studio to create rebranded openSUSE derivatives.
Another issue crops up for LUGs and advocacy projects like Spread Ubuntu. With the Ubuntu refresh, all of the Loco groups and downstream advocacy projects are left with outdated materials. Projects should tread carefully, and slowly, to reduce issues here.
Distributions not only have to consider the effect of branding on derivative projects, but also have to consider the presentation of their upstream projects. For example, the branding decisions made by distributions have a direct impact on the presentation of GNOME or KDE.
The KDE Project's Aaron Seigo has taken some exception to distributions rebranding KDE and creating "microbrands:"
[...] we have made it hard for people to take notice of what we are doing with the Linux Desktop since none of the brands are identifiable as "belonging to the same thing". Instead we end up with microbrands that nearly no one outside of the server room or the hardcore F/OSS community recognizes.
Many (most?) operating systems bearing KDE packages come with their own logo as the application launcher button, many ship their own icon sets or their own (for branding purposes) customizations of the default icon set, many ship their own wallpapers, many change the default window borders or widget themes.
This is even more unfortunate because there is a scarcity of quality artists in the F/OSS world, and when each distribution or project gobbles up one of them to work exclusively on their own mojo ... we just divide and conquer ourselves.
While this may make their individual believers-in-the-cause users happy and may make corporate management feel they are getting some good corporate marketing out there with that happy little logo in the bottom left hand corner ... this is obviously inspired by the "old way" of doing things that is centered around corporate balkanization of the consumer space: flags or fruits, right? (Microsoft and Apple .. :) We need to rise above that and consider the long term benefits of the entire ecosystem because each of our projects thrives or diminishes in step with it.
What KDE has done in this case is to propose a service to work with distributions to customize some elements to achieve a shared visual identity. For instance, the openSUSE 11.2 release included work from Nuno Pinheiro of KDE to provide a shared theme.
Ultimately, one has to wonder how much branding actually affects open source adoption. While Ubuntu received a great deal of media attention surrounding its rebranding for the 10.04 release, it was only because Ubuntu is already popular and well-received — thus making the rebrand effort noteworthy even in mainstream IT press.
One might argue that some small amount of Ubuntu's success stems from previous branding efforts, but very little. What has proven more effective for Ubuntu is the distribution itself and an effective marketing campaign beyond branding. ShipIt to put the distribution in the hands of, literally, millions of users and a motivated community of advocates that have provided word-of-mouth marketing for the distribution.
This isn't to say the look and feel of projects is unimportant. Some have argued, persuasively, that pretty is a feature. Clearly if a project is gunning for mainstream acceptance it has to be attractive enough to stand side-by-side with proprietary software. But it would be a mistake to assume that a distribution's brand is the key to success.
But it can be a key to failure. As Seigo and others have pointed out, ugly software or "micro" branding make gaining traction with a wider audience unlikely. Projects that are looking to succeed with mainstream audiences will need to pay attention to branding and find ways to harness the relatively small number of talented designers that are interested in working with open projects, or find new ways to attract them.fill out the survey. There is some more information on the media kit project there as well. Please take the time to tell us a bit about yourself; doing so will help us provide a more pleasant and successful LWN for everybody.
The LWN site code does not allow for comments on surveys directly, and we noticed that some of you had thoughts about it. Thanks for the email, but you can now also comment below. We are aware that some of the survey questions can have multiple interpretations or don't cover every situation and for that we apologize.
Page editor: Jonathan Corbet
Copyright © 2010, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds