Leading items

Updates from Linaro

Over the last month or so, I have sat in on a few different talks about Linaro and the work that it is doing to improve the ARM Linux landscape. While Linaro is a consortium of six major ARM companies, its work is meant to be available to all ARM companies, developers, or users. The focus may be on the needs of its member companies, but Linaro's influence and efforts are likely to spread well beyond just those needs. It is an organization that wants to try to keep the best interests of the entire ARM ecosystem in mind—perhaps that's not so surprising with both ARM, Ltd. and major ARM silicon fabricator IBM on board.

In the year since Linaro began to take shape, and roughly eleven months since it was announced to the world, the organization has expanded its scope, changed to a monthly release cycle, and stepped in to to try to help head off a crisis in the ARM tree. It has also made progress in many of the problem areas (kernel, toolchain, graphics, ...) that it set out for itself. But there is, of course, lots more to do.

Linaro Developer Summit

Linaro CEO George Grey spoke briefly at the opening of the Linaro Developer Summit (LDS), which was co-located with Ubuntu Developer Summit in Budapest, and described some industry trends and things he sees coming for Linaro. Grey noted that the last twelve months have shown some "extraordinary changes" in the use of open source software in real world products. "Android in particular has startled a lot of people", he said. The Android share of the smartphone market has risen from 5% to 25% in that time span, he said, which is something that has never happened before.

Device manufacturers are no longer happy to get a board support package (BSP) that is out of date and requires a BSP-specific toolchain. Instead, they are looking for product-quality open source platforms, which is something that Linaro is delivering. The Linaro automated validation architecture (LAVA)—a test and validation platform—will be very important to that effort as it will help increase the quality of the software that Linaro delivers.

Getting development platforms into the hands of open source developers is another area where Linaro can make a difference, Grey said. In the past, it has been difficult for smaller players to get their hands on these development platforms because it is hard to get the attention of the system-on-chip (SoC) vendors. But, these days there are development platforms at reasonable prices for ARM SoCs from Texas Instruments, Freescale, and ST-Ericsson (all Linaro members), with more coming.

That solves the hardware problem, and Linaro will be providing software to run on those boards. That means that companies or developers with limited budgets can get complete hardware with multiple I/O devices and a product quality software stack. Grey is excited to see what the community can do with all of that.

Grey also announced that there would be no Linaro 11.11 (it had been making releases based on Ubuntu's schedule, though delayed by one month) in favor of monthly (or more frequent) code drops. The various Linaro working groups will be continuously pushing their code upstream, while there will be daily releases of some code, mostly for internal use. There will also be monthly supported releases of the kernel and toolchain for external use.

Looking to the future, Grey noted that work was progressing on support for the Cortex A15 processor. The intent is to have the processor supported by Linux when it releases, rather than a year or two later as has been the case in the past. He also said that there "clearly is going to be a market for ARM-based servers", and Linaro is doing some early work on that market segment as well.

Linaro VP of Engineering Christian "Kiko" Reis spoke after Grey, mostly about the nuts and bolts of how LDS would operate, and how attendees could get the most out of it. He also looked back over Linaro's first year, noting that a lot of progress had been made, particularly in the areas of communication and collaboration between various ARM vendors. Starting Linaro wasn't easy, he said, but there are harder things still to be done, including bringing the whole ARM community together.

Specifically, Reis mentioned the ARM kernel consolidation work that still needs to be done. Linaro "spent a year not really doing this", and now is the time to get it done. Determining the right course for memory management for embedded graphics devices is also high on the list. Delivering that will take more than just a year, but planning for that is one of the priorities for the week.

The organization of Linaro

David Rusling, Linaro CTO, gave a talk at the Embedded Linux Conference back in April to give an overview of the organization, highlight some of the accomplishments in the first ten months, and look ahead to some future plans. Rusling started off by clarifying that Linaro is meant to be an engineering organization and not a "tea and cakes in Honolulu standards thing". He also pointed out that "Linaro" was the "least hated name" among the candidates, which mirrored Dirk Hohndel's explanation earlier in the day of the choice of the name "Yocto" for that project.

Rusling said that he recognized that the fates of ARM and Linux were intertwined in 2009 or so, when he was at ARM, Ltd., but there were lots of companies "running around" and not collaborating. There was a clear need for more collaboration on the things that were common between various SoCs, but there were not enough engineers working on those problems. Linaro was born out of that need.

Linaro started out with around 20 engineers and was envisioned as an "upstream engineering" organization. The "genius move" was to do all of that in the open, he said. Everything is on the wiki, though some things may be hard to find, and "upstream is our mantra".

Much of what Linaro does is "social engineering", Rusling said. There are a number of misconceptions about Linux and open source that need to be dispelled, including the idea that open source is difficult to deal with. There are gatekeepers in Linux and other projects that have strong views, but interested organizations and vendors simply need to "engage with them". The "really bad false statement" is that open source is cheaper. That's not really true as working in the open source communities requires a deeper involvement because it's all about influencing development direction; there is no control, he said.

The six member companies want to drive the technical agenda for Linaro, which does its work through the working groups that have been established. Those working groups are "very independent" and the member companies aren't trying to run the projects directly, but are instead allowing the groups to work upstream on solving problems in their areas.

There is also a platform group that builds, tests, and benchmarks the work that is done by the working groups (and upstream projects). The idea is to prove that new kernel features work or that tool changes make things go faster by creating and testing evaluation builds. "Any time you do any changes you have to measure" the impact of those changes, he said. There are also landing teams for each of the member SoC vendors (Samsung, Texas Instruments, ST-Ericsson, and Freescale) that take the outputs from the platform team and turn it into usable builds for their customers. The landing teams are the only teams in Linaro that are closed to community participation.

It is not just kernel work that lands upstream, as the toolchain working group is doing a lot of work on GCC and other tools. Support for ARMv7a, Thumb 2, Neon, and SMP are being added to GCC 4.7, which won't be released until April 2012, and won't get into distributions until October 2012 or so, sometime after the 4.7.1 release. In the interim, the toolchain group will be making "consolidation builds" that can be used by ARM developers prior to the GCC release. In addition to work on GCC, the group is also adding functionality to tools like gdb, QEMU, and valgrind, he said.

After the first release in November, two new working groups were added to address graphics and multimedia issues. In addition, the other working groups started looking at long-term problems, Rusling said. The kernel group started adding device tree support for all of the Linaro members' hardware. Work on vectorizing support for GCC was one focus of the toolchain group, and the power management group started tackling segmented memory so that portions of the memory can be powered down. All of those things are "tricky areas that require a lot of coordination within the ARM space, but also upstream", he said.

For multimedia, much of the work involves testing, benchmarking, and tuning various codecs for ARM. Standardizing on the OpenMax media libraries and the GStreamer framework is the direction that Linaro is going. Android has gone its own way in terms of multimedia support, he said.

Rusling, like Grey, also pointed to the work being done on LAVA as something that is very important to Linaro, but also to the community. It is a "completely open" test and validation system that could be used by others in the ARM community or beyond.

There were some hard lessons learned in the first year or so of Linaro's existence, Rusling said. It is difficult to build a new engineering organization from scratch with engineers donated from multiple companies. On the other hand, people thought that the ARM community couldn't collaborate but Linaro has shown that not to be the case. Everything takes longer than he would like, and there is still a lot to learn, he said. "Open source is wonderful", but there are challenges to using it.

It is clear that ARM has become a very important architecture for Linux, and is completely dominating the low-power mobile device market. That doesn't look likely to change anytime soon, and it may be that ARM's efforts to move into the server space will bear fruit in the next few years. Certainly power consumption (and the associated heat produced) are very important not just in pockets, but in data centers as well. Linaro has, so far, been making many of the right moves to ensure that ARM is well-supported—and maintainable—in Linux. It will be interesting to see what the next year (and beyond) hold.

Comments (16 posted)

Examining MeeGo's openness and transparency

The MeeGo project is barely a year old, and in addition to the usual bootstrapping issues, it has had to deal with the headaches of merging two pre-existing, corporate-backed Linux distributions (Intel's Moblin and Nokia's Maemo), and spawning several independent target platforms: tablets, handhelds, netbooks, vehicles, and set-top boxes. At the MeeGo Conference in San Francisco, several speakers tackled the openness and transparency of the project head-on.

The elephant in the room at the event is Nokia's February announcement that it was backing off its own MeeGo Handset UX plans in favor of a deal with Microsoft for Windows Phone 7 devices. That event spawned a great public perception problem for MeeGo, in which many consumers — for whom MeeGo was synonymous with "Linux phone OS" — assumed the project was finished. Both the Maemo and Moblin sides of the MeeGo community are still extremely active, however. MeeGo's importance as an OEM platform seems solid (particularly in tablet space and in non-"consumer device" verticals such as set-top boxes and in-vehicle infotainment). The community seems resigned to the fact that until there is a phone on the market, many outsiders won't pay attention to the platform, but it appears to be undeterred.

Messaging

Carsten Munk is a veteran of the Maemo project and maintainer of the MeeGo port for Nokia's N900 phone. He also started the short-lived Mer project which sought to re-build an entirely community-maintained version of Maemo for officially unsupported devices. Thus, in spite of holding an official position in the MeeGo project, he has well-earned clout when it comes to the subject of openness. His talk on Monday afternoon was entitled "Transparency, Inclusion, and Meritocracy in MeeGo: Theory and Practice," and was a hard (but not disparaging) examination of how well the project is living up to its advertised principles.

The three factors Munk explored, transparency, inclusion and meritocracy, derive indirectly from the project's branding. The meego.com web site and MeeGo marketing materials mention meritocracy repeatedly as a core value, Munk said, but never define what the project thinks it means or how it works in practice. Digging further into the site, Munk turned up only one vague description of the project's governance as "based on meritocracy and the best practices and values of the Open Source culture" and scattered references to vendor-neutrality and community encouragement.

Searching with Google reveals what the broader public consensus is of open source's "best practices and values," he said, including transparent processes, peer review, and the absence of membership fees or contracts before getting involved. Thus, real transparency and real inclusiveness are prerequisites to a meritocracy culture, he argued: after all, how can one be meritocratic if one cannot see what is happening and actually participate.

The project is failing to define and argue for these core values to outsiders, he said, it needs to state them clearly and prominently both on the site and in its public relations campaigns. Without a clear message on these core values, he said, developers will eventually drift away to other projects: the meritocracy culture is what keeps volunteer developers coming back to donate their time.

Practice

Munk then turned to an examination of the MeeGo project's real-life processes and structures, measuring each on transparency, inclusion, and meritocracy. He came away with three recurring patterns of behavior he sees in MeeGo, and provided feedback on how to improve problem areas in each measured category.

The first pattern Munk explained was the "gatekeeper" pattern, in which a distinct team follows a transparent, well-defined process, but where all of the decisions are made by the team alone. Typically, the team uses transparent processes to interact with the larger project and community, but its internal discussions and decision-making are a black box. The intra-team black-box makes it difficult for non-team-members to get involved (a low inclusiveness quotient), and as a result, gatekeeper teams fall short on meritocracy -- even if, in theory, the team is receptive to ideas from outsiders.

Munk's primary example of the gatekeeper pattern is MeeGo's release engineering team, which he describes as perpetually operating out-of-sight. Most developers within the MeeGo project never see the release engineering team; they simply get an email from them when a release is ready. Other examples from around the project include the IT infrastructure team, legal services, the architecture team, the program- and product-management groups, and the Technical Steering Group (TSG). Fortunately, Munk said, improving on gatekeeper patterns is straightforward: simply having the team conduct its meetings and discussions in the open. There are always going to be areas where legal or security requirements dictate closing the meeting-room door, he said, but openness should be the default.

The second pattern is the "open office" pattern, typified by broad openness in discussions, communication tools, and meetings, placing all participants (core and casual) into one shared virtual space. Although this pattern scores extremely high on transparency, Munk said, it can actually cause unintentional harm on inclusiveness and meritocracy simply because the volume of information can create overload. Suggestions can be lost in the din, and individual contributions can be accidentally overlooked.

Munk's example open office team is the MeeGo Community Office, which incorporates forum, mailing list, wiki, and IRC communication channels, as well as community bug tracking and feedback. The MeeGo developer and user community can be large enough for individual contributions or comments to get lost in the crowd, and often includes almost disjoint sets of contributors (Dave Neary and Dawn Foster discussed that last issue in the "MeeGo Community Dashboard" talk: forum users and mailing list users represent largely non-overlapping communities, although both regularly contribute to IRC discussions). Some other MeeGo teams that operate using the open office pattern include the ARM/n900 team, the SDK team, QA and bug triage, the events team, and the internationalization team.

Munk's suggestions for improving the open office pattern's weak points include creating formal processes to separate out and recognize contributions, and subdividing teams that become too large. Recognizing individuals' contributions, of course, is necessary for inclusiveness, but in order to do it, the team needs to have a regular process in place to report and analyze community involvement. The MeeGo Community Office does have this, as Foster and Neary explained in their talk, although it still needs streamlining.

The third (and by far the most problem-ridden) pattern Munk identified is the "invisible effort," in which not only are a team's communication and processes closed to outside eyes, but even its membership and makeup is a mystery. In the worst case scenario, there is no public mailing list, wiki presence, bug tracker activity, source code repository, roadmap, or even documentation of the team's existence. Along with the complete lack of transparency, this pattern makes it impossible for the team to be inclusive (since it is impossible to contact a team whose identity is hidden), and impossible for it to be meritocratic.

MeeGo does have several invisible effort teams, Munk said, most notably the UI design team. No one on the public side of the project is sure who is on the design team, and their work remains undocumented and secret up until the moment it is revealed to the general public. Other examples include the MeeGo Bugzilla support team (who Munk said is invisible until a change to the project Bugzilla is rolled out), and teams responsible for the occasional "big reveal" announcements (which are typically product-related or reveal the involvement of new partners). There are also a few smaller MeeGo teams that are not formally defined, so they function similar to invisible efforts, albeit unintentionally.

On the plus side, Munk's "how to improve" list for this pattern is quite simple: "Anything!" From a complete lack of transparency, inclusiveness, and meritocracy, any step forward is welcome. Although, he said, improving on transparency is the obvious first step. Many of the invisible effort patterns arise because the team in question is not distributed, and instead is an internal office inside one vendor, which has never before had to interact with an open source project. It may be a radical change in mindset, but fixing the invisible effort team pattern is not optional just because the team has always done things its way.

Traps and pitfalls

In the final section of his talk, Munk outlined common traps and pitfalls that any team or individual might slip into, and which have a detrimental effect on transparency, inclusiveness, and meritocracy. The first is the CC List, in which an individual starts an important discussion outside of the public mailing list by including a long string of CC'ed addresses. This bypasses the transparency expected of the project, and is often a relic of corporate culture. The "big reveals" mentioned earlier fall into this category as well, as does the "BS bug", in which someone files a bug report or feature request that states unequivocally "MeeGo Needs X," without supporting evidence or discussion, then uses the bug report to urge action.

"Corporate nominations" are another trap, in which the TSG appoints someone from outside the MeeGo project to fill a new or important role, without discussion or an open nomination period. Often, the person so nominated does not have a publicly-accessible resume or known reputation to justify their new leadership role. Munk allowed that a certain amount of this was necessary in the early days of MeeGo, when there were not enough active contributors to fill every need, but said now that the project is no longer in "bootstrap" mode, the practice needed to end. He did not give examples of these nominations, but the audience seemed to concur that they have, in fact, happened.

In Monday's keynote, Imad Sousou touched on this same topic in reference to the TSG membership itself in response to an audience question about the TSG's makeup. Sousou said that the TSG's membership was open to the community, and anyone was welcome to make a nomination (including nominating themselves). I asked several community members about that comment after the keynote. All regarded it with disbelief, with several pointing out that there was no documented nomination or election process, nor a charter that spelled out the TSG's makeup at all.

The first few pitfalls were corporate-behavior-related, but Munk mentioned several common to the community side as well. The "loudest argument wins" trap is a common one, where one voice simply wears out all of the others (by volume or belligerence), by flooding lists or forums. This sidetracks meritocracy, because the discussion does not stick to debatable facts. A second pitfall is the "more talk / less doing" trap, which many in open source would recognize: someone who refuses to participate in the solution, but prolongs debate endlessly or repeatedly makes complaints. The third is the "information vampire" pitfall, which is often unintentional. In that trap, someone slows down a team or a process by making too many personal requests for information. The sheer "paperwork" of keeping a vampire satisfied detracts from a team's energy to push the project forward. Lest anyone think he was only out to point fingers, Munk identified himself as often being guilty of information vampirism.

In conclusion, Munk reiterated that the core values of transparency, inclusiveness, and meritocracy are not merely nice ideals, but are a practical "win" for everyone involved in MeeGo (or any other large open source effort). The three are interrelated, but all begin with transparency, he said. 100% transparency is not practical, but he encouraged the community to hold the MeeGo project to its advertised principles, for the benefit of all sides.

On the day after his talk, I asked Munk whether he had received any feedback since the talk, and he said that he obviously hadn't offended enough people, because so far no one had complained to him about what he said. Obviously he could have gone further had he wanted to, such as pointing out specific instances of the pitfalls and shortcomings he outlined in the session, but that would not have really changed the point. My impression of the MeeGo community is that everyone is well aware of what parts of the project are working smoothly and where there are hiccups, strained communications, and difficult processes. Fifteen months is not a lot of time to roll out a large, multi-faceted distributed project like MeeGo, especially one that targets multiple audiences (such as hardware makers, developers, and consumers).

Perhaps the best thing about Munk's talk was his ability to systematically break down the issues that various pieces of the MeeGo machine are struggling with. By shining some light directly on the problems, there is less chance of argument and more room for repairing the cracks — which is clearly what everyone wants. Other large projects could probably take a cue from Munk's taxonomy, and take a look at their own teams, subprojects, and processes: regardless of the target audience of the project, most in the free software world share the same set of goals.

Comments (5 posted)

OpenOffice.org and contributor agreements

As part of an interview in LWN, Mark Shuttleworth is quoted as wanting the community to view the use of contribution licensing agreements (CLAs) as a necessary prerequisite for open source growth and the refusal by others to donate their work under them as a barrier to commercial success. He implies that the use of a CLA by Sun for OpenOffice.org was a good thing. Mark is also quoted accusing The Document Foundation (TDF) of somehow destroying OpenOffice.org (OO.o) because of its decision to fork rather than contribute changes upstream under the CLA. But I'd suggest a different view of both matters.

[ Disclosures: Having been directly involved while at Sun in some of the events I discuss, I should say that everything recounted here is a matter of public record. While I'd love to explain some of the events at greater length, I'm not free to do so - such is the penalty of employment contracts. Suffice to say that just because a company takes certain actions, it doesn't mean everyone agrees with them, even senior people. I should also disclose that I was honored to be granted membership in TDF recently, although I'm not speaking on their official behalf here. ]

My experience of CLAs has led me to conclude that they prevent competitors from collaborating over a code base. The result is that any project which can only be monetized in its own right rather than as a part of a greater collective work will have a single commercial contributor if a CLA is in place. Mark would like us all to accept contributor agreements so that a legacy business model based on artificial scarcity can be used to make a profit that would otherwise be unavailable without the unrewarded donation of the work of others - who are not permitted the same opportunity. I've covered this in greater detail in my article "Transparency and Privacy".

While pragmatically there may be isolated circumstances under which CLAs are the lesser of evils, their role in OO.o has contributed more to its demise than offered it hope. By having a CLA for OpenOffice.org, Sun guaranteed that core contributions by others would be marginal and that Sun would have to invest the vast majority of the money in its development. That was a knowing, deliberate choice made in support of the StarOffice business models.

Despite that, others did work on the code - a testimony to the importance of OO.o. The most significant community work was localization, and the importance of the work done by the many teams globally making OpenOffice.org work for them can't be overstated. In the core code, Novell worked inside the community, accepting the premise Mark is asserting and contributing significant value in small quantities over an extended period. Smaller contributions came from Red Hat and others. IBM worked outside the community, developing the early version of Symphony, contributing nothing back to the OO.o community, and thus preventing any network effect from being accelerated by their work.

Sun used the aggregated copyright in 2005 to modernize the licensing arrangements for OO.o, closing the dual-license loophole IBM had been using to avoid contribution. Some (me included) hoped this would bring IBM into the community, but instead Sun also used the aggregated copyright to allow IBM to continue Symphony development outside the community rather than under the CLA. Stung by this rejection of transparent community development - and in distaste at proprietary deals rather than public licensing driving contribution - Novell started to bulk up the Go-OO fork, by ceasing to give their contributions back. This all happened long before the Oracle acquisition.

The act of creating The Document Foundation and its LibreOffice project did no demonstrable harm to Oracle's business. There is no new commercial competition to Oracle Open Office (their commercial edition of OO.o) arising from LibreOffice. No contributions that Oracle valued were ended by its creation. Oracle's ability to continue development of the code was in no way impaired. Oracle's decision appears to be simply that, after a year of evaluation, the profit to be made from developing Oracle Open Office and Oracle Cloud Office did not justify the salaries of over 100 senior developers working on them both. Suggesting that TDF was in some way to blame for a hard-headed business decision that seemed inevitable from the day Oracle's acquisition of Sun was announced is at best disingenuous.

So what's left of the OpenOffice.org community now? Almost all of the people who worked on it outside of Sun/Oracle have joined together to create the Document Foundation. As far as I can tell all that's left of OO.o is the project shell, with no development work in progress because Oracle has now stood down all the staff it had working on the project, plus one or two voices trying desperately to make us believe they still have some authority to comment because they used to be involved with it.

The CLA bears a great deal of responsibility for this situation, having kept the cost of development high for Sun/Oracle and prevented otherwise willing contributors from fully engaging. There is one hope left from it though. Because of the CLA, Oracle is free to choose to donate the whole of OO.o to The Document Foundation where it can be managed by the diverse and capable community that's gathered there. If they don't do that they probably need to change the license to a weak copyleft license - the new Mozilla Public License v2 would be perfect - to allow IBM to join the main community. It's my sincere hope that they will finally put the CLA to a good use and allow it to reunite the community it has divided for so long.

Comments (44 posted)

Next week's weekly edition comes out June 3

Due to the US Memorial Day holiday on May 30, next week's weekly edition will be delayed by one day. That means that it will be published on Friday, June 3. Several US holidays are arranged to land on Mondays and we have finally decided to go ahead and take some of those days off. July 4 (Independence Day) and September 5 (Labor Day) are also holidays that we plan to take off this year. Hope this doesn't inconvenience anyone too much, and have a nice Monday whether you are celebrating the holiday or not.

Comments (2 posted)

Page editor: Jonathan Corbet
Next page: Security>>