Over the last month or so, I have sat in on a few different talks about
Linaro and the work that it is doing to improve the ARM Linux landscape.
While Linaro is a
consortium of six major ARM companies, its work
is meant to be available to all ARM companies, developers, or users.
The focus may be on the needs of its member companies, but Linaro's
efforts are likely to spread well beyond just those needs. It is an
organization that wants to try to keep the best interests of the entire ARM
ecosystem in mind—perhaps that's not so surprising with both ARM,
Ltd. and major ARM silicon fabricator IBM on board.
In the year since Linaro began to take shape, and roughly eleven months
since it was announced to
the world, the organization has expanded its scope, changed to a monthly
release cycle, and stepped in to to try to help head
off a crisis in the ARM tree. It has also made progress in many of the
problem areas (kernel, toolchain, graphics, ...) that it set out for
itself. But there is, of course, lots more to do.
Linaro Developer Summit
Linaro CEO George Grey spoke briefly at the opening of the Linaro Developer
Summit (LDS), which was co-located with Ubuntu Developer Summit in
Budapest, and described some industry trends and things he sees coming for Linaro. Grey
noted that the last twelve months have shown some "extraordinary
changes" in the use of open source software in real world products.
"Android in particular has startled a lot of people", he
said. The Android share of the smartphone market has risen from 5% to 25%
in that time span, he said, which is something that has never happened before.
Device manufacturers are no longer happy to get a board support package
(BSP) that is out of date and requires a BSP-specific toolchain. Instead,
they are looking for product-quality open source platforms, which is
something that Linaro is delivering. The Linaro automated validation
architecture (LAVA)—a test and validation platform—will be very
important to that effort as it will help increase the quality of the
software that Linaro delivers.
Getting development platforms into the hands of open source developers is
another area where Linaro can make a difference, Grey said. In the past,
it has been difficult for smaller players to get their hands on these
development platforms because it is hard to get the attention of the
system-on-chip (SoC) vendors. But, these days there are development
platforms at reasonable prices for ARM SoCs from Texas Instruments,
Freescale, and ST-Ericsson (all Linaro members), with more coming.
That solves the hardware problem, and Linaro will be providing software to
run on those boards. That means that companies or developers with limited
budgets can get complete hardware with multiple I/O devices and a product
quality software stack. Grey is excited to see what the community can do
with all of that.
Grey also announced that there would be no Linaro 11.11 (it had been
making releases based on Ubuntu's schedule, though delayed by one month) in
favor of monthly (or more frequent) code drops. The various Linaro working
groups will be continuously pushing their code upstream, while there will
be daily releases of some code, mostly for internal use. There will also be
monthly supported releases of the kernel and toolchain for external use.
Looking to the future, Grey noted that work was progressing on support for
A15 processor. The intent is to have the processor supported by Linux
when it releases, rather than a year or two later as has been the case in
the past. He also said that there "clearly is going to be a market
for ARM-based servers", and Linaro is doing some early work on that
market segment as well.
Linaro VP of Engineering Christian "Kiko" Reis spoke after Grey, mostly
about the nuts and bolts of how LDS would operate, and how attendees could
get the most out of it. He also looked back over Linaro's first year,
noting that a lot of progress had been made, particularly in the areas of
communication and collaboration between various ARM vendors. Starting
Linaro wasn't easy, he said, but there are harder things still to be done,
including bringing the whole ARM community together.
Specifically, Reis mentioned the ARM kernel consolidation work that still
needs to be done. Linaro "spent a year not really doing
this", and now is the time to get it done. Determining the right
course for memory management for
embedded graphics devices is also high on the list. Delivering that will
than just a year, but planning for that is one of the priorities for the week.
The organization of Linaro
David Rusling, Linaro CTO, gave a talk at the Embedded Linux Conference back in
April to give an overview of the organization, highlight some of the
accomplishments in the first ten months, and look ahead to some future
plans. Rusling started off by clarifying that Linaro is meant to be an
engineering organization and not a "tea and cakes in Honolulu
standards thing". He also pointed out that "Linaro" was the "least
hated name" among the candidates, which mirrored Dirk Hohndel's
explanation earlier in the day of the choice of the name "Yocto" for that
Rusling said that he recognized that the fates of ARM and Linux were
intertwined in 2009 or so, when he was at ARM, Ltd., but there were lots of
companies "running around" and not collaborating. There was
a clear need for more collaboration on the things that were common between
various SoCs, but there were not enough engineers working on those
problems. Linaro was born out of that need.
Linaro started out with around 20 engineers and was envisioned as an
"upstream engineering" organization. The "genius
move" was to do all of that in the open, he said. Everything is on
the wiki, though some things may be hard to find, and "upstream is
Much of what Linaro does is "social engineering", Rusling
said. There are a number of misconceptions about Linux and open source that
need to be dispelled, including the idea that open source is difficult to
deal with. There are gatekeepers in Linux and other projects that have
strong views, but interested organizations and vendors simply need to
"engage with them". The "really bad false
statement" is that open source is cheaper. That's not really true
as working in the open source communities requires a deeper involvement
because it's all about influencing development direction; there is no
control, he said.
The six member companies want to drive the technical agenda for Linaro, which
does its work through the
working groups that have been established. Those working groups are
"very independent" and the member companies aren't trying to
run the projects directly, but are instead allowing the groups to work
upstream on solving problems in their areas.
There is also a platform
group that builds, tests, and benchmarks the work that is done by the
working groups (and upstream projects). The idea is to prove that new
kernel features work or that tool changes make things go faster by creating
and testing evaluation builds. "Any time you do any changes you have
to measure" the impact of those changes, he said. There are also
landing teams for each of the member SoC vendors (Samsung, Texas
Instruments, ST-Ericsson, and Freescale) that take the outputs from the
platform team and turn it into usable builds for their customers. The
landing teams are the only teams in Linaro that are closed to community
It is not just kernel work that lands upstream, as the toolchain working
group is doing a lot of work on GCC and other tools. Support for ARMv7a,
Thumb 2, Neon, and SMP are being added to GCC 4.7, which won't be released
until April 2012, and won't get into distributions until October 2012 or so,
sometime after the 4.7.1 release. In the interim, the toolchain group will
be making "consolidation builds" that can be used by ARM developers prior
to the GCC release. In addition to work on GCC, the group is also adding
functionality to tools like gdb, QEMU, and valgrind, he said.
After the first release in November, two new working groups were added to
address graphics and multimedia issues. In addition, the other working
groups started looking at long-term problems, Rusling said. The kernel
group started adding device tree support for all of the Linaro members'
hardware. Work on vectorizing support for GCC was one focus of the
toolchain group, and the power management group started tackling segmented
memory so that portions of the memory can be powered down. All of those
things are "tricky areas that require a lot of coordination within the
ARM space, but also upstream", he said.
For multimedia, much of the work involves testing, benchmarking, and tuning
various codecs for ARM. Standardizing on the OpenMax media libraries and the
GStreamer framework is the direction that Linaro is going. Android has gone
its own way in terms of multimedia support, he said.
Rusling, like Grey,
also pointed to the work being done on LAVA as something that is very
important to Linaro, but also to the community. It is a "completely
open" test and validation system that could be used by others in the
ARM community or beyond.
There were some hard lessons learned in the first year or so of Linaro's
existence, Rusling said. It is difficult to build a new engineering
organization from scratch with engineers donated from multiple companies.
On the other hand, people thought that the ARM community couldn't
collaborate but Linaro has shown that not to be the case. Everything takes
longer than he would like, and there is still a lot to learn, he
said. "Open source is wonderful", but there are challenges to
It is clear that ARM has become a very important architecture for Linux,
and is completely dominating the low-power mobile device market. That
doesn't look likely
to change anytime soon, and it may be that ARM's efforts to move into the
server space will bear fruit in the next few years. Certainly power consumption
(and the associated heat produced) are very important not just in pockets,
but in data centers as well. Linaro has, so far, been making many of the
right moves to ensure that ARM is well-supported—and
maintainable—in Linux. It will be interesting to see what the next
year (and beyond) hold.
Comments (16 posted)
The MeeGo project is barely a year old, and in addition to the usual
bootstrapping issues, it has had to deal with the headaches of merging two
pre-existing, corporate-backed Linux distributions (Intel's Moblin and
Nokia's Maemo), and spawning several independent target platforms: tablets,
handhelds, netbooks, vehicles, and set-top boxes. At the MeeGo Conference in San Francisco,
several speakers tackled the openness and transparency of the project
The elephant in the room at the event is Nokia's February announcement that
it was backing off its own MeeGo Handset UX plans in favor of a deal with
Microsoft for Windows Phone 7 devices. That event spawned a great public
perception problem for MeeGo, in which many consumers — for whom MeeGo
was synonymous with "Linux phone OS" — assumed the project was
finished. Both the Maemo and Moblin sides of the MeeGo community are still
extremely active, however. MeeGo's importance as an OEM platform seems
solid (particularly in tablet space and in non-"consumer device" verticals
such as set-top boxes and in-vehicle infotainment). The community seems resigned to the fact
that until there is a phone on the market, many outsiders won't pay
attention to the platform, but it appears to be undeterred.
Carsten Munk is a veteran of the Maemo project and maintainer of the
MeeGo port for Nokia's N900 phone. He also started the short-lived Mer project which sought to re-build an entirely community-maintained version of Maemo for officially unsupported devices. Thus, in spite of holding an official position in the MeeGo project, he has well-earned clout when it comes to the subject of openness. His talk on Monday afternoon was entitled "Transparency, Inclusion, and Meritocracy in MeeGo: Theory and Practice," and was a hard (but not disparaging) examination of how well the project is living up to its advertised principles.
The three factors Munk explored, transparency, inclusion and meritocracy, derive indirectly from the project's branding. The meego.com web site and MeeGo marketing materials mention meritocracy repeatedly as a core value, Munk said, but never define what the project thinks it means or how it works in practice. Digging further into the site, Munk turned up only one vague description of the project's governance as "based on meritocracy and the best practices and values of the Open Source culture" and scattered references to vendor-neutrality and community encouragement.
Searching with Google reveals what the broader public consensus is of open source's "best practices and values," he said, including transparent processes, peer review, and the absence of membership fees or contracts before getting involved. Thus, real transparency and real inclusiveness are prerequisites to a meritocracy culture, he argued: after all, how can one be meritocratic if one cannot see what is happening and actually participate.
The project is failing to define and argue for these core values to outsiders, he said, it needs to state them clearly and prominently both on the site and in its public relations campaigns. Without a clear message on these core values, he said, developers will eventually drift away to other projects: the meritocracy culture is what keeps volunteer developers coming back to donate their time.
Munk then turned to an examination of the MeeGo project's real-life processes and structures, measuring each on transparency, inclusion, and meritocracy. He came away with three recurring patterns of behavior he sees in MeeGo, and provided feedback on how to improve problem areas in each measured category.
The first pattern Munk explained was the "gatekeeper" pattern, in which a distinct team follows a transparent, well-defined process, but where all of the decisions are made by the team alone. Typically, the team uses transparent processes to interact with the larger project and community, but its internal discussions and decision-making are a black box. The intra-team black-box makes it difficult for non-team-members to get involved (a low inclusiveness quotient), and as a result, gatekeeper teams fall short on meritocracy -- even if, in theory, the team is receptive to ideas from outsiders.
Munk's primary example of the gatekeeper pattern is MeeGo's release engineering team, which he describes as perpetually operating out-of-sight. Most developers within the MeeGo project never see the release engineering team; they simply get an email from them when a release is ready. Other examples from around the project include the IT infrastructure team, legal services, the architecture team, the program- and product-management groups, and the Technical Steering Group (TSG). Fortunately, Munk said, improving on gatekeeper patterns is straightforward: simply having the team conduct its meetings and discussions in the open. There are always going to be areas where legal or security requirements dictate closing the meeting-room door, he said, but openness should be the default.
The second pattern is the "open office" pattern, typified by broad openness in discussions, communication tools, and meetings, placing all participants (core and casual) into one shared virtual space. Although this pattern scores extremely high on transparency, Munk said, it can actually cause unintentional harm on inclusiveness and meritocracy simply because the volume of information can create overload. Suggestions can be lost in the din, and individual contributions can be accidentally overlooked.
Munk's example open office team is the MeeGo Community Office, which
incorporates forum, mailing list, wiki, and IRC communication channels, as
well as community bug tracking and feedback. The MeeGo developer and user
community can be large enough for individual contributions or comments to
get lost in the crowd, and often includes
almost disjoint sets of contributors (Dave Neary and Dawn Foster discussed
that last issue in the "MeeGo Community Dashboard" talk: forum users and mailing list users represent largely non-overlapping communities, although both regularly contribute to IRC discussions). Some other MeeGo teams that operate using the open office pattern include the ARM/n900 team, the SDK team, QA and bug triage, the events team, and the internationalization team.
Munk's suggestions for improving the open office pattern's weak points include creating formal processes to separate out and recognize contributions, and subdividing teams that become too large. Recognizing individuals' contributions, of course, is necessary for inclusiveness, but in order to do it, the team needs to have a regular process in place to report and analyze community involvement. The MeeGo Community Office does have this, as Foster and Neary explained in their talk, although it still needs streamlining.
The third (and by far the most problem-ridden) pattern Munk identified is the "invisible effort," in which not only are a team's communication and processes closed to outside eyes, but even its membership and makeup is a mystery. In the worst case scenario, there is no public mailing list, wiki presence, bug tracker activity, source code repository, roadmap, or even documentation of the team's existence. Along with the complete lack of transparency, this pattern makes it impossible for the team to be inclusive (since it is impossible to contact a team whose identity is hidden), and impossible for it to be meritocratic.
No one on the public side of the project is sure who
is on the design team, and their work remains undocumented and secret up
until the moment it is revealed to the general public.
MeeGo does have several invisible effort teams, Munk said, most notably
the UI design team. No one on the public side of the project is sure who
is on the design team, and their work remains undocumented and secret up
until the moment it is revealed to the general public. Other examples
include the MeeGo Bugzilla support team (who Munk said is invisible until a
change to the project Bugzilla is rolled out), and teams responsible for
the occasional "big reveal" announcements (which are typically
product-related or reveal the involvement of new partners). There are also
a few smaller MeeGo teams that are not formally defined, so they function
similar to invisible efforts, albeit unintentionally.
On the plus side, Munk's "how to improve" list for this pattern is quite
simple: "Anything!" From a complete lack of transparency,
inclusiveness, and meritocracy, any step forward is welcome. Although, he
said, improving on transparency is the obvious first step. Many of the
invisible effort patterns arise because the team in question is not
distributed, and instead is an internal office inside one vendor, which has
never before had to interact with an open source project. It may be a
radical change in mindset, but fixing the invisible effort team pattern is not optional just because the team has always done things its way.
Traps and pitfalls
In the final section of his talk, Munk outlined common traps and pitfalls that any team or individual might slip into, and which have a detrimental effect on transparency, inclusiveness, and meritocracy. The first is the CC List, in which an individual starts an important discussion outside of the public mailing list by including a long string of CC'ed addresses. This bypasses the transparency expected of the project, and is often a relic of corporate culture. The "big reveals" mentioned earlier fall into this category as well, as does the "BS bug", in which someone files a bug report or feature request that states unequivocally "MeeGo Needs X," without supporting evidence or discussion, then uses the bug report to urge action.
"Corporate nominations" are another trap, in which the TSG appoints someone from outside the MeeGo project to fill a new or important role, without discussion or an open nomination period. Often, the person so nominated does not have a publicly-accessible resume or known reputation to justify their new leadership role. Munk allowed that a certain amount of this was necessary in the early days of MeeGo, when there were not enough active contributors to fill every need, but said now that the project is no longer in "bootstrap" mode, the practice needed to end. He did not give examples of these nominations, but the audience seemed to concur that they have, in fact, happened.
In Monday's keynote, Imad Sousou touched on this same topic in reference to the TSG membership itself in response to an audience question about the TSG's makeup. Sousou said that the TSG's membership was open to the community, and anyone was welcome to make a nomination (including nominating themselves). I asked several community members about that comment after the keynote. All regarded it with disbelief, with several pointing out that there was no documented nomination or election process, nor a charter that spelled out the TSG's makeup at all.
The first few pitfalls were corporate-behavior-related, but Munk
mentioned several common to the community side as well. The "loudest
argument wins" trap is a common one, where one voice simply wears out all
of the others (by volume or belligerence), by flooding lists or forums. This sidetracks meritocracy, because the discussion does not stick to debatable facts. A second pitfall is the "more talk / less doing" trap, which many in open source would recognize: someone who refuses to participate in the solution, but prolongs debate endlessly or repeatedly makes complaints. The third is the "information vampire" pitfall, which is often unintentional. In that trap, someone slows down a team or a process by making too many personal requests for information. The sheer "paperwork" of keeping a vampire satisfied detracts from a team's energy to push the project forward. Lest anyone think he was only out to point fingers, Munk identified himself as often being guilty of information vampirism.
In conclusion, Munk reiterated that the core values of transparency,
inclusiveness, and meritocracy are not merely nice ideals, but are a
practical "win" for everyone involved in MeeGo (or any other large open
source effort). The three are interrelated, but all begin with
transparency, he said. 100% transparency is not practical, but he
encouraged the community to hold the MeeGo project to its advertised principles, for the benefit of all sides.
On the day after his talk, I asked Munk whether he had received any feedback since the
talk, and he said that he obviously hadn't offended enough people, because
so far no one had complained to him about what he said. Obviously he could
have gone further had he wanted to, such as pointing out specific instances
of the pitfalls and shortcomings he outlined in the session, but that would
not have really changed the point. My impression of the MeeGo community is
that everyone is well aware of what parts of the project are working
smoothly and where there are hiccups, strained communications, and difficult
processes. Fifteen months is not a lot of time to roll out a large,
multi-faceted distributed project like MeeGo, especially one that targets
multiple audiences (such as hardware makers, developers, and consumers).
Perhaps the best thing about Munk's talk was his ability to systematically break down the issues that various pieces of the MeeGo machine are struggling with. By shining some light directly on the problems, there is less chance of argument and more room for repairing the cracks — which is clearly what everyone wants. Other large projects could probably take a cue from Munk's taxonomy, and take a look at their own teams, subprojects, and processes: regardless of the target audience of the project, most in the free software world share the same set of goals.
Comments (5 posted)
As part of an interview in
LWN, Mark Shuttleworth is quoted as wanting the community to view the use of contribution licensing agreements (CLAs) as a necessary prerequisite for open source growth and the refusal by others to donate their work under them as a barrier to commercial success. He implies that the use of a CLA by Sun for OpenOffice.org was a good thing. Mark is also quoted accusing The Document Foundation (TDF) of somehow destroying OpenOffice.org (OO.o) because of its decision to fork rather than contribute changes upstream under the CLA. But I'd suggest a different view of both matters.
[ Disclosures: Having been directly involved while at Sun in some
of the events I discuss, I should say that everything recounted here is a
matter of public record. While I'd love to explain some of the events at
greater length, I'm not free to do so - such is the penalty of employment
contracts. Suffice to say that just because a company takes certain
actions, it doesn't mean everyone agrees with them, even senior people. I
should also disclose that I was honored to be granted membership in TDF
recently, although I'm not speaking on their official behalf here. ]
My experience of CLAs has led me to conclude that they prevent competitors
from collaborating over a code base. The result is that any project which
can only be monetized in its own right rather than as a part of a greater
collective work will have a single commercial contributor if a CLA is in
place. Mark would like us all to accept contributor agreements so that a
legacy business model based on artificial scarcity can be used to make a
profit that would otherwise be unavailable without the unrewarded donation
of the work of others - who are not permitted the same opportunity. I've covered this in greater detail in my article "Transparency and Privacy".
While pragmatically there may be isolated circumstances under which CLAs
are the lesser of evils, their role in OO.o has contributed more to its demise than offered it hope. By having a CLA for OpenOffice.org, Sun guaranteed that core contributions by others would be marginal and that Sun would have to invest the vast majority of the money in its development. That was a knowing, deliberate choice made in support of the StarOffice business models.
Despite that, others did work on the code - a testimony to the importance of OO.o. The most significant community work was localization, and the importance of the work done by the many teams globally making OpenOffice.org work for them can't be overstated. In the core code, Novell worked inside the community, accepting the premise Mark is asserting and contributing significant value in small quantities over an extended period. Smaller contributions came from Red Hat and others. IBM worked outside the community, developing the early version of Symphony, contributing nothing back to the OO.o community, and thus preventing any network effect from being accelerated by their work.
Sun used the aggregated copyright in 2005 to modernize the licensing arrangements for OO.o, closing the dual-license loophole IBM had been using to avoid contribution. Some (me included) hoped this would bring IBM into the community, but instead Sun also used the aggregated copyright to allow IBM to continue Symphony development outside the community rather than under the CLA. Stung by this rejection of transparent community development - and in distaste at proprietary deals rather than public licensing driving contribution - Novell started to bulk up the Go-OO fork, by ceasing to give their contributions back. This all happened long before the Oracle acquisition.
The act of creating The Document Foundation and its LibreOffice project did no demonstrable harm to Oracle's business. There is no new commercial competition to Oracle Open Office (their commercial edition of OO.o) arising from LibreOffice. No contributions that Oracle valued were ended by its creation. Oracle's ability to continue development of the code was in no way impaired. Oracle's decision appears to be simply that, after a year of evaluation, the profit to be made from developing Oracle Open Office and Oracle Cloud Office did not justify the salaries of over 100 senior developers working on them both. Suggesting that TDF was in some way to blame for a hard-headed business decision that seemed inevitable from the day Oracle's acquisition of Sun was announced is at best disingenuous.
So what's left of the OpenOffice.org community now? Almost all of the people who worked on it outside of Sun/Oracle have joined together to create the Document Foundation. As far as I can tell all that's left of OO.o is the project shell, with no development work in progress because Oracle has now stood down all the staff it had working on the project, plus one or two voices trying desperately to make us believe they still have some authority to comment because they used to be involved with it.
The CLA bears a great deal of responsibility for this situation, having kept the cost of development high for Sun/Oracle and prevented otherwise willing contributors from fully engaging. There is one hope left from it though. Because of the CLA, Oracle is free to choose to donate the whole of OO.o to The Document Foundation where it can be managed by the diverse and capable community that's gathered there. If they don't do that they probably need to change the license to a weak copyleft license - the new Mozilla Public License v2 would be perfect - to allow IBM to join the main community. It's my sincere hope that they will finally put the CLA to a good use and allow it to reunite the community it has divided for so long.
Comments (44 posted)
Due to the US Memorial Day holiday on May 30, next week's weekly edition
will be delayed by one day. That means that it will be published on
Friday, June 3. Several US holidays are arranged to land on Mondays and we
have finally decided to go ahead and take some of those days off. July 4
(Independence Day) and September 5 (Labor Day) are also holidays that we
plan to take off this year. Hope this doesn't inconvenience anyone too
much, and have a nice Monday whether you are celebrating the holiday or not.
Comments (2 posted)
Page editor: Jonathan Corbet
Inside this week's LWN.net Weekly Edition
- Security: WebGL vulnerabilities; New vulnerabilities in cyrus-imapd, feh, kernel, kvm, ...
- Kernel: What's coming in $NEXT_KERNEL_VERSION; Kernel address randomization; The problem with prefetch.
- Distributions: Linux Mint goes to 11; Fedora 15, MeeGo 1.2, RHEL 6.1, ...
- Development: Scale fail 2; 0install, Calligra, Libcloud, Jato, Pinpoint, ...
- Announcements: OVA, TD, Humble Bundle, WebP, OSI, Skype, Android, conference videos, ...