The theme of this year's Embedded Linux Conference (ELC) is "Ubiquity" and Dirk Hohndel opened the conference with a keynote on just that topic. Hohndel, Intel's chief Linux and open source technologist, looked at how widespread Linux is in consumer electronics, but also how many other, far less obvious devices it has been embedded into. In addition, he discussed some of the problems caused by vendors and manufacturers not engaging with the community and how that can lead to suboptimal devices; he also reviewed the value proposition of Linux, pointing out that the "zero-cost" is not zero and even if it were, that's not where Linux's strengths lie.
Tim Bird, who organized the conference for the CE Linux Forum (CELF), introduced Hohndel by calling him "something of a legend in the open source community" for his work in the community over the last 15 years or more. Bird recalled some advice Hohndel had given him some years ago that the secret to having an organization that works well with open source is to keep everything open. It seems tautological, but that is exactly what Bird and CELF have done with great success. CELF keeps all of its information available to everyone, not just members, and welcomes the participation of the community. In large part, that came straight out of Hohndel's advice.
Hohndel opened his presentation by contrasting the ELC participants with those that had attended his talk at the Open Source Business Conference (OSBC) earlier this year. Based on the traditional show of hands, he noted that there were far fewer lawyers and far more people who had used Linux at ELC than at OSBC. But part of the point he was making is that even though the folks at OSBC didn't think they were using Linux, they almost certainly are—and on a daily basis.
He noted that based on his title, "Ubiquitous Linux", and the dictionary definition, he could give the shortest keynote on record: "Linux is everywhere, thank you very much." He pointed out that servers were the first commercial success for Linux, but were not really the purpose for which Linux was created. Linus Torvalds's lack of a desktop Unix was what really spawned Linux. Now, though, you can hardly do anything on the Internet without bumping into Linux.
If you want to search (Google), buy a book (Amazon), book a flight, trade NASDAQ stocks, or participate in an auction (eBay), you are dealing with Linux. In fact, Hohndel says, "if you can spend five minutes on the Internet and do not run Linux, you're a genius". He asked for a show of hands to see how many Eee PC owners there were in the audience; he was disappointed to only see about seven. He claimed that even the lawyers at OSBC had more of these systems. But computers are boring, Hohndel says, everyone knows Linux runs on computers.
With about three minutes invested in a Google search, Hohndel was able to come up with 22 phone vendors with a Linux phone before he stopped looking. Most of those vendors are in Asia and he was not sure why the US was "behind the curve". Part of it is that vendors don't talk about Linux on their phones, he said. Half of the Motorola Razr phones run Linux, but you wouldn't know that based on the marketing.
Hohndel then started listing things that run Linux but are far less known, starting with MRI and CAT scan medical devices. Vehicles of all types use Linux in their avionics (airplanes), in-vehicle entertainment systems (planes, trains, and cars), repair centers (cars). He noted that for redundancy purposes, the avionics vendor's system had a matching implementation in Windows, which wasn't necessarily very comforting to Hohndel. He noted that he had written kernel code along the way and wasn't completely comfortable with that code running planes he flies on either.
Hohndel challenged the audience to see how long they could go without interacting with a Linux system, listing the kinds of things one would have to do without. He also related a conversation he had with an executive from a large, unnamed, software company (perhaps located in the Pacific Northwest) who claimed he never used Linux. By the end of the conversation, they had come up with a dozen different Linux systems he used on a daily basis. "I don't think the general public realizes how much of this scary stuff they have around them," he said.
He then turned to the question of why Linux is everywhere. He noted that "because it's free" is the "worst possible answer" but this supposed zero cost is an answer that is frequently given by other audiences of his talks. "If you are using Linux because it's free, you are in for a very rude awakening".
Intel has hundreds of Linux developers, he said, but "we didn't get the memo, we actually pay these guys". He started listing some of the costs associated with using Linux: hardware likely doesn't come with Linux drivers, or the drivers only work with a different version of the kernel than the one needed to get other kernel features necessary to the product, etc. And "then you talk to your lawyers" about licenses and such. None of that is free.
The strength of Linux is that if you run into a problem, you can solve it yourself or hire someone to do it for you. He contrasted that with a proprietary solution where you pay $20,000 up front and some small per-unit royalty, which might actually be cheaper, at least on paper, but if there is a problem, you have no leverage with the vendor. How can you meet your market window when you have no way to fix problems that you encounter, he asked.
Choosing Linux is about customizability as well as security, Hohndel says. You can control the footprint of the system because you have the source code and can customize it as needed. Anyone who says they are going with Linux because of zero cost have proven to Hohndel that they don't understand what you can do with Linux.
This leads to a problem with the current crop of consumer devices that run Linux: they don't take advantage of the strengths of the OS. Hohndel thinks that most vendors using Linux in embedded devices are doing it wrong. They are focused on the price to the exclusion of building a community around the device. They don't make any money on a device they have already sold, so they focus on the next device and ignore the idea that just by opening things up, they could build a community that would help them sell that next device.
Hohndel said that being open to the community will reap many benefits. New features and functionality will be added by others to "do things you never thought possible". He mentioned Linksys wireless routers and the DD-WRT and OpenWRT communities that have sprung up around them. Linksys got many things wrong in the early going, but eventually turned that around. Companies need to recognize that there many more smart people outside of their company.
Intel is trying to lead by example, to some extent, with its Moblin efforts. Intel has turned the "stewardship" over to the Linux Foundation, but it is in no way abandoning it. According to Hohndel, engineers have been added to Moblin and the company would like to see what else the community can do with it. There are lots of things Intel hasn't thought of, "but the community will, and we hope they do."
Hohndel's talk didn't cover too much in the way of new ground—much of what he said has been bandied about before—but he tied the ubiquity of Linux and the foot-dragging of vendors with respect to the community in an interesting way. For a number of years, folks have been talking about ways to get Linux into more devices of various sorts; that battle has been won to a large extent. The next step is to bring the device manufacturers into the community; that battle has only recently started. One senses that Linux and the community will win that one as well.
The GNOME desktop environment made its 2.0 release in June of 2002, and quickly established a six-month cycle between stable releases. Now the release team has drafted a plan for GNOME 3.0, tentatively to arrive two cycles from now in March of 2010. A few user-visible changes are slated to appear, accompanied by far more refinements in the dependencies, language bindings, and the structure of what constitutes the core of GNOME.
Vincent Untz sent a planning statement to the GNOME desktop development mailing list on April 2 to outline the big issues and the release team's plan. The discussion is of course ongoing, but the basic idea consists of three components: new technologies that will directly affect the user experience, structural changes to the modules and module sets that define GNOME, and ways to promote GNOME in hopes of growing the surrounding community.
GNOME Zeitgeist is a non-hierarchical file management system. Rather than finding files based on their location in the filesystem, Zeitgeist provides a suite of alternative interfaces to use: a last-accessed-on calendar, an easy-to-use bookmark system, tags, and content-type filters. Other entry points are in the works, including attached comments and location awareness (as is "files last opened when I was in Barcelona").
Developers will probably be more interested in changes to the platform itself, including shuffling out of old libraries, inclusion of new ones, and possible changes to GNOME's module sets.
According to Andre Klapper, several deprecated libraries will be removed in 3.0, such as the sound server esound, the file system layer libgnomevfs, 2-D graphics library libart_lgpl, and printing library libgnomeprint. Some new libraries will be introduced, including the aforementioned Clutter, GeoClue for location-awareness, and libchamplain for easy rendering of maps. Also new is the idea of "staging" level libraries; components hopefully in transition towards full support in a future GNOME release, but still making API or ABI changes. The key example here is GStreamer, which is widely used and enjoys tremendous support form the GNOME community, but still undergoing rapid development. The project also hopes to encourage developers to increase the usage of non-GNOME dependencies like D-Bus and Avahi.
More subtle changes are likely to come in the way GNOME is packaged. The current and long standing scheme divides the code into "module sets," as Untz and Lucas Rocha explained. Each set contains libraries and applications for a particular usage profile: the desktop, mobile, developer tools, platform bindings, and so on. One concern is that the current module sets make unnecessary divisions. Untz noted that the "developer platform" modules set (which contains C bindings) should really be combined with the "platform bindings" module set (which contains bindings for other languages); keeping them divided adds to the perception that C is blessed, but other languages are not.
Another problem is that the module sets have slowly become too rigid over the course of the 2.x releases, no longer encompassing all of the applications in the GNOME community, and perhaps unintentionally communicating that some applications are "official" while others are not.
Untz and Rocha cited several examples where two or more excellent applications of the same type seem to compete, including Rhythmbox and Banshee, Empathy and Pidgin, and gThumb and F-Spot. If the project selects one for inclusion in the desktop module set, it inadvertently slights the other, which is not the intention. "My personal opinion is that we should have a very small 'desktop core' module set and that's what we build and package 'officially,' then we have a separate process of certificating apps as 'GNOME-compliant.'" said Rocha. He continued:
"There's also the fact that we can create a 'brand' that can easily be recognized by users: if it's a GNOME app, then it's good," added Untz. Both agreed that the project should take steps to be more welcoming to outside projects and contributors.
The third focal point for 3.0 is promoting GNOME better. To be sure, widening the community through reorganizing the module sets will draw in more developers, but as Untz observed, developers are only one of three target audiences. The others are users and vendors — a group that includes Linux distributions and mobile device makers.
In recent years, Untz felt that the project has not had a "crystal clear message." It has a good message with respect to usability, accessibility, and internationalization, but is not as coherent at presenting the GNOME desktop platform as a whole.
Exactly how to proceed is not as clear as identifying the issue. GNOME is a large project, but it is in the middle of the stack — neither a single application that a user can try out with a simple package install, nor a full operating system comprising a complete solution. GNOME Foundation executive director Stormy Peters observed that the vast majority of GNOME users use the environment courtesy of an install-time option from their Linux distribution. "That said, if a user has a question about a GNOME feature or application and search for it on the web, they are likely to end up either on the GNOME pages or a random distributor site (not necessarily the distribution they are using.) We want to make sure we are ready for those people."
The promotional effort will entail revamping the gnome.org web site, and a concerted effort from the GNOME marketing team. Untz summarized the challenge, "the hardest part (setting goals) is done. Now, it's about achieving them, and even doing more than that if we can."
With two full release cycles in which to work, the discussion is just the beginning. There are several additional changes that could make their way into GNOME 3.0, including replacing the aging gconf configuration system with dconf, and an emphasis on "social desktop" technologies like the Telepathy framework.
Rocha and Ken VanDine both observed that they hope the 3.0 cycle widens the GNOME community. Rocha said he would like to see more space for experimentation, and VanDine added that he would like to see GNOME project infrastructure (such as git and bugzilla access) opened up to individual developers, so that they could create their own branches, propose merges, and participate.
However it plays out, Peters is confident it will reflect the wishes of the GNOME community. "The interesting thing from my perspective is how this is done in an open fashion. Before plans are even finalized we are reaching out to the GNOME community, our partners, the distributions and even users to hear everyone's input and to involve them in the process as much as they wish to be. It's likely that we'll hear a lot of dissenting opinions in the process — that's part of a good discussion — but the end product will be even better for it." The members of the release team are tracking the process in public; you can follow the roadmap as it develops from the GNOME project's wiki.
Anybody who has administered Unix-like systems for long enough has probably ended up swinging from that rope at least once. So one would think that there might be support for work which reduces the potential for self-hanging. And indeed there is, but that doesn't mean that all such changes are welcome.
Readers with a lot of spare time and a desire to wander into email flamewars could probably occupy themselves with this fedora-devel thread for quite some time. It seems that the X.org developers recently decided that the three-finger salute (alt-control-backspace) should no longer, by default, immediately kill the X server. The reasoning behind this change is clear enough: it can be really irritating to hit the wrong key sequence and watch all of one's work evaporate before one's eyes. Besides, the environmental costs of replacing all of those thrown-across-the-room keyboards is increasingly hard to justify.
Unfortunately for the polar bear population, the change inspired a rather severe storm of flying keyboards in its own right. A certain Gerry Reno complained on fedora-devel that Fedora should have overridden X.org's decision regarding this key sequence. Unsatisfied with the hundreds of responses found there, he took the discussion to the X.org development list, wherein he claimed:
So, it seems, we have a conspiracy of Emacs users working to deprive the wider user community of a useful tool. Daniel Stone, the developer who committed this change, denies this charge:
(It's worth noting that the Fedora Weekly Webcomic blames a different conspiracy for this change).
In truth, it's clear that a number of reasonably capable users have, at times, lost work as a result of hitting this key sequence by mistake. Enough of those users complained that the X.org developers looked at the issue, and, according to Matthew Garrett, "Everyone involved agreed that not having a keystroke that caused immediate data loss was a sensible idea." So, while many of the world's ills can legitimately be blamed on Emacs users, that would not appear to be the case this time around.
A reversal of this decision is unlikely. But the development community would still like to accommodate users who feel the need for the full length of rope. Said users can reverse the default in their xorg.conf file now, of course. The openSUSE approach has been to require that the sequence be hit twice before bringing the world to an end, but it's not clear that other distributors will follow suit. There has been discussion of moving the action to a key sequence which is harder to hit by accident. There may eventually be a per-user configuration option to enable this behavior as well, though that will require some X server changes first.
Meanwhile, Ubuntu developers have cut off a classic piece of Unix rope by boldly disabling the "rm -rf /" command. It seems that the rm command has a --preserve-root option which prevents the removal of the root directory. In Ubuntu, this option was not enabled by default, leading to the bug filed by a concerned user. The distribution's developers agreed that the ability to remove the root directory was not a particularly useful feature, and, additionally, that issuing an "rm -rf /" command was easier than one might expect - poorly-written scripts are evidently a common source of that kind of mistake. So, in October, 2008, they made --preserve-root the default for the Intrepid and Hardy releases.
Some months later, we have started to see complaints like this:
Those who are concerned about this change have more to worry about: it would appear that Fedora has followed suit. Even so, the rope has not been shortened by any great length; those wishing to hang themselves can use any of a number of alternatives, including:
rm -rf /. rm -rf ~ rm -rf *
and so on. And, of course, the --no-preserve-root option remains available for those to can't think of any other way to destroy their systems.
But is this contrary to the Unix philosophy? If so, one should certainly complain about the much more obnoxious
alias rm='rm -i'
.bashrc entries that Fedora has been inflicting on the root account for years. That is the sort of change that trains users to blindly agree to anything the system asks; your editor (who immediately removes such things) feels that overall user safety is not improved by asking "really do this?" questions all the time.
The truth of the matter, though, is that Linux has moved beyond the "hardy pioneers on the dangerous frontier" stage. Simple ability to hang one's self is of limited value even to pioneers; it is positively detrimental to those who come after. It is not surprising that developers and distributors are trying to disarm some of the most surprising and least useful booby traps in the system. That process is likely to continue. But this is still Linux, so those of us who feel the desire will always be able to break out the full length of rope; we'll just have to remove the warning label first.
Recently, Openmoko CEO Sean Moss-Pultz announced at OpenExpo in Bern that the company was reducing staff and postponing the development of the GTA03, its first consumer-oriented phone, in favor of an undefined Project B. Available as a YouTube video, the announcement was a confirmation of recent rumors that the company was in trouble. In fact, many concluded that the announcement meant the end of the company, or at least the beginning of the end.
Either conclusion is premature, but the announcement does highlight the problems Openmoko faces as a business, as well as its uncertain future. These problems are evident not only in the company's history, but in Moss-Pultz's announcement and the company's web site as well.
Openmoko began in 2006 as a project within First International Computer (FIC), a Taiwanese computer manufacturer. Soon spun off into a separate company, Openmoko became the center of a small but active community, due largely to its intention of using only free software and free hardware. Its popularity was helped by the fact that, prior to the announcement of Android in November 2007, it was the first effort to introduce free software into the mobile phone market.
The company and community began work on Openmoko Linux and the hardware to run it on. As a development community, Openmoko has had some success, with GNU/Linux, FreeBSD, and L4 kernels ported to its devices, as well as versions of the Google Android operating system and a number of utilities and games.
However, as a commercial manufacturer, Openmoko has struggled continuously to coordinate its software and hardware in all its products, up to and including the GTA03. As Moss-Pultz explained in February 2007, "each hardware revision takes at least one month of time. Each month without stable hardware means serious delays for software. One time we received the wrong memory from our vendors and we failed to catch this before production. Another time some key components ran out of supply."
Despite such difficulties, in July 2007, the company produced the Neo 1973, a development phone, following it with the Neo FreeRunner in June 2008. According to Moss-Pultz in his announcement last week, the Neo sold 3,000 units, and the FreeRunner 10,000 units. These are modest numbers that, more than anything else, indicate just how small a player Openmoko is.
Openmoko's progress has not been helped by the countless complaints and problems about the FreeRunner, all of which also affect the development of its successor, the GTA03. For one thing, the phone does not support 3G standards for telecommunication hardware. In his announcement, Moss-Pultz explains this lack as being due partly to the difficulty of implementing 3G without using proprietary software and hardware, and partly due to the fact that doing so would increase the cost by at least two-thirds. But, although these are sound reasons, without 3G support, Openmoko's products are inevitably going to be seen by customers as inferior to other mobile devices.
Moreover, if you look through the Openmoko community mailing list over the last few months, very few aspects of the FreeRunner have escaped being mentioned in bug reports.
Many of these bugs have been collected on the Neo FreeRunner Hardware Issues page on the community wiki. Active bugs include poor audio quality, the inability to boot without a charger, the corruption of the SD card's partition table when using the suspend function, incompatibility with SIM cards, problems with the GPS feature, unreliable reporting of the battery charge, and short battery life — and this is far from a complete list. Workarounds exist for some of these problems, but the disheartening cumulative effect is suggested by the desperate-sounding plea near the top of the page: "Please DON'T PANIC when reading this page. Please give Openmoko employees time to investigate these issues and to develop a solution." Even making allowances for the fact that the FreeRunner is not intended for general consumers, such problems give it the appearance of having been released before it was ready.
With such a history, nobody should be surprised that the company has recently seen an exodus of many of its employees. To what extent these departures were voluntary or layoffs is uncertain.
But, either way, they increase the difficulties for the company. Harald Welte, the former Lead System Architect at OpenMoko, wrote in his blog, "There used to be really great engineers at Openmoko some time ago, but at least a number of good, senior folks are no longer working there at this point in time, or are working on a much smaller scope for Openmoko Inc." In addition, Welte suggested that, by not making any public statements about the departure of key staff, Openmoko is contributing to the rumors and uncertainties that already surround it. Increasingly, the impression of a struggling company is becoming impossible to avoid.
In the video, Moss-Pultz talks candidly about the challenges that Openmoko faces and the mistakes it has made.
Some of the challenges are ones that no company can do much about. For example, Moss-Pultz began by explaining that, while small companies or individuals can disrupt software markets, the expense of developing new computer chips and the difficulty of finding a place to manufacture them means that existing companies have a practical monopoly on hardware.
Later in the video, he revealed that, although Openmoko had enough monthly sales to break even by the end of 2008, the recession has caused a serious decline in sales in 2009. Having not anticipated this downturn, the company is left with a large inventory of unused hardware components, a depreciating investment that can only be recouped by sales. Meanwhile, extra inventory may incur storage costs if the company is like many high-tech startups and lacks its own warehouses.
At the same time, Moss-Pultz acknowledged that the company has made tactical errors. He suggested that the company has been slow to realize that "you can't compile hardware" — by which he apparently meant that fixing errors in hardware is much more expensive and time-consuming than debugging software. In addition, Moss-Pultz said that the company had tried to develop too many markets at the same time. It has also attempted to manage direct world sales by itself, rather than going through an established distributor, an effort that has caused it endless time and effort in dealing with custom duties and varying regulations.
More specifically, Moss-Pultz pointed to two direct mistakes. First, Openmoko could have sold more than 3000 Neos if it had not been overly conservative in ordering components (a situation that might lead observers to wonder whether the overstocking of the FreeRunner was over-compensation for this earlier error). Second, in wishing to honor its commitments, the company spent months directing what Moss-Pultz estimates as 90% of its resources to an unspecified single contract. This situation was a particular drain on resources because it occurred after Openmoko became a separate company and could no longer draw upon the resources of FIC.
All these events, both external and internal, have taken place against a background of both too little and too much publicity, according to Moss-Pultz. On the one hand, outside of the free software community, Openmoko remains little known among mobile manufacturers and distributors. This admission suggests that the company has been doing little or no advertising in its market niche. On the other hand, within the free software community, Openmoko was widely hailed as "the iPhone-killer," even after the company tried to explain that it was not trying to compete against Apple's popular device. Such a view created inflated expectations that a company of less than sixty employees could have no hope of matching, even if it made no mis-steps. It may also have pressured Openmoko executives into making hasty decisions, although Moss-Pultz did not mention such a possibility.
Listening to Moss-Pultz, what seems clear is that, for all the surrounding buzz, Openmoko has suffered largely from an inexperienced team that was learning as it staggered towards market. This impression is confirmed by the company web site, which is surprisingly sparse and unprofessional for a tech company shipping products.
Even by the sometimes eccentric standards of free software, the site seems strangely incomplete. The site's front page has no explanation of what the company does. Even the About page contains only the translation of a Chinese poem and a few flowery generalities, and no mention whatsoever of the management team.
Nor can you buy directly from the company. Instead, clicking on the Store link in the menu takes you to the distribution page, where you find that Openmoko has only six distributors in the United States and Canada, fifteen in Europe, and one in Asia. Many of these distributors are obviously minor. That brings up another problem for Openmoko: As countless other startups have found, you cannot get major distributors to carry your products unless you have a track record, but you can hardly hope to get a track record unless the major distributors carry your products.
The point is, the company web site creates an impression of a company that is not ready to do business. By comparison, the Openmoko community page is far more detailed, which suggests that Openmoko executives are far more comfortable in a community of developers than in a board-room. In this light, the company's problems and mistakes seem completely understandable.
The necessity for Moss-Pultz's announcement was spelled out by an email to the Openmoko community by vice-president of marketing Steve Mosher. The company can only make money through completing development on either the GTA03 or the mysterious Project B, but cannot afford to complete both just now. Given that completing the GTA03 would cost three times as much, the sensible choice is to focus first on Project B.
This logic seems clear enough. Yet it would be clearer still if anyone gave an indication of what Project B actually was. All Moss-Pultz said on the subject in his video is that "I always have a backup plan, no matter what I do" and that this focus is a "short-term adjustment." Observers have suggested that Project B is a "non-mobile / non-smartphone" or, alternatively, that it will involve using Android to reduce development costs. But the truth is that no one has any concrete information.
No matter what Project B turns out to be, whether it can turn Openmoko around remains to be seen. Welte commented that, "Over time, I have started to have severe doubts whether Openmoko Inc. is really the most productive and/or best environment to do this kind of development. Priorities and directions changed a lot." But at least with this announcement, he added, "I no longer have to hope that Openmoko Inc. gets their act together to actually get an (to my standards) acceptable product out into the market." Possibly, Welte can be dismissed as a disgruntled former employee, but, by the time that a company makes the sorts of cuts that Openmoko has made, the chances of reversing the slide into bankruptcy seem small.
Certainly, the attempts by Openmoko executives to reassure everyone have not convinced most observers. Many, such as Nilay Patel at engadget, wondered whether news indicate the lack of a market for free hardware devices in the mobile market.
However, such speculations are based on too little evidence. At least two-thirds of all startups fail within ten years, and the rate is generally assumed to be even higher in technology companies. Openmoko is only a single company in the mobile space, and, if you look at Openmoko's past record, nothing indicates that the company's problems are due specifically to its business plan. A simpler explanation is that the problems are due simply to inexperience and poor decisions. In the end, Openmoko's future depends far more on its executives' ability to learn from past mistakes than on their choice of ideals.
Page editor: Jonathan Corbet
Copyright © 2009, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds