LWN.net Weekly Edition for July 23, 2009

The grumpy editor's e-book reader

Your editor recently "celebrated" yet another birthday; one asks "which birthday?" at the risk of making him grumpy indeed. During that celebration, a surprising present turned up, in the form of an Amazon Kindle book reader. That presents an opportunity to play with a new toy, something your editor is not known for turning down, even when the toy is as problematic as the Kindle. In the process, your editor turned up some free software which helps to make the device rather more useful.

The Kindle is a device which inspires mixed feelings. It is a nice piece of hardware, showing what can be done with electronic-ink displays attached to a Linux-based system. It is small, light, and able to store a long list of books. The display is nicely readable, even in strong outdoor light, though it is also somewhat slow to respond and changes pages with a distracting black flash. The built-in cellular modem makes the acquisition of books from almost anywhere easy; no computer is required. The keyboard is awful, but one does not normally expect to do a lot of typing on such a device.

What one does expect to do with it is to read. There is a great deal of written material on the net, much of it available under free (or, at least, "freely distributable") licenses. But your editor cannot be the only one who, despite being an avid reader, lacks enthusiasm for reading a novel from the computer screen after spending the working day staring at that same screen. Kindle-like devices offer a solution to this problem; they are portable and much easier on the eyes. They are still not as nice as a real book, but they are nice enough to make much of that online content more accessible. Electronic book readers are an interesting class of gadget, even for those of us who have no real interest in ditching words printed on dead trees.

Your editor's Kindle showed up just a couple of days after the widely-reported "memory hole" incident, in which Amazon deleted copies of two George Orwell books from the devices of customers who had "purchased" them. LWN has said little about this event for the simple reason that there is very little to add; the unbooking of 1984 must be shocking even to the most irony-challenged among us. But one could well add that Amazon has made it clear that running Linux does not necessarily make a device serve its owner. The Kindle is very much a closed, captive platform under the control of its vendor. As far as your editor can tell, nobody has yet claimed to have achieved root access on a Kindle 2 device.

Managing the device

One of your editor's first questions was, naturally: can the Kindle be loaded with books which do not come from Amazon? Many of us have materials in PDF format sitting around already. A lot of free software documentation is, for some strange reason, not available via the Amazon store. That is also true of the bulk of the public-domain and freely-licensed material which has been made available on the net. There are good reasons to put all of this stuff onto a reader where it will be available, someday, when the urge to read it strikes. There are a few other motivations as well: avoidance of payments, avoidance of DRM, privacy, protection against having material deleted by Big Bezos, etc.

The Kindle has a USB port, and it shows up as a normal USB mass storage device. So gaining access to the library from a Linux machine is a straightforward thing to do. Among other things, this access can be used to back up one's books, perhaps giving a degree of permanence to a book collection which, otherwise, appears to be subject to a discouraging degree of control from outside. It would also be nice to be able to place new files there; that, too, is possible, with one little problem: the Kindle 2 lacks a PDF reader.

Now, one could come up with no end of choice words for whoever thought that an electronic book without PDF capability made any sense whatsoever. But it's better to do something about it. Your editor spent some time searching for answers before stumbling across an interesting program named calibre. This GPLv3-licensed, Python-implemented, multi-platform program aims to solve the the PDF problem, and quite a bit more as well. It is, in fact, a general electronic book library manager with support for a number of reader devices.

Installing calibre was only mildly painful. It requires Python 2.6 and a fairly wide range of libraries; your editor's Fedora 11 machine (the time and courage to return to Rawhide have been lacking) does not have a calibre package, but all of the dependencies are available. The installation instructions are based on the idea that feeding text from a web site directly into a root shell is a good idea, but one can get around that. There is also the installation of a udev rule which sets the permissions for USB-connected Sony readers (for the Kindle, the program doesn't open the USB device directly).

Let it be said: calibre (version 0.5.14) needs a lot of work. The interface is strange and requires a certain amount of figuring out. If it can't communicate with the reader, there is no real information as to why (one hint: it expects that the device filesystem will be automatically mounted at plugin time, something your editor has disabled on his system). It will happily tell you that a given book is not available in the right format for the reader, but is short on information on how to get it into an appropriate format. One needs to be prepared to spend some time just messing with it. These gripes notwithstanding, calibre has the makings of a nice tool.

The main screen is dominated by a list of books in the library; clicking on the "reader" icon at the top yields a list of books on the device instead. One could imagine a useful combined listing mode which showed all books, with a concise indication of where they are to be found, but calibre does not do that. There's a pretty - but relatively useless - "browse by cover" mode. As with the Kindle itself, the book list is a single, flat listing, with no provision for organizing the books into hierarchies. This can only get painful as the list of books grows. Yes, this is 2009, we do everything with tags now, and calibre supports tags. Your editor would still like directories. Call them "bookshelves" if that fits the theme better. It would also be really nice if calibre could treat a group of files as all being part of a single book. No such luck.

The "view" operation can be used to read a book on the Linux system. It opens an internal reader for a number of formats; this reader seems to fail, silently, fairly often. For PDF files, calibre just starts evince, which works just fine.

The "send to device" button is, naturally enough, the way to move a specific file (or set of files) to the reader. There doesn't appear to be a "just keep the two in sync" option; books all must be loaded onto the reader explicitly. It would sure be nice if "send to device" would just convert the file into an appropriate format if need be, but that doesn't happen; it throws up a dialog saying that the transfer isn't possible. In other words, the user must explicitly perform a format conversion on (say) a PDF file before it can be sent to the Kindle. "Convert E-books" does that, providing a nice set of options on how the conversion is to be done. It works, but it should work automatically.

The conversion of PDF files into the "MOBI" format understood by the Kindle works reasonably well, but the books suffer somewhat in the translation. Paragraph breaks tend to vanish, page headers and footers get mixed into the text, and so on. The formatting of code samples loses little details like indentation. All told, the result isn't quite what it should be; it seems like it should be possible to do better.

One nice feature built into calibre is the "fetch news" operation. The program can go to the web sites of a large number of publications, download the current edition of whatever news is published there, and convert it into a format suitable for loading into the reader. If you leave calibre running, it can perform regular downloads, keeping the library populated with current newspaper and library editions. Needless to say, this feature is appealing when compared with the payment-required newspaper offerings from Amazon. On the other hand, as a web publisher, your editor does have a certain affection for the "payment required" mode of operation.

Also worthy of note is Savory, a repackaging of the calibre format-conversion code which runs on the Kindle itself. This tool lurks on the device as a daemon; whenever it sees a new PDF file, it goes off and converts it to the MOBI format automatically. Getting Savory to work can be a bit tricky, but, once the right incantations have been made, it works as advertised. The process is quite slow (the Kindle is not known for the data-crunching power of its CPU) and the end results are, not surprisingly, about the same as those obtained by using calibre directly - with one difference. Savory's conversion process actually makes two copies, one of which is a series of PNG images reflecting the real appearance of the source PDF file. The images have their own problems (they are not amenable to searching, for example), but they do look nicer. In summary, Savory is a nice enhancement, but it's in no way the same as having the device be able to just display PDF files natively.

Source

The Kindle is based on GPL-licensed software. Since Amazon is distributing this software with the device, it is required by the GPL to either include the source with the device or include a written offer to provide the source. Your editor read through all of the fine print, including absolute restrictions on modifying the device and discouraging stuff like:

Big brother does, indeed, know what you are reading. But your editor could not find the written source offer. So, technically, it would appear that Amazon is in violation of the GPL. That said, Amazon has made the source available for each version of the operating software shipped with Kindle devices; one simply needs to know where to look for it.

That source distribution takes the form of a 140MB compressed tarball, which, in turn, contains 44 other compressed tarballs. Among other things, Amazon has tossed in the source for the 2.6.22.19 kernel, GCC, bootchart, powertop, and iptables. There is no separate patch to the kernel, but a quick diff shows significant changes, mostly in the form of the addition of support for the Freescale i.MX27 and i.MX31 processor architectures. The i.MX27 code is not currently upstream, even in the 2.6.31-rc kernels. Amazon has patched in drivers for a Freescale PATA controller, i2c controllers, the Kindle "five-way controller," the "Fiona" keyboard, a "magnetic sensor" device, a trackball device, a video output device, a "run time integrity checker" device, a number of electronic ink devices, and far more. There's also the yaffs2 filesystem and, bizarrely, a version of Andi Kleen's superseded unlocked_fasync patch. None of this code is upstream, and there appears to be little interest in getting it there.

Sadly, the Kindle is a closed device, so there is little point in trying to build and boot this code. That integrity checker device seems likely to get in the way. The unhackable nature of the device does not come as a surprise; that is how things tend to be done these days. But one can still wish that things were different. A user-modifiable Kindle would not just be more resistant to Orwellian monitoring and control; it could also be extended in ways that Amazon never dreamed of. Maybe it could even get a PDF reader. What a fun device that could be.

Comments (58 posted)

Fighting small bugs

Paper cuts, points of pain, obstacles and annoyances — whatever description you prefer, in the last six weeks, small bugs have started receiving closer attention from developers of the free software desktop. In Ubuntu, they are the focus of One Hundred Paper Cuts, and in Fedora of Fit and Finish, but in both cases, the goal is the same: to significantly improve the user experience by tackling bugs that can be quickly corrected. As an important side effect, these efforts are also allowing free software developers to approach usability in new ways.

Concentrated efforts at bug-squashing have a long history in free software development, so the idea of focusing on small bugs probably has multiple origins. However, one origin is the 0.8 release of GNOME Do in January 2009. The release fixed 111 bugs — over three times the number fixed in the 0.8.1 release — and received enthusiastic feedback from users.

When GNOME Do leader David Siegel joined the Design and User Experience team at Canonical a few weeks after the 0.8 Do release, he took the small bug meme with him. The Design and User Experience team is the group within Canonical whose task is to realize Mark Shuttleworth's challenge of improving the Ubuntu desktop, and Siegel soon saw similarities to Do: "We started to notice all these small things that were in each release that were never getting fixed", Siegel said. "And I said to Ivanka Majic, who's the leader of the Design team, 'We need a way to red flag these things that are obviously wrong, and make sure they are fixed before they go out.'" Within a couple of months, Siegel found himself leading One Hundred Paper Cuts.

According to Red Hat employee Matthias Clasen, a similar situation exists in Fedora. "There is a challenge in working on the Fedora desktop between feeling squeezed to finish cool new features in time for the next release, and fighting to get Rawhide [Fedora's development branch] into a somewhat working state, with too little time to devote to fit and finish" — that is, to polishing and removing rough edges. Like Siegel, Clasen now finds himself at the head of an effort to provide that missing attention to detail.

Whether Fedora and Ubuntu influenced each other in these developments is uncertain. However, in both cases, the advantage of focusing on small bugs is obvious. As Siegel explained, "They're low-hanging fruit. They allow us to quickly, inexpensively improve the user experience. We don't have to create new interfaces; we just have to fix these tiny, trivial bugs. It's just a small component, but it's something that can have an immediate impact."

Same problem, different approaches

Despite the similarity of the problems, Ubuntu and Fedora have organized their solutions in somewhat different ways. At Canonical, the One Hundred Paper Cuts project set a goal of addressing one hundred bugs during the development cycle of the upcoming "Karmic Koala" release, which is scheduled for October 2009. To narrow the focus enough to make it manageable, the project decreed that the bugs would center around what Siegel calls "the space between applications" or features such as the panel and the Nautilus file browser. Users were invited to report a paper cut bug via Ubuntu's Launchpad, and the initial one hundred paper cuts were chosen from the several thousand that were submitted.

For this first effort, the project chose (despite its name) to divide its efforts into ten rounds of eleven bugs each, ten in each round concerning Ubuntu's main GNOME desktop, and one concerning Kubuntu, the Ubuntu KDE variation. The project also defined what it would cover with extreme clarity. According to Siegel, a paper cut is a bug that users would encounter with a default installation from a Live CD. Although it would not actually prevent users from completing everyday tasks, it might cause momentary annoyance or distraction.

Theoretically, a paper cut must be correctable by one developer in one day. But, in practice, Siegel said, "We set a sort of gray area. For example, some of the paper cuts we've confirmed and want to fix for Karmic really take weeks to fix, but the work is going to take place anyway, and the paper cut is just a little bit of extra work beyond that."

By contrast, Fedora's Fit and Finish project chose a less formal approach. As Clasen explained, Fedora's Quality Assurance team was already in the habit of holding test days, in which participants prepared by downloading designated software and discussing what they found on IRC. Fit and Finish simply decided to hold its own test days during the development of Fedora 12, based on input submitted to the project's Bugzilla or by email. For each test day, instructions for participation are listed on a separate page, and both bugs and "things that work right" are summarized on the page after the discussion.

One difference from Ubuntu is that Fedora chose "to focus on user tasks, as opposed to individual features," according to Clasen. Nor did Fit and Finish limit the number of bugs to be covered in a single test day or to be fixed. However Clasen did add, "I realize that numbers — like the points awarded by bugzilla.gnome.org — can be a strong motivation, so we may revisit this at some point."

Early results

To date, Fit and Finish has held just one test day on the subject of display configuration, although pages for the topics batteries and suspend and peripherals are already posted. Clasen noted that this first day did not have "an overwhelming participation." He added, though, that the relatively low turnout was probably due to the fact that many desktop developers were at the Grand Canaria Desktop Summit that was in progress on the same day. A week later,the bugs arising from the testing day have been filed and assigned to a tester, but none have been closed.

In comparison, One Hundred Paper Cuts has had two rounds so far, and the bugs to tackle in each future round are already posted. Of the ten bugs in the first round, seven are now listed as fixed, one as having had a fix committed, and another two as "in progress," with only one listed as incomplete. In the second round, which finished on July 11, two are marked as fixed, another three have a fix committed but not yet applied, four are in progress, and two are listed as confirmed. The preliminary appearance is that One Hundred Paper Cuts is producing quicker results, possibly because its goals are better defined.

However, a disadvantage of the One Hundred Paper Cuts approach is the appearance it creates that some bugs are being given special treatment. For this reason, Siegel felt compelled to emphasize that "Paper cuts are just an additional level of importance being attached to bugs. A lot of people whose bugs get rejected for paper cut status will get angry and frustrated and say, 'Does this mean that the bug's not going to get fixed?' But it just means it's not going to be the focus. Many, many bugs will be fixed for Karmic; this set of one hundred is just getting a little extra push and a little extra attention."

As might be expected given Fit and Finish's more narrow topic, the bugs it generated fall into recognizable categories. "Naturally, a lot of the bugs that we have collected on that day are X driver bugs, bugs in the GNOME modules that play a role in display configuration (the display capplet, gnome-settings-daemon, and libgnome-desktop)," said Clasen. "Another cluster of bugs has to do with applications that have some sort of 'presentation mode,'" that requires a dual monitor setup.

"In terms of their severity," Clasen added, "The issues ranged from minor UI annoyances (wrong colors, too large fonts) to feature improvements (make the gthumb slideshow mode display on a connected projector) to serious bugs (rotating a second monitor renders the screen at an offset."

In comparison, bugs addressed by One Hundred Paper Cuts tend to be of low or medium severity and more diverse. Many, though, center on Nautilus and the names given to menu items, and the composition and behavior of the path bar and toolbar. Others, no doubt inspired by Canonical's efforts to improve notifications, center on system messages, some asking for messages that explain more clearly, and others for the removal or editing of unclear statements.

Usability testing at last

At this point, any evaluation of Fit and Finish or One Hundred Paper cuts must be tentative. Both are scheduled to be evaluated after the development versions they are focused upon are officially released. However, one early problem that has already emerged is that the upstream project — GNOME — does not give the small bugs the same priority that Ubuntu and Fedora are assigning them. "Our goal is to get them fixed on a weekly basis," Siegel says, "But an upstream bug will just sit there. They don't have the same sense of urgency. That's been just a little bit frustrating, because priorities are different."

Still, while the work flow in the two projects may be refined by internal review, talking to the organizers of these small bug projects, you get the impression that neither effort is going away. As Siegel described the situation, such projects represent "the biggest bang for your development buck." In other words, they have quick results, and are obvious to the average user. Another advantage of these projects is that they encourage user participation in the development cycle. As a result of One Hundred Paper Cuts, "A lot of people are submitting their first time bugs," Siegel observes — an advantage that, after his experience with GNOME Do, he describes as "definitely calculated."

In much the same way, Clasen observed that "test days also serve as 'meet our user' days. While the audience is far from unbiased — most participants are certainly tech-savvy fedora-devel-list readers — having written use cases helps a lot when trying to shed the developer perspective." An important aspect of this exchange is that it encourages developers to look beyond their own sets of packages, as happened when X Window developer Adam Jackson fixed and enhanced GNOME's handling of monitors.

In the short term, too, such projects have the advantage of encouraging more people to try development releases — a goal that many projects often find elusive. Referring to Fedora's development release, Clasen says, "If Rawhide is more stable, more people will use it, broadening our tester base. And if we don't have to fight Rawhide breakage, we have more time to devote to the user experience issues identified by our test days."

Usability testing has always been difficult in free software, partly because people rarely meet face to face and partly because it requires a trained perspective to do well. But with projects like Fit and Finish and One Hundred Paper Cuts, free software may have just discovered its own method for approaching usability issues and for giving people a chance to learn by doing. In the end, this encouragement of usability testing might be as significant a result of the small bug meme as the improvements it brings to the desktop.

Comments (30 posted)

Community Leadership Summit 2009

More than 200 people gathered at San Jose's McEnery Convention Center on July 18 and 19 for the inaugural Community Leadership Summit (CLS). The event was primarily an unconference, with both days' programs assembled on the fly by the attendees themselves. The majority of those attendees identified themselves as participants in free software or open source communities, but a significant minority came from other realms — closed source companies interested in dealing with their customer communities, web services, and online communities unaffiliated with software altogether. Regardless of the field, of course, many of the issues are the same — from dealing with the tensions within online communities, to grappling with the technical challenges of communication overload, to avoiding burnout.

The CLS was organized in large part by Canonical's Jono Bacon, who explained in the Saturday morning plenary session that he wanted to have an event that dealt with community management issues above the "product" level — thus enabling participation by people from all Linux distributions, desktop environments, languages, and user or development groups. The unconference format facilitated that "open to all" principle; only the opening and closing slots of each day were pre-planned, participants filled out the rest of the schedule by announcing sessions that they themselves were interested in facilitating, then selecting an open time slot and meeting room on the large program board in the hallway. The sessions were round-table discussions, not presentations. In several instances, the person who proposed the session announced at the outset that he or she had no answers and was primarily interested in hearing the thoughts of the others in the room.

Common questions

Over the course of the two days, some recurring themes emerged in multiple sessions led by different facilitators. The mechanics of community management was one such subject; several sessions dealt explicitly with how leaders interacted with their communities: the tools of communication, tools for tracking issues, conflicts, and participation, and the metrics used to measure community health, growth, and participation. From the experiences of the attendees, it is clear that there are no clear-cut solutions in this area. Even in software itself, most community managers are re-using tools created for other purposes, such as bug tracking software and customer relationship management (CRM) systems, with mixed results.

The role of women in online communities was also central to several sessions, including the different communication styles exhibited by men and women and how to adapt to both in the community, and how to respond to conflict and gender bias (both explicit and perceived). The open source community has been addressing the inclusion of women more and more frequently in recent years, both in attracting more participants and in adjusting the male-dominated "engineer" culture that many see as a barrier to entry in open source. The discussion is ongoing, naturally, but one key benefit to addressing the topic at the CLS was the opportunity to learn from the experiences of other online communities, including those that are not majority-male.

Finally, several sessions dealt with the legal issues facing online communities, focusing on open source communities in particular. Specifics included trademark and branding issues, budgeting, fundraising and volunteer compensation, and the details of nonprofit tax exempt foundations. As widespread as open source software is, many groups — particularly smaller ones — still face the same issues and raise the same questions.

The reinvention problem

Danese Cooper of the Open Source Initiative (OSI) led an interesting discussion on the tricky task of reinventing an existing community. She drew from the OSI's recent experiences as it tries to redefine itself and its associated community, but pointed to several other examples as well, including when a community grows up naturally around a vendor product and then must change when the product changes hands or is itself redefined, as has happened with Java. In some cases, redefining the community is not a choice — such as the change that naturally occurs when a product changes or the redefinition following the forking of a large project — but it can also be a conscious decision taken in order to avert obsolescence or burnout.

OSI has learned some valuable lessons through its early attempts and false starts, Cooper said, including the fact that it is not enough to simply gather interested stakeholders together and expect a community to coalesce by virtue of shared values and interests. OSI's attempt to bootstrap a community by "starting with the membership" failed, she said, because of infighting and arguments. The organization has had more success growing a community naturally by starting actual projects, then allowing interested participants to join in the effort voluntarily.

Participants in the discussion related experiences with redefining communities like Java, LiveJournal, and the Open Web Foundation (OWF). The OSI's success with projects as a driving force was an indicator that projects are the "currency" of the OSI community, according to the discussion, but that different groups might find a different currency to be the solution to their own problem. Most agreed that the underlying problem was not one of attracting people, but of redefining the vision for the community that would attract the right people. Especially for communities that involve both companies and outside volunteers, agreeing upon that vision can be a difficult process.

That point led into a discussion on the issue of forking a community when participants disagree over vision or core values. Although, on the surface, forking an existing community sounded like a negative, the group decided that there were sufficiently many examples of positive community forks to conclude that it is sometimes a very good idea. One example was the Ubuntu Linux distribution. Debian is (and always has been) committed to building a completely free Linux distribution. Among others, Mark Shuttleworth felt that the Debian Project was limiting itself (taking too long between releases, etc.), but rather than attempting to change the way the project operated, he created Ubuntu as a derivative with different goals and a different message. The result has been a success for both distributions, whereas attempting to force the Debian Project to change would likely have ended in failure.

Open source communities in the developing world

Perhaps the most challenging session was Nnenna Nwakanma's look at developing open source in the developing world. Nwakanma is a member of the Free Software and Open Source Foundation for Africa (FOSSFA), which promotes open source software across the African continent. Both Nwakanma and Bruno Souza from Brazil spoke about the obstacles facing open source communities in developing countries and, in particular, the ways in which the traditional approaches that have proven successful in North America and Europe fail under the radically different circumstances found in other countries.

Some of the differences are well-known, such as the fact that governments are by far the largest IT spenders in developing countries. Other differences took attendees by surprise, such as the challenges in developing sustainable open source communities. Many of the strategies common in the West do not work as well — or at all — in Africa, Nwakanma said. For example, most open source projects and communities use the Internet as their default (if not their only) means of communicating, organizing, and working. In contrast, the vast majority of the people in Africa do not have Internet access at all, much less in the evening at home, so local user groups that meet in person are the premiere way of spreading and educating about open source. African open source advocates also have significant trouble keeping developers active after they leave college, since so many people have difficultly simply finding paying jobs. FOSSFA has additional difficulties that result from targeting the entire continent of Africa; its initiatives must be equally accessible in all 53 African countries, or else risk fracturing the community along regional lines.

Despite the challenges, Nwakanma did have ideas for building the open source community in Africa, including targeting school age children with open source education and clubs, observing that proprietary software vendors relentlessly pursue lucrative government contracts, but are never interested in investing in school-age children as potential developers. The other attendees shared their ideas as well, such as linking local open source groups in Africa with established Linux user groups (LUGs) in the West in a "sister city"-style program, and publicizing opportunities for visitors to Africa to help local groups by volunteering to speak about open source.

More community leadership

At the closing session on Sunday, the vast majority of attendees said that CLS was valuable and were enthusiastic to see it return next year. Opinion was split about the time and place; this year's event was the weekend immediately preceding the massive O'Reilly Open Source Convention (OSCON), which was a plus for those already planning to attend OSCON but a minus for those who found OSCON too expensive. Bacon promised that CLS would reappear in some form next year, to further the discussions that so many found useful. Managing online communities is a topic that is growing in importance, which should guarantee the continued success of the event. But it is also critically important for the open source movement as a whole, because it depends on healthy and vibrant communities for its survival.

A Belorussian translation has been provided by Patricia Clausnitzer

Comments (4 posted)