Fighting small bugs

Paper cuts, points of pain, obstacles and annoyances — whatever description you prefer, in the last six weeks, small bugs have started receiving closer attention from developers of the free software desktop. In Ubuntu, they are the focus of One Hundred Paper Cuts, and in Fedora of Fit and Finish, but in both cases, the goal is the same: to significantly improve the user experience by tackling bugs that can be quickly corrected. As an important side effect, these efforts are also allowing free software developers to approach usability in new ways.

Concentrated efforts at bug-squashing have a long history in free software development, so the idea of focusing on small bugs probably has multiple origins. However, one origin is the 0.8 release of GNOME Do in January 2009. The release fixed 111 bugs — over three times the number fixed in the 0.8.1 release — and received enthusiastic feedback from users.

When GNOME Do leader David Siegel joined the Design and User Experience team at Canonical a few weeks after the 0.8 Do release, he took the small bug meme with him. The Design and User Experience team is the group within Canonical whose task is to realize Mark Shuttleworth's challenge of improving the Ubuntu desktop, and Siegel soon saw similarities to Do: "We started to notice all these small things that were in each release that were never getting fixed", Siegel said. "And I said to Ivanka Majic, who's the leader of the Design team, 'We need a way to red flag these things that are obviously wrong, and make sure they are fixed before they go out.'" Within a couple of months, Siegel found himself leading One Hundred Paper Cuts.

According to Red Hat employee Matthias Clasen, a similar situation exists in Fedora. "There is a challenge in working on the Fedora desktop between feeling squeezed to finish cool new features in time for the next release, and fighting to get Rawhide [Fedora's development branch] into a somewhat working state, with too little time to devote to fit and finish" — that is, to polishing and removing rough edges. Like Siegel, Clasen now finds himself at the head of an effort to provide that missing attention to detail.

Whether Fedora and Ubuntu influenced each other in these developments is uncertain. However, in both cases, the advantage of focusing on small bugs is obvious. As Siegel explained, "They're low-hanging fruit. They allow us to quickly, inexpensively improve the user experience. We don't have to create new interfaces; we just have to fix these tiny, trivial bugs. It's just a small component, but it's something that can have an immediate impact."

Same problem, different approaches

Despite the similarity of the problems, Ubuntu and Fedora have organized their solutions in somewhat different ways. At Canonical, the One Hundred Paper Cuts project set a goal of addressing one hundred bugs during the development cycle of the upcoming "Karmic Koala" release, which is scheduled for October 2009. To narrow the focus enough to make it manageable, the project decreed that the bugs would center around what Siegel calls "the space between applications" or features such as the panel and the Nautilus file browser. Users were invited to report a paper cut bug via Ubuntu's Launchpad, and the initial one hundred paper cuts were chosen from the several thousand that were submitted.

For this first effort, the project chose (despite its name) to divide its efforts into ten rounds of eleven bugs each, ten in each round concerning Ubuntu's main GNOME desktop, and one concerning Kubuntu, the Ubuntu KDE variation. The project also defined what it would cover with extreme clarity. According to Siegel, a paper cut is a bug that users would encounter with a default installation from a Live CD. Although it would not actually prevent users from completing everyday tasks, it might cause momentary annoyance or distraction.

Theoretically, a paper cut must be correctable by one developer in one day. But, in practice, Siegel said, "We set a sort of gray area. For example, some of the paper cuts we've confirmed and want to fix for Karmic really take weeks to fix, but the work is going to take place anyway, and the paper cut is just a little bit of extra work beyond that."

By contrast, Fedora's Fit and Finish project chose a less formal approach. As Clasen explained, Fedora's Quality Assurance team was already in the habit of holding test days, in which participants prepared by downloading designated software and discussing what they found on IRC. Fit and Finish simply decided to hold its own test days during the development of Fedora 12, based on input submitted to the project's Bugzilla or by email. For each test day, instructions for participation are listed on a separate page, and both bugs and "things that work right" are summarized on the page after the discussion.

One difference from Ubuntu is that Fedora chose "to focus on user tasks, as opposed to individual features," according to Clasen. Nor did Fit and Finish limit the number of bugs to be covered in a single test day or to be fixed. However Clasen did add, "I realize that numbers — like the points awarded by bugzilla.gnome.org — can be a strong motivation, so we may revisit this at some point."

Early results

To date, Fit and Finish has held just one test day on the subject of display configuration, although pages for the topics batteries and suspend and peripherals are already posted. Clasen noted that this first day did not have "an overwhelming participation." He added, though, that the relatively low turnout was probably due to the fact that many desktop developers were at the Grand Canaria Desktop Summit that was in progress on the same day. A week later,the bugs arising from the testing day have been filed and assigned to a tester, but none have been closed.

In comparison, One Hundred Paper Cuts has had two rounds so far, and the bugs to tackle in each future round are already posted. Of the ten bugs in the first round, seven are now listed as fixed, one as having had a fix committed, and another two as "in progress," with only one listed as incomplete. In the second round, which finished on July 11, two are marked as fixed, another three have a fix committed but not yet applied, four are in progress, and two are listed as confirmed. The preliminary appearance is that One Hundred Paper Cuts is producing quicker results, possibly because its goals are better defined.

However, a disadvantage of the One Hundred Paper Cuts approach is the appearance it creates that some bugs are being given special treatment. For this reason, Siegel felt compelled to emphasize that "Paper cuts are just an additional level of importance being attached to bugs. A lot of people whose bugs get rejected for paper cut status will get angry and frustrated and say, 'Does this mean that the bug's not going to get fixed?' But it just means it's not going to be the focus. Many, many bugs will be fixed for Karmic; this set of one hundred is just getting a little extra push and a little extra attention."

As might be expected given Fit and Finish's more narrow topic, the bugs it generated fall into recognizable categories. "Naturally, a lot of the bugs that we have collected on that day are X driver bugs, bugs in the GNOME modules that play a role in display configuration (the display capplet, gnome-settings-daemon, and libgnome-desktop)," said Clasen. "Another cluster of bugs has to do with applications that have some sort of 'presentation mode,'" that requires a dual monitor setup.

"In terms of their severity," Clasen added, "The issues ranged from minor UI annoyances (wrong colors, too large fonts) to feature improvements (make the gthumb slideshow mode display on a connected projector) to serious bugs (rotating a second monitor renders the screen at an offset."

In comparison, bugs addressed by One Hundred Paper Cuts tend to be of low or medium severity and more diverse. Many, though, center on Nautilus and the names given to menu items, and the composition and behavior of the path bar and toolbar. Others, no doubt inspired by Canonical's efforts to improve notifications, center on system messages, some asking for messages that explain more clearly, and others for the removal or editing of unclear statements.

Usability testing at last

At this point, any evaluation of Fit and Finish or One Hundred Paper cuts must be tentative. Both are scheduled to be evaluated after the development versions they are focused upon are officially released. However, one early problem that has already emerged is that the upstream project — GNOME — does not give the small bugs the same priority that Ubuntu and Fedora are assigning them. "Our goal is to get them fixed on a weekly basis," Siegel says, "But an upstream bug will just sit there. They don't have the same sense of urgency. That's been just a little bit frustrating, because priorities are different."

Still, while the work flow in the two projects may be refined by internal review, talking to the organizers of these small bug projects, you get the impression that neither effort is going away. As Siegel described the situation, such projects represent "the biggest bang for your development buck." In other words, they have quick results, and are obvious to the average user. Another advantage of these projects is that they encourage user participation in the development cycle. As a result of One Hundred Paper Cuts, "A lot of people are submitting their first time bugs," Siegel observes — an advantage that, after his experience with GNOME Do, he describes as "definitely calculated."

In much the same way, Clasen observed that "test days also serve as 'meet our user' days. While the audience is far from unbiased — most participants are certainly tech-savvy fedora-devel-list readers — having written use cases helps a lot when trying to shed the developer perspective." An important aspect of this exchange is that it encourages developers to look beyond their own sets of packages, as happened when X Window developer Adam Jackson fixed and enhanced GNOME's handling of monitors.

In the short term, too, such projects have the advantage of encouraging more people to try development releases — a goal that many projects often find elusive. Referring to Fedora's development release, Clasen says, "If Rawhide is more stable, more people will use it, broadening our tester base. And if we don't have to fight Rawhide breakage, we have more time to devote to the user experience issues identified by our test days."

Usability testing has always been difficult in free software, partly because people rarely meet face to face and partly because it requires a trained perspective to do well. But with projects like Fit and Finish and One Hundred Paper Cuts, free software may have just discovered its own method for approaching usability issues and for giving people a chance to learn by doing. In the end, this encouragement of usability testing might be as significant a result of the small bug meme as the improvements it brings to the desktop.

Note: you can avoid this step in the future by logging into your LWN account.