GNOME gains a usability specialist

GNOME's last formal usability testing was conducted by Sun Microsystems in 2001—before the release of GNOME 2. Since then, any usability testing that has occurred has been informal, usually carried out by individual developers with a variety of methods. However, the situation shows signs of improvement for GNOME 3, thanks to the work of Aakanksha Gaur, a graduate student currently completing her thesis at the National Institute of Design in Bangalore, India.

Gaur's interest in testing GNOME began when her computer crashed, and she replaced her proprietary operating system with Ubuntu and GNOME. Looking around for help, she realized that she was one of the few students at the Institute who was using free software, and that GNOME could be the focus she needed for her thesis on usability. Later, when she successfully applied for a mentorship with the Outreach Program for Women (OPW), it also became a source of funding for four months. In turn, Gaur has become one of the OPW's success stories.

In the three months since the conclusion of the mentorship, Gaur has continued to work with her mentor, GNOME design lead Allan Day, filing bugs and suggestions for improvements. At the start of Gaur's research, Day commented in an email that it "will be one of the first opportunities we have had to do an extended research study." GNOME executive director Karen Sandler has helped to arrange token payments for Gaur's usability testers — a standard practice in academic research involving test subjects.

Designing the tests

With this support, Gaur has focused her thesis on usability testing of GNOME — mainly, utilities and configuration tools — and on users' perceptions of GNOME. Unfortunately the blog notes on her early work became unavailable when her provider closed down, and Gaur has yet to repost them. For now, the clearest record of them is from an article I did when she was beginning her work. These blog notes included Gaur's early research into the desktop metaphor and her earliest informal testing, together with some rough suggestions for redesign of the To Do and Character Map utilities.

This early work also shows Gaur learning how to conduct her research. After trying to study usability by constantly asking questions as people worked, she wrote, "I was under the confident assumption that I shall take long interviews of users and magically they will reveal the design mistakes which we shall fix and hence, rule the world." In practice, though, she immediately found the technique lacking. The feedback was "very vague and very unfocused" and she realized that, "I ended up putting words in the mouth of the interviewee." The end result was a complete lack of "data that challenged my existing beliefs about the system in any way," and was therefore of minimal use.

Gaur's work beyond this point is documented on her current blog. Instead of micro-managing interviewees' experience, she opted for a test script in which interviewees are given a dozen basic tasks, such as changing the desktop wallpaper, searching for documents, and managing virtual workspaces. Meanwhile, Gaur observed how efficiently interviewees did each task, what mistakes they made, whether they could recall tasks later, and their emotional states after finishing a task.

The blog includes transcripts of pilot test sessions, as well as recordings of the sessions. Like her earlier blog entries, the available ones show Gaur making mistakes and improving her methodology, a degree of transparency that she suggests is appropriate for a free software project. After the pilot sessions, Gaur went on to interview eight testers with her revised methodology.

The first observations

Gaur is still finalizing her results. However, she does have a few general observations about both her methods and the state of GNOME usability. First, based on her research, Gaur concludes that "GNOME suffers from the issue of discoverability." That is, the tools users want are available, but may not be easy to discover. "The problem to crack is how to make them visible," she said, adding that "utilities like the Tweak tool and the rich set of extensions" might be the most immediate way to deliver improvements.

Second, people's expectations of GNOME are based heavily on the operating systems with which they are familiar, and the web applications that they use. While she has not finished assembling her research, she suspects that web and mobile applications have become more important than the operating system for people who spend more time on the web.

Never forgetting the self-criticism, Gaur also observed that her work would be improved if she made greater efforts towards "making a user comfortable in the first few seconds of interaction."

Her preliminary conclusion? "GNOME is doing the best it can," especially since free software development has traditionally been driven by developers rather than users.

Integrating Usability into GNOME Practice

These tentative conclusions are hardly startling. However, the point is that they have not been systematically recorded for GNOME 3. Instead, like all free software projects, GNOME has relied on bug reports and personal impressions, both of which are considerably better than nothing, but do not necessarily provide accurate views of the average user's experience. Bug reports, for example, are likely to describe the experience of those with enough knowledge to know how to file them, and not newcomers. Similarly, for all the controversy over GNOME 3, all available records indicate that the designers believed that they were providing simple and practical solutions to major problems. By putting matters on a more impartial basis, usability testing like Gaur's may act as a reality check to design proposals.

Certainly, GNOME is taking Gaur's work, as preliminary as it is, seriously. "I'm hopeful that this work will serve as a template for user testing exercises in the future," Day said. "There are certainly challenges involved in doing usability testing without a dedicated lab and equipment, so having a publicly accessible account of a successful open research exercise will be valuable." Sandler agreed, adding, "The GNOME Foundation whole-heartedly supports this work. GNOME 3.8 has had a really good response, but employing systematic tests will help us improve further."

For now, Gaur is focusing on completing her thesis. Once it is accepted, her first concern will be to make her work as widely available as possible, especially outside of GNOME. Then, she plans to go into more detail:

Gaur's work is just beginning. Yet the degree to which it has been accepted in eight months speaks highly of its quality and transparency, to say nothing of its originality. Perhaps in another twelve years, usability testing won't have to be re-introduced, but will have long ago become a routine concern for GNOME and other free software projects.