Distributions

openSUSE Conference 2010: Making testing easier

This year's openSUSE conference had some interesting sessions about testing topics. One of those described a framework to automate testing of the distribution's installation. That way testers don't have to do the repetitive installation steps themselves. Another session described Testopia, which is a test case management extension for Bugzilla. OpenSUSE is using Testopia to guide users that want to help testing the distribution. And last but not least, a speaker from Mozilla QA talked about how to attract new testers. The common thread in all these sessions is that testing should be made as easy as possible, to attract new testers and keep the current testers motivated.

Automated testing

Testing is an important task for distributions, because a Linux distribution is a very complex amalgam of various interacting components, but it would be pretty tiresome and boring for testers to test the openSUSE Factory snapshots daily. Bernhard Wiedemann, a member of the openSUSE Testing Core Team, presented the logical solution to this problem: automate as much as possible. Computers don't get tired and they don't stop testing out of boredom, even with dozens of identical tests.

But why is automation so important for testing? To answer this question, Bernhard emphasized that the three chief virtues of a programmer according to Larry Wall (laziness, impatience, and hubris) also hold for testers. What we don't want is poor testing, which leads to poor quality of the distribution, which leads to frustrated testers, which leads to even poorer testing. This is a vicious circle. What we want instead is good testing and good processes, which leads to high quality for the distribution and to happy testers who make the testing and hence the distribution even better. Testers, as much as programmers, want to automate things because they want to reduce their overall efforts.

So what are possible targets for automated testing? You could consider automating the testing of a distribution's installation, testing distribution upgrades, application testing, regression testing, localization testing, benchmarking, and so on. But whatever you test, there will always be some limitations. As the Dutch computer scientist and Turing Award winner Edsger W. Dijkstra once famously said: "Testing can only prove the presence of bugs, not their absence."

Bernhard came up with a way to automate distribution installation testing using KVM. He now has a cron job that downloads a new ISO for openSUSE Factory daily and runs his Perl script autoinst for the test. This script starts openSUSE from the ISO file in a virtual machine with a monitor interface that accepts commands like sendkey ctrl-alt-delete to send a key to the machine or screendump foobar.ppm to create a screenshot. The script compares the screenshots to known images, which is done by computing MD5 hashes of the pixel data.

When the screen shot of a specific step of the running installer matches the known screen shot of the same step in a working installer, the script marks the test of this step as passed. If they don't match (e.g. because of an error message), the test is marked as failed. The keys that the script sends to the virtual machine can also depend on what is shown on the screen: the script then compares the screen shot to various possible screen shots of the working installer, each them representing a possible execution path.

By using the screen shots, a script can test whether an installation of an openSUSE snapshot worked correctly and whether Firefox or OpenOffice.org can be started on the freshly installed operating system without segfaulting. At the end of the test, all images are encoded into a video, which can be consulted by a human tester in circumstances where a task couldn't be marked automatically as passed or failed. Some examples of installation videos can be found on Bernhard's blog.

It's also nice to see that Bernhard is following the motto of this year's openSUSE conference, "collaboration across borders": while parts of his testing framework are openSUSE-specific, it is written in a modular way and can be used to test any operating system that runs on Qemu and KVM. More information can be found on the OS-autoinst web site.

Test plans with Testopia

Holger Sickenberg, the QA Engineering Manager in charge of openSUSE testing, talked about another way to improve openSUSE's reliability: make test plans available to testers and users with Testopia, a test case management extension for Bugzilla. In the past, openSUSE's Bugzilla bug tracking system only made Testopia available to the openSUSE Testing Core Team, but since last summer it is open to all contributors. Testopia is available on Novell's Bugzilla, where logged-in users can click on "Product Dashboard" and choose a product to see the available test plans, test cases, and test runs. In his talk, Holger gave an overview about how to create your own test plan and how to file a bug report containing all information from a failed test plan.

A test plan is a simple description for the Testopia system and is actually just a container for test cases. Each test plan targets a combination of a specific product and version, a specific component, and a specific type of activity. For example, there is a test plan for installing openSUSE 11.3. A test plan can also have more information attached, e.g. a test document.

A test case, then, is a detailed description of what should be done by the tester. It lists the preparation that is needed before executing the test, a step-by-step list of what should be done, a description of the expected result, and information about how to get the system back into a clean state. Other information can also be attached, such as a configuration file or a test script for an automated test system. Holger emphasized that the description of the expected result is really important: "If you don't mention the expected result exactly, your test case can go wrong because the tester erroneously thinks his result is correct."

And then there's a test run, which is a container for test cases for a specific product version and consists of one or more test plans. It also contains test results and a test history. At the end of executing a test run, the user can easily create a bug report if a test case fails by switching to the Bugs tab. The information from the test case is automatically put into the description and summary of the bug report, and when the report is submitted it also appears in the web page of the test run, including its status (e.g. fixed or not).

The benefits of test plans are obvious: users that want to help a project by testing have a detailed description of what and how to test, and the integration with Bugzilla makes reporting bugs as easy as possible. It also lets developers easily see what has been tested and get the results of the tests. These results can also be tracked during the development cycle or compared between different releases. Holger invited everyone with a project in openSUSE to get in touch with the openSUSE Testing Core Team to get a test plan created. The team can be found on the opensuse-testing mailing list and on the #opensuse-testing IRC channel on Freenode.

Mozilla QA

Carsten Book, QA Investigations Engineer at the Mozilla Corporation, gave a talk about how to get involved in the Mozilla Project and he focused on Mozilla QA, which has its home on the QMO web site. This QA portal has a lot of documentation, e.g. for getting started with QA. And there are links to various Mozilla QA tools such as Bugzilla, Crash Reporter, the Litmus system that has test cases written by Mozilla QA for manual software testing, and some tools to automate software testing. For example, Mozilla's test system automatically checks whether performance has degraded after every check-in of a new feature, to try to ensure that Firefox won't get any slower.

People who want to help test can of course run a nightly build and file bug reports. There are also Mozilla test days that teach how to get development builds, how to file bugs, and how to work with developers on producing a fix. Contributors with some technical expertise can join one of the quality teams, each focusing on a specific area: Automation, Desktop Firefox, Browser Technologies, WebQA, and Services. Each of the teams has a short but instructive web page with information about what they do and how you can contact them.

An important point that Carsten made was that it should also be easy for interested people to immediately get an overview of different areas where they can contribute without having to read dozens of wiki pages. Mozilla even has a special Get involved page where you just enter your email address and an area of interest, with an optional message. After submitting the form, you will get an email to put you in touch with the right person.

Low entry barrier

These three projects are all about lowering the barriers for new testers — to be able to attract as many testers as possible and to make the life of existing testers easier — by automating boring and repetitive tasks. In this way you can keep testers motivated. Wiedemann's autoinst project seems especially interesting: at the moment it has just the basic features, but it has a lot of potential, e.g. if the feature for comparing screen shots is refined. From a technical point of view, this is an exciting testing project that hopefully finds its way into other distributions.

Comments (5 posted)

Brief items

Distribution quotes of the week

-- Adam Jackson

-- Gregory Maxwell

Comments (17 posted)

Red Hat Enterprise Linux 6 available

It's official: RHEL6 is available. "Enhancements range from kernel improvements for resource management, RAS, performance, scalability, virtualization and power saving, through a greatly extended and standards-compliant development environment, to a comprehensive range of updated server and desktop applications. It is designed to improve agility, lower costs and reduce IT complexity for customers."

Comments (7 posted)

Distribution News

Fedora

Fedora rejects SQLninja

The minutes from the November 8 Fedora Board meeting include a discussion of whether SQLninja - an SQL injection testing tool - should be included in the distribution. The answer was "no," and a new guideline was added to the rules: "Where, objectively speaking, the package has essentially no useful foreseeable purposes other than those that are highly likely to be illegal or unlawful in one or more major jurisdictions in which Fedora is distributed or used, such that distributors of Fedora will face heightened legal risk if Fedora were to include the package, then the Fedora Project Board has discretion to deny inclusion of the package for that reason alone."

Full Story (comments: 47)

Smith: Changing of the seasons

Jared Smith welcomes the Fedora Project's new release engineering team lead, Dennis Gilmore, and the new Fedora Program Manager, Robyn Bergeron. "Just as with nature, we have cyclical changes within the Fedora Project as well. I think it's both useful and healthy to point out a few of those changes, for a couple of reasons. First of all, I want to point out that every person in the Fedora community is a potential leader. Our policies of rotating leadership help ensure that everyone who is so inclined has a chance to lead and serve. Second, I'd like to personally thank those people who have diligently served the Fedora community, and wish them success as they move on to other endeavors."

Comments (none posted)

Mandriva Linux

New Mandriva's cooker manager

Mandriva has a new manager for "cooker", the development branch. "Eugeni Dodonov. Eugeni is well known in the community, a very active Mandriva's contributor, an activist of free software in Brasil, and also a doctor in computer science."

Comments (none posted)

Ubuntu family

Shuttleworth: Unity on Wayland

Mark Shuttleworth has described the next major step for the Unity interface: putting it on the Wayland OpenGL-based display system. "But we don't believe X is setup to deliver the user experience we want, with super-smooth graphics and effects. I understand that it's *possible* to get amazing results with X, but it's extremely hard, and isn't going to get easier. Some of the core goals of X make it harder to achieve these user experiences on X than on native GL, we're choosing to prioritize the quality of experience over those original values, like network transparency."

Comments (173 posted)

Other distributions

Mageia: under construction

The Mageia Blog has a progress report on the new Mandriva offshoot, Mageia. Topics include the buildsystem, blog and website, mirrors, wiki, and a roadmap.

Comments (none posted)

Newsletters and articles of interest

Distribution newsletters

• Debian Misc Developer News (#24) (November 7)
• Debian Project News (November 8)
• DistroWatch Weekly, Issue 379 (November 8)
• Fedora Weekly News Issue 250 (November 3)
• openSUSE Weekly News, Issue 148 (November 6)
• Ubuntu Weekly Newsletter, Issue 217 (November 7)

Comments (none posted)

The MeeGo Progress Report: A+ or D-? (Vision Mobile)

Dave Neary has posted a MeeGo progress report on the Vision Mobile site. "Long-awaited MeeGo compliance specifications have resulted in drawn out and sometimes acrimonious debate. Trademark guidelines have been a sticking point for community ports of the MeeGo netbook UX to Linux when these ports do not include required core components. Related to the technical governance of the project, there is some uncertainty around the release process, and the means and criteria which will be used when considering the inclusion of new components. And there are some signs that the 'all open, all the time' message at the project launch has been tempered by the reality of building a commercial device."

Comments (7 posted)

What's new in Fedora 14 (The H)

The H takes Fedora 14 for a test drive and finds it a little thin on new features. There is, of course, the usual pile of "up-to-the-minute" versions of various packages. "Some Fedora users will also feel the benefits of an update, with Fedora 14 including newer versions of many applications. Version 14 includes hundreds of enhancements and bug fixes not explicitly mentioned above – the new features to be found in OpenOffice 3.3 are just one example of many. Objectively speaking, Laughlin contains significantly fewer enhancements than previous versions. The likely culprit is the end of the development phase of Red Hat Enterprise Linux 6, which has kept many of the Fedora developers employed by Red Hat busy over recent months."

Comments (8 posted)

Pardus 2011 on the way with new goodies (Linux Journal)

Susan Linton takes a look at the upcoming release of Pardus 2011. "Pardus Linux, a popular independent distribution funded and developed by the Scientific & Technological Research Council of Turkey, will be releasing version 2011 in the coming weeks and with it lots of nice updates and improvements."

Comments (none posted)

The Five Best Linux Live CDs (Linux.com)

Joe "Zonker" Brockmeier reveals his favorite live CD distributions. "So how were the distros chosen? You'll notice that none of the major Linux distros (a.k.a. Ubuntu, Debian, Fedora, openSUSE, Slackware, etc.) appear in the list, though most of the picks are derived from one of the major distros. Though Ubuntu, Linux Mint, et al. have perfectly serviceable live CDs or DVDs, they're not really designed for long-term use as a live distro. I'm sure some folks do use them that way, but they're the cream of the crop for installing to a hard drive - not for live media."

Comments (none posted)

Page editor: Rebecca Sobol
Next page: Development>>