This year's openSUSE
conference had some interesting sessions about testing topics. One of
those described a framework to automate testing of the distribution's
installation. That way testers don't have to do the repetitive installation
steps themselves. Another session described Testopia, which is a test case
management extension for Bugzilla. OpenSUSE is using Testopia to guide
users that want to help testing the distribution. And last but not least, a
speaker from Mozilla QA talked about how to attract new testers. The common thread in all these sessions is that testing should be made as easy as possible, to attract new testers and keep the current testers motivated.
Testing is an important task for distributions, because a Linux
distribution is a very complex amalgam of various interacting components,
but it would be pretty tiresome and boring for testers to test the openSUSE Factory snapshots
daily. Bernhard Wiedemann, a member of the openSUSE
Testing Core Team, presented the logical solution to this problem:
automate as much as possible. Computers don't get tired and they don't stop
testing out of boredom, even with dozens of identical tests.
But why is automation so important for testing? To answer this question,
Bernhard emphasized that the three chief virtues of a programmer according
to Larry Wall (laziness, impatience, and hubris) also hold for
testers. What we don't want is poor testing, which leads to poor quality of
the distribution, which leads to frustrated testers, which leads to even
poorer testing. This is a vicious circle. What we want instead is good
testing and good processes, which leads to high quality for the
distribution and to happy testers who make the testing and hence the
distribution even better. Testers, as much as programmers, want to automate
things because they want to reduce their overall efforts.
So what are possible targets for automated testing? You could consider
automating the testing of a distribution's installation, testing distribution upgrades, application testing, regression testing, localization testing, benchmarking, and so on. But whatever you test, there will always be some limitations. As the Dutch computer scientist and Turing Award winner Edsger W. Dijkstra once famously said: "Testing can only prove the presence of bugs, not their absence."
Bernhard came up with a way to automate distribution installation
testing using KVM. He now has a cron job that downloads a new ISO for
openSUSE Factory daily and runs his Perl script autoinst for the
test. This script starts openSUSE from the ISO file in a virtual machine
with a monitor interface that accepts commands like sendkey
ctrl-alt-delete to send a key to the machine or screendump
foobar.ppm to create a screenshot. The script compares the screenshots
to known images, which is done by computing MD5 hashes of the pixel
When the screen shot of a specific step of the running installer matches
the known screen shot of the same step in a working installer, the script
marks the test of this step as passed. If they don't match (e.g. because of
an error message), the test is marked as failed. The keys that the script
sends to the virtual machine can also depend on what is shown on the
screen: the script then compares the screen shot to various possible screen
shots of the working installer, each them representing a possible execution
By using the screen shots, a script can
test whether an installation of an openSUSE snapshot worked correctly and
whether Firefox or OpenOffice.org can be started on the freshly installed
operating system without segfaulting. At the end of the test, all images are encoded into a video, which can be consulted by a human tester in circumstances where a task couldn't be marked automatically as passed or failed. Some examples of installation videos can be found on Bernhard's blog.
It's also nice to see that Bernhard is following the motto of this year's openSUSE conference, "collaboration across borders": while parts of his testing framework are openSUSE-specific, it is written in a modular way and can be used to test any operating system that runs on Qemu and KVM. More information can be found on the OS-autoinst web site.
Test plans with Testopia
Holger Sickenberg, the QA Engineering Manager in charge of openSUSE
testing, talked about another way to improve openSUSE's reliability: make
test plans available to testers and users with Testopia, a test case
management extension for Bugzilla. In the past, openSUSE's Bugzilla bug
tracking system only made Testopia available to the openSUSE Testing Core
Team, but since last summer it is open to all contributors. Testopia is
available on Novell's Bugzilla,
where logged-in users can click on "Product Dashboard" and choose a product to see the available test plans, test cases, and test runs. In his talk, Holger gave an overview about how to create your own test plan and how to file a bug report containing all information from a failed test plan.
A test plan is a simple description for the Testopia system and is
actually just a container for test cases. Each test plan targets a
combination of a specific product and version, a specific component, and a specific type of activity. For example, there is a test plan for installing openSUSE 11.3. A test plan can also have more information attached, e.g. a test document.
A test case, then, is a detailed description of what should be done by the tester. It lists the preparation that is needed before executing the test, a step-by-step list of what should be done, a description of the expected result, and information about how to get the system back into a clean state. Other information can also be attached, such as a configuration file or a test script for an automated test system. Holger emphasized that the description of the expected result is really important: "If you don't mention the expected result exactly, your test case can go wrong because the tester erroneously thinks his result is correct."
And then there's a test run, which is a container for test cases for a specific product version and consists of one or more test plans. It also contains test results and a test history. At the end of executing a test run, the user can easily create a bug report if a test case fails by switching to the Bugs tab. The information from the test case is automatically put into the description and summary of the bug report, and when the report is submitted it also appears in the web page of the test run, including its status (e.g. fixed or not).
The benefits of test plans are obvious: users that want to help a
project by testing have a detailed description of what and how to test, and
the integration with Bugzilla makes reporting bugs as easy as possible. It
also lets developers easily see what has been tested and get the results of the tests. These results can also be tracked during the development cycle or compared between different releases. Holger invited everyone with a project in openSUSE to get in touch with the openSUSE Testing Core Team to get a test plan created. The team can be found on the opensuse-testing mailing list and on the #opensuse-testing IRC channel on Freenode.
Carsten Book, QA Investigations Engineer at the Mozilla Corporation,
gave a talk about how to get involved in the Mozilla Project and he focused
on Mozilla QA, which has its home on the QMO web site. This QA portal has a
lot of documentation,
e.g. for getting started
with QA. And there are links to various Mozilla QA tools such as Bugzilla, Crash Reporter,
the Litmus system that has test
cases written by Mozilla QA for manual software testing, and some tools to
testing. For example, Mozilla's test system automatically checks
whether performance has degraded after every check-in of a new feature, to
try to ensure that Firefox won't get any slower.
People who want to help test can of course run a nightly build and file bug reports. There are also Mozilla test days that teach how to get development builds, how to file bugs, and how to work with developers on producing a fix. Contributors with some technical expertise can join one of the quality teams, each focusing on a specific area: Automation, Desktop Firefox, Browser Technologies, WebQA, and Services. Each of the teams has a short but instructive web page with information about what they do and how you can contact them.
An important point that Carsten made was that it should also be easy for
interested people to immediately get an overview of different areas where
they can contribute without having to read dozens of wiki pages. Mozilla
even has a special Get
involved page where you just enter your email address and an area of
interest, with an optional message. After submitting the form, you will get an email to put you in touch with the right person.
Low entry barrier
These three projects are all about lowering the barriers for new testers
— to be able to attract as many testers as possible and to make the
life of existing testers easier — by automating boring and repetitive
tasks. In this way you can keep testers motivated. Wiedemann's autoinst
project seems especially interesting: at the moment it has just the basic features, but it has a lot of potential, e.g. if the feature for comparing screen shots is refined. From a technical point of view, this is an exciting testing project that hopefully finds its way into other distributions.
to post comments)