This year's openSUSE
conference had some interesting sessions about testing topics. One of
those described a framework to automate testing of the distribution's
installation. That way testers don't have to do the repetitive installation
steps themselves. Another session described Testopia, which is a test case
management extension for Bugzilla. OpenSUSE is using Testopia to guide
users that want to help testing the distribution. And last but not least, a
speaker from Mozilla QA talked about how to attract new testers. The common thread in all these sessions is that testing should be made as easy as possible, to attract new testers and keep the current testers motivated.
Testing is an important task for distributions, because a Linux
distribution is a very complex amalgam of various interacting components,
but it would be pretty tiresome and boring for testers to test the openSUSE Factory snapshots
daily. Bernhard Wiedemann, a member of the openSUSE
Testing Core Team, presented the logical solution to this problem:
automate as much as possible. Computers don't get tired and they don't stop
testing out of boredom, even with dozens of identical tests.
But why is automation so important for testing? To answer this question,
Bernhard emphasized that the three chief virtues of a programmer according
to Larry Wall (laziness, impatience, and hubris) also hold for
testers. What we don't want is poor testing, which leads to poor quality of
the distribution, which leads to frustrated testers, which leads to even
poorer testing. This is a vicious circle. What we want instead is good
testing and good processes, which leads to high quality for the
distribution and to happy testers who make the testing and hence the
distribution even better. Testers, as much as programmers, want to automate
things because they want to reduce their overall efforts.
So what are possible targets for automated testing? You could consider
automating the testing of a distribution's installation, testing distribution upgrades, application testing, regression testing, localization testing, benchmarking, and so on. But whatever you test, there will always be some limitations. As the Dutch computer scientist and Turing Award winner Edsger W. Dijkstra once famously said: "Testing can only prove the presence of bugs, not their absence."
Bernhard came up with a way to automate distribution installation
testing using KVM. He now has a cron job that downloads a new ISO for
openSUSE Factory daily and runs his Perl script autoinst for the
test. This script starts openSUSE from the ISO file in a virtual machine
with a monitor interface that accepts commands like sendkey
ctrl-alt-delete to send a key to the machine or screendump
foobar.ppm to create a screenshot. The script compares the screenshots
to known images, which is done by computing MD5 hashes of the pixel
When the screen shot of a specific step of the running installer matches
the known screen shot of the same step in a working installer, the script
marks the test of this step as passed. If they don't match (e.g. because of
an error message), the test is marked as failed. The keys that the script
sends to the virtual machine can also depend on what is shown on the
screen: the script then compares the screen shot to various possible screen
shots of the working installer, each them representing a possible execution
By using the screen shots, a script can
test whether an installation of an openSUSE snapshot worked correctly and
whether Firefox or OpenOffice.org can be started on the freshly installed
operating system without segfaulting. At the end of the test, all images are encoded into a video, which can be consulted by a human tester in circumstances where a task couldn't be marked automatically as passed or failed. Some examples of installation videos can be found on Bernhard's blog.
It's also nice to see that Bernhard is following the motto of this year's openSUSE conference, "collaboration across borders": while parts of his testing framework are openSUSE-specific, it is written in a modular way and can be used to test any operating system that runs on Qemu and KVM. More information can be found on the OS-autoinst web site.
Test plans with Testopia
Holger Sickenberg, the QA Engineering Manager in charge of openSUSE
testing, talked about another way to improve openSUSE's reliability: make
test plans available to testers and users with Testopia, a test case
management extension for Bugzilla. In the past, openSUSE's Bugzilla bug
tracking system only made Testopia available to the openSUSE Testing Core
Team, but since last summer it is open to all contributors. Testopia is
available on Novell's Bugzilla,
where logged-in users can click on "Product Dashboard" and choose a product to see the available test plans, test cases, and test runs. In his talk, Holger gave an overview about how to create your own test plan and how to file a bug report containing all information from a failed test plan.
A test plan is a simple description for the Testopia system and is
actually just a container for test cases. Each test plan targets a
combination of a specific product and version, a specific component, and a specific type of activity. For example, there is a test plan for installing openSUSE 11.3. A test plan can also have more information attached, e.g. a test document.
A test case, then, is a detailed description of what should be done by the tester. It lists the preparation that is needed before executing the test, a step-by-step list of what should be done, a description of the expected result, and information about how to get the system back into a clean state. Other information can also be attached, such as a configuration file or a test script for an automated test system. Holger emphasized that the description of the expected result is really important: "If you don't mention the expected result exactly, your test case can go wrong because the tester erroneously thinks his result is correct."
And then there's a test run, which is a container for test cases for a specific product version and consists of one or more test plans. It also contains test results and a test history. At the end of executing a test run, the user can easily create a bug report if a test case fails by switching to the Bugs tab. The information from the test case is automatically put into the description and summary of the bug report, and when the report is submitted it also appears in the web page of the test run, including its status (e.g. fixed or not).
The benefits of test plans are obvious: users that want to help a
project by testing have a detailed description of what and how to test, and
the integration with Bugzilla makes reporting bugs as easy as possible. It
also lets developers easily see what has been tested and get the results of the tests. These results can also be tracked during the development cycle or compared between different releases. Holger invited everyone with a project in openSUSE to get in touch with the openSUSE Testing Core Team to get a test plan created. The team can be found on the opensuse-testing mailing list and on the #opensuse-testing IRC channel on Freenode.
Carsten Book, QA Investigations Engineer at the Mozilla Corporation,
gave a talk about how to get involved in the Mozilla Project and he focused
on Mozilla QA, which has its home on the QMO web site. This QA portal has a
lot of documentation,
e.g. for getting started
with QA. And there are links to various Mozilla QA tools such as Bugzilla, Crash Reporter,
the Litmus system that has test
cases written by Mozilla QA for manual software testing, and some tools to
testing. For example, Mozilla's test system automatically checks
whether performance has degraded after every check-in of a new feature, to
try to ensure that Firefox won't get any slower.
People who want to help test can of course run a nightly build and file bug reports. There are also Mozilla test days that teach how to get development builds, how to file bugs, and how to work with developers on producing a fix. Contributors with some technical expertise can join one of the quality teams, each focusing on a specific area: Automation, Desktop Firefox, Browser Technologies, WebQA, and Services. Each of the teams has a short but instructive web page with information about what they do and how you can contact them.
An important point that Carsten made was that it should also be easy for
interested people to immediately get an overview of different areas where
they can contribute without having to read dozens of wiki pages. Mozilla
even has a special Get
involved page where you just enter your email address and an area of
interest, with an optional message. After submitting the form, you will get an email to put you in touch with the right person.
Low entry barrier
These three projects are all about lowering the barriers for new testers
— to be able to attract as many testers as possible and to make the
life of existing testers easier — by automating boring and repetitive
tasks. In this way you can keep testers motivated. Wiedemann's autoinst
project seems especially interesting: at the moment it has just the basic features, but it has a lot of potential, e.g. if the feature for comparing screen shots is refined. From a technical point of view, this is an exciting testing project that hopefully finds its way into other distributions.
Comments (5 posted)
Wayland's not a usable default yet. It'll probably be packaged in F15
as something you can play with. We don't even have a complete list of
transition criteria yet, let alone a timeframe for switching the
default. But it's likely to happen eventually because it's a serious
win for a lot of things, and the downsides are pretty negligible despite
the fear from the peanut gallery.
-- Adam Jackson
The only hardened opinion I have is that network transparency is an
essential opinion. Beyond that I have no clue. I'm waiting to be
educated. If only Adam Jackson were responding I would have walked
away satisfied by now. Perhaps I should ignore everyone else. :)
-- Gregory Maxwell
Comments (17 posted)
RHEL6 is available. "Enhancements range from kernel improvements for
resource management, RAS, performance, scalability, virtualization and
power saving, through a greatly extended and standards-compliant
development environment, to a comprehensive range of updated server and
desktop applications. It is designed to improve agility, lower costs and
reduce IT complexity for customers.
Comments (7 posted)
The minutes from the November 8 Fedora Board meeting include a discussion
of whether SQLninja
- an SQL
injection testing tool - should be included in the distribution. The
answer was "no," and a new guideline was added to the rules: "Where,
objectively speaking, the package has essentially no useful foreseeable
purposes other than those that are highly likely to be illegal or unlawful
in one or more major jurisdictions in which Fedora is distributed or used,
such that distributors of Fedora will face heightened legal risk if Fedora
were to include the package, then the Fedora Project Board has discretion
to deny inclusion of the package for that reason alone.
Full Story (comments: 47)
Jared Smith welcomes
the Fedora Project's new release engineering team lead, Dennis Gilmore, and
the new Fedora Program Manager, Robyn Bergeron. "Just as with nature, we have cyclical changes within the Fedora Project as well. I think it's both useful and healthy to point out a few of those changes, for a couple of reasons. First of all, I want to point out that every person in the Fedora community is a potential leader. Our policies of rotating leadership help ensure that everyone who is so inclined has a chance to lead and serve. Second, I'd like to personally thank those people who have diligently served the Fedora community, and wish them success as they move on to other endeavors.
Comments (none posted)
Mandriva has a new
for "cooker", the development branch. "Eugeni Dodonov. Eugeni is well known in the community, a very active Mandriva's contributor, an activist of free software in Brasil, and also a doctor in computer science.
Comments (none posted)
Mark Shuttleworth has described
the next major step for the Unity interface: putting it on the Wayland
OpenGL-based display system.
"But we don't believe X is setup to deliver the user experience we want, with super-smooth graphics and effects. I understand that it's *possible* to get amazing results with X, but it's extremely hard, and isn't going to get easier. Some of the core goals of X make it harder to achieve these user experiences on X than on native GL, we're choosing to prioritize the quality of experience over those original values, like network transparency.
Comments (173 posted)
The Mageia Blog has a progress
on the new Mandriva offshoot, Mageia. Topics include the
buildsystem, blog and website, mirrors, wiki, and a roadmap.
Comments (none posted)
Newsletters and articles of interest
Comments (none posted)
Dave Neary has posted a
MeeGo progress report
on the Vision Mobile site. "Long-awaited
MeeGo compliance specifications have resulted in drawn out and sometimes
acrimonious debate. Trademark guidelines have been a sticking point for
community ports of the MeeGo netbook UX to Linux when these ports do not
include required core components. Related to the technical governance of
the project, there is some uncertainty around the release process, and the
means and criteria which will be used when considering the inclusion of new
components. And there are some signs that the 'all open, all the time'
message at the project launch has been tempered by the reality of building
a commercial device.
Comments (7 posted)
The H takes Fedora 14 for a test drive
and finds it a little thin on new features. There is, of course, the usual pile of "up-to-the-minute
" versions of various packages. "Some Fedora users will also feel the benefits of an update, with Fedora 14 including newer versions of many applications. Version 14 includes hundreds of enhancements and bug fixes not explicitly mentioned above the new features to be found in OpenOffice 3.3 are just one example of many. Objectively speaking, Laughlin contains significantly fewer enhancements than previous versions. The likely culprit is the end of the development phase of Red Hat Enterprise Linux 6, which has kept many of the Fedora developers employed by Red Hat busy over recent months.
Comments (8 posted)
Susan Linton takes
at the upcoming release of Pardus 2011. "Pardus Linux, a popular independent distribution funded and developed by the Scientific & Technological Research Council of Turkey, will be releasing version 2011 in the coming weeks and with it lots of nice updates and improvements.
Comments (none posted)
Joe "Zonker" Brockmeier reveals
live CD distributions. "So how were the distros chosen? You'll notice that none of the major Linux distros (a.k.a. Ubuntu, Debian, Fedora, openSUSE, Slackware, etc.) appear in the list, though most of the picks are derived from one of the major distros. Though Ubuntu, Linux Mint, et al. have perfectly serviceable live CDs or DVDs, they're not really designed for long-term use as a live distro. I'm sure some folks do use them that way, but they're the cream of the crop for installing to a hard drive - not for live media.
Comments (none posted)
Page editor: Rebecca Sobol
Next page: Development>>