Writing software, at least for most open source contributors, is the fun part. Testing, on the other hand, is often seen as boring and repetitive work. The openSUSE project is hoping that openQA will take some of the drudgery out of testing and boost the quality of released software.
The openQA project has been in the works for some time, but the openSUSE project officially unveiled the 1.0 release of openQA on October 11. According to the release notes from openSUSE community manager Jos Poortvliet, "openQA is the only comprehensive testing tool which can run tests on every level of the OS, from core functionality like the bootloader and booting the kernel up to testing applications like Firefox and LibreOffice."
The problem that openQA creator Bernhard Wiedemann and others in the
openSUSE community are trying to solve is the chicken-and-egg
problem with testing distributions. That is, users complain that
early pre-releases are too buggy to even test. Developers complain that users
don't start testing (and therefore, filing bugs) until release candidates
— which is too late to fix some bugs. This has been a problem for
openSUSE releases, as
well as for other distributions. openQA, on the other hand, lets
the computers do much of the work so that when users enter the picture
they're not discouraged from testing by bugs that prevent even installing
That, and the fact that frequent and in-depth testing of software is
"difficult, time-consuming and boring." Lots of QA tools
exist, but Poortvliet says that openQA is the first of its kind for
comprehensive testing of a full operating system. openSUSE has been using openQA to test the openSUSE Factory (development) distribution for quite some time. Wiedemann mentioned it in a post in January, pointing users to the openQA overview page that shows testing results for the latest Factory release.
Using openQA, openSUSE has been able to provide tested Factory releases
that can guarantee a few minimum requirements. For example, users should be
confident that the installer works and X11 will start, at least in a KVM
virtual host. (It is, of course, impossible to automate testing for all
physical hardware users might have.) Users should also find that the Zypper
package manager is capable of installing software on the system.
What openQA is
The openQA project is actually two parts, OS-autoinst and the openQA web interface. The OS-autoinst project is operating system-independent, and Wiedemann says that it should work with just about any OS that can be run under KVM or VirtualBox. The supported OSes for OS-autoinst are openSUSE, Fedora, Debian, SUSE Linux Enterprise Server, FreeBSD, and OpenIndiana. Currently, the openQA web interface is openSUSE-specific and integrated with the Open Build System, though both projects are GPLv2-licensed and any project could take the pair and adapt them for testing another operating system.
OS-autoinst can be used to test everything from operating system components like the bootloader, installer, and kernel to the end-user applications like Firefox or office suites. Test modules are written in Perl. A simple example of an OS-autoinst module is on the site, with a line-by-line explanation. The OS-autoinst system spins up a virtual machine of the test system and compares screenshots of the system to reference images to determine if tests are passed or not.
The openQA web site provides test overviews with full test results, and
movies and screenshots captured during testing. The test results show how many
tests passed with an "OK", "unknown", or "fail" result. Users can also view
the testing scripts in the web interface to see what was tested. It's also worth noting that openQA has an IRC notification bot, so users can send test notifications to an IRC channel. The openSUSE project is using that to send test results to #opensuse-openqa on Freenode.
The openQA project can do a lot of automated testing, but it still can't test everything. For one thing, if new artwork comes in or the resolution changes for a testing scenario, Wiedemann says that tests will come back with an "unknown" result instead of failing or passing a test. Then users will have check it manually. "This can be done over the openQA web interface in a matter of seconds." However, he notes that with openSUSE "console tests and tests in xterm hardly ever change (approximately once since 11.1 was released ~3 years ago)."
He also notes that "there is a quasi infinite number of inputs
even to something as simple as 'a+b' and thus, to keep things simple,
openQA just tests the core/base/default-case of important programs."
In other words, there's only so much automated testing will get you before
actual user-testing is required. Another limitation for openQA is that hardware and driver testing is limited to KVM-supported drivers. Wiedemann says that 99% of drivers can't be tested with openQA, so that will also be left to users to try.
openQA is also not going to work well with programs that have
"things that vanish quickly" like pop-ups that close
themselves. Then again, discouraging developers from having a pop-up that
closes itself automatically might not be a bad thing. Programs that change themes or UI elements a lot can cause problems as well. Wiedemann singled out Firefox and LibreOffice as difficult programs to test with openQA. Finally, Wiedemann says that mouse-only input is difficult to test with openQA due to a qemu bug. Perhaps that will change if the bug is fixed.
Apart from that, Wiedemann says "nearly everything is testable."
The idea, of course, is not only to use the openQA project for openSUSE, but to provide it to other projects as well. Wiedemann says that he's talked to Debian and Fedora project members, but hasn't heard of it being used by either project. He also says that the most interest shown so far, outside of openSUSE, has come from people developing antivirus software or doing QA for antivirus software. "Which surprised me at first, but they certainly do OS-level software with potential of wrecking people's lives, so they could profit from automated testing at the OS level."
Developers who want to pitch in or start using OS-autoinst for
themselves can check out the
code from Gitorious and consult the short user guide
guide. The web-based interface is also on
Gitorious. Some of the items on the to-do list for now include
adding stress-tests for btrfs, Ext4, and XFS. Also planned is to add audio
tests for Amarok and Banshee, and to add more application tests.
Overall, the openQA project looks fairly impressive. It seems to be working well for the openSUSE project, one hopes that it might find use in other projects as well.
Comments (2 posted)
It'd probably be handy to have a small set of useful example commit and
push hooks published somewhere to enforce various things like:
- thou shalt not add new branches because thou read something about git
- thou shalt not push new heads even if thoust flail at the -f switch
- thou shalt not merge default into stable
- thou shalt not commit your JAR files
-- Matt Mackall
You appear to be advocating a new:
[ ] functional [ ] imperative [ ] object-oriented [ ] procedural [ ] stack-based
[ ] "multi-paradigm" [ ] lazy [ ] eager [ ] statically-typed [ ] dynamically-typed
[ ] pure [ ] impure [ ] non-hygienic [ ] visual [ ] beginner-friendly
[ ] non-programmer-friendly [ ] completely incomprehensible
programming language. Your language will not work. Here is why it will not work.
-- Colin McMillen et al
In my opinion, I don't think it's best for the project to put some
of the most skilled developers to work in documentation, when other
developers can also do this task perfectly well.
-- Jesús Corrius
Furthermore, I (personally) almost never read documentation - only
code or headers or specs - and the best hackers I know tend to do
-- Michael Meeks
Comments (15 posted)
Version 1.3.0 of the LedgerSMB accounting system has been released.
"This is the most significant release to date in many ways. It is
the most secure release, and it performs the best under heavy load. At the
same time, it also provides the most features that many businesses rely on
heavily. This means that LedgerSMB 1.3.0 is suitable for a much larger
businesses and installations than previous versions.
" There are
many new features beyond the security enhancements; see the announcement
Full Story (comments: 2)
is a system for the creation and
management of personal cloud resources; the Owncloud 2 release
available. It includes support for file access and sharing, network music
streaming, calendar management, contact management, and more. (LWN looked at Owncloud
Comments (2 posted)
Pgwatch is a monitoring tool for PostgreSQL 9.x databases; the 1.0 release
is now available. See the
for details, downloads, and lots of screenshots. (Note
that the announcement claims that pgwatch is under "the Creative Commons
License," but the
is Artistic 2.0).
Full Story (comments: none)
The KDE project has announced
first release of the Plasma Active
for tablets and other touchscreen-oriented devices.
"Plasma Active One's touchscreen
interface is more than just an application launcher. As soon as the device
is turned on, rather than the traditional grid of applications, you see the
Activities view showing your current project, task or idea. With
Activities, you can collect all of the documents, people, web sites, media
and widgets related to a topic in one place, building personalized and
interactive views of your life.
Comments (9 posted)
The Samba project has changed its longstanding policy and will now accept
code contributions owned by corporations. They have adopted a
kernel-inspired "certificate of origin" which must be filed with the
project; after that, corporate-owned code can go in with a simple
Signed-off-by line. See the announcement (click below) for the details of
how it works.
Full Story (comments: none)
Version 1.1 of the Sphinx documentation system (reviewed in LWN
in 2010) has been released.
Significant changes include Python 3 support, easier inline index
entries, and a number of new builders (texinfo, gettext, and websupport).
Full Story (comments: none)
Version 1.7.0 of the subversion source code management system is out. New
features include centralized metadata storage, better HTTP support, a new
remote dump tool, a new "svn patch
" command and more; see the
Full Story (comments: 9)
Newsletters and articles
Comments (none posted)
Opensource.com has a
of the Novacut video editor. "In other words, Novacut
creates a truly agile environment for the creative process. While this may
not be a feature that a one-man show would need for a personal film
project, it's something that only the most expensive editing solutions even
begin to touch. For a free, GPLv3 video editor to be based on such a
paradigm is not just unique; it's a game-changer.
Comments (14 posted)
The H reports
on Oracle's plans to bring DTrace to (Oracle) Linux. "There is no word on what the Linux port of 'Dynamic Tracing Facility' (also known as Dtrace) will look like, but it is considered a major function in newer versions of Solaris. A blog post by Dtrace co-inventor Adam Leventhal indicates that few details are known, though he does say that it will be interesting to see how Oracle clears up licensing issues.
Comments (23 posted)
The Daily Parker reports
on the shut down of the Olson time zone database (used by most Unix-like systems) due to a copyright infringement lawsuit. Evidently, an astrology company bought the rights to The American Atlas
, which was used (and credited) for some of the historical time zone information, and now the company is suing for infringement of something that looks an awful lot like facts
(which are not supposed to be copyrightable). "What's even stupider about this lawsuit is that comments in the database encourage people to buy the book. So even if Astrolabe owns the copyright to the facts about time zone rules—a troubling proposition—their republication in the Olson database increases the likelihood that they'll make money off it.
" (Thanks to Dirkjan Ochtman for the heads up.)
Comments (74 posted)
Page editor: Jonathan Corbet
Next page: Announcements>>