openSUSE introduces openQA

Writing software, at least for most open source contributors, is the fun part. Testing, on the other hand, is often seen as boring and repetitive work. The openSUSE project is hoping that openQA will take some of the drudgery out of testing and boost the quality of released software.

The openQA project has been in the works for some time, but the openSUSE project officially unveiled the 1.0 release of openQA on October 11. According to the release notes from openSUSE community manager Jos Poortvliet, "openQA is the only comprehensive testing tool which can run tests on every level of the OS, from core functionality like the bootloader and booting the kernel up to testing applications like Firefox and LibreOffice."

The problem

The problem that openQA creator Bernhard Wiedemann and others in the openSUSE community are trying to solve is the chicken-and-egg problem with testing distributions. That is, users complain that early pre-releases are too buggy to even test. Developers complain that users don't start testing (and therefore, filing bugs) until release candidates — which is too late to fix some bugs. This has been a problem for openSUSE releases, as well as for other distributions. openQA, on the other hand, lets the computers do much of the work so that when users enter the picture they're not discouraged from testing by bugs that prevent even installing the distribution.

That, and the fact that frequent and in-depth testing of software is "difficult, time-consuming and boring." Lots of QA tools exist, but Poortvliet says that openQA is the first of its kind for comprehensive testing of a full operating system. openSUSE has been using openQA to test the openSUSE Factory (development) distribution for quite some time. Wiedemann mentioned it in a post in January, pointing users to the openQA overview page that shows testing results for the latest Factory release.

Using openQA, openSUSE has been able to provide tested Factory releases that can guarantee a few minimum requirements. For example, users should be confident that the installer works and X11 will start, at least in a KVM virtual host. (It is, of course, impossible to automate testing for all physical hardware users might have.) Users should also find that the Zypper package manager is capable of installing software on the system.

What openQA is

The openQA project is actually two parts, OS-autoinst and the openQA web interface. The OS-autoinst project is operating system-independent, and Wiedemann says that it should work with just about any OS that can be run under KVM or VirtualBox. The supported OSes for OS-autoinst are openSUSE, Fedora, Debian, SUSE Linux Enterprise Server, FreeBSD, and OpenIndiana. Currently, the openQA web interface is openSUSE-specific and integrated with the Open Build System, though both projects are GPLv2-licensed and any project could take the pair and adapt them for testing another operating system.

OS-autoinst can be used to test everything from operating system components like the bootloader, installer, and kernel to the end-user applications like Firefox or office suites. Test modules are written in Perl. A simple example of an OS-autoinst module is on the site, with a line-by-line explanation. The OS-autoinst system spins up a virtual machine of the test system and compares screenshots of the system to reference images to determine if tests are passed or not.

The openQA web site provides test overviews with full test results, and movies and screenshots captured during testing. The test results show how many tests passed with an "OK", "unknown", or "fail" result. Users can also view the testing scripts in the web interface to see what was tested. It's also worth noting that openQA has an IRC notification bot, so users can send test notifications to an IRC channel. The openSUSE project is using that to send test results to #opensuse-openqa on Freenode.

Limitations

The openQA project can do a lot of automated testing, but it still can't test everything. For one thing, if new artwork comes in or the resolution changes for a testing scenario, Wiedemann says that tests will come back with an "unknown" result instead of failing or passing a test. Then users will have check it manually. "This can be done over the openQA web interface in a matter of seconds." However, he notes that with openSUSE "console tests and tests in xterm hardly ever change (approximately once since 11.1 was released ~3 years ago)."

He also notes that "there is a quasi infinite number of inputs even to something as simple as 'a+b' and thus, to keep things simple, openQA just tests the core/base/default-case of important programs." In other words, there's only so much automated testing will get you before actual user-testing is required. Another limitation for openQA is that hardware and driver testing is limited to KVM-supported drivers. Wiedemann says that 99% of drivers can't be tested with openQA, so that will also be left to users to try.

openQA is also not going to work well with programs that have "things that vanish quickly" like pop-ups that close themselves. Then again, discouraging developers from having a pop-up that closes itself automatically might not be a bad thing. Programs that change themes or UI elements a lot can cause problems as well. Wiedemann singled out Firefox and LibreOffice as difficult programs to test with openQA. Finally, Wiedemann says that mouse-only input is difficult to test with openQA due to a qemu bug. Perhaps that will change if the bug is fixed.

Apart from that, Wiedemann says "nearly everything is testable."

The future

The idea, of course, is not only to use the openQA project for openSUSE, but to provide it to other projects as well. Wiedemann says that he's talked to Debian and Fedora project members, but hasn't heard of it being used by either project. He also says that the most interest shown so far, outside of openSUSE, has come from people developing antivirus software or doing QA for antivirus software. "Which surprised me at first, but they certainly do OS-level software with potential of wrecking people's lives, so they could profit from automated testing at the OS level."

Developers who want to pitch in or start using OS-autoinst for themselves can check out the code from Gitorious and consult the short user guide or install guide. The web-based interface is also on Gitorious. Some of the items on the to-do list for now include adding stress-tests for btrfs, Ext4, and XFS. Also planned is to add audio tests for Amarok and Banshee, and to add more application tests.

Overall, the openQA project looks fairly impressive. It seems to be working well for the openSUSE project, one hopes that it might find use in other projects as well.

---

(Log in to post comments)