Development

openSUSE introduces openQA

Writing software, at least for most open source contributors, is the fun part. Testing, on the other hand, is often seen as boring and repetitive work. The openSUSE project is hoping that openQA will take some of the drudgery out of testing and boost the quality of released software.

The openQA project has been in the works for some time, but the openSUSE project officially unveiled the 1.0 release of openQA on October 11. According to the release notes from openSUSE community manager Jos Poortvliet, "openQA is the only comprehensive testing tool which can run tests on every level of the OS, from core functionality like the bootloader and booting the kernel up to testing applications like Firefox and LibreOffice."

The problem

The problem that openQA creator Bernhard Wiedemann and others in the openSUSE community are trying to solve is the chicken-and-egg problem with testing distributions. That is, users complain that early pre-releases are too buggy to even test. Developers complain that users don't start testing (and therefore, filing bugs) until release candidates — which is too late to fix some bugs. This has been a problem for openSUSE releases, as well as for other distributions. openQA, on the other hand, lets the computers do much of the work so that when users enter the picture they're not discouraged from testing by bugs that prevent even installing the distribution.

That, and the fact that frequent and in-depth testing of software is "difficult, time-consuming and boring." Lots of QA tools exist, but Poortvliet says that openQA is the first of its kind for comprehensive testing of a full operating system. openSUSE has been using openQA to test the openSUSE Factory (development) distribution for quite some time. Wiedemann mentioned it in a post in January, pointing users to the openQA overview page that shows testing results for the latest Factory release.

Using openQA, openSUSE has been able to provide tested Factory releases that can guarantee a few minimum requirements. For example, users should be confident that the installer works and X11 will start, at least in a KVM virtual host. (It is, of course, impossible to automate testing for all physical hardware users might have.) Users should also find that the Zypper package manager is capable of installing software on the system.

What openQA is

The openQA project is actually two parts, OS-autoinst and the openQA web interface. The OS-autoinst project is operating system-independent, and Wiedemann says that it should work with just about any OS that can be run under KVM or VirtualBox. The supported OSes for OS-autoinst are openSUSE, Fedora, Debian, SUSE Linux Enterprise Server, FreeBSD, and OpenIndiana. Currently, the openQA web interface is openSUSE-specific and integrated with the Open Build System, though both projects are GPLv2-licensed and any project could take the pair and adapt them for testing another operating system.

OS-autoinst can be used to test everything from operating system components like the bootloader, installer, and kernel to the end-user applications like Firefox or office suites. Test modules are written in Perl. A simple example of an OS-autoinst module is on the site, with a line-by-line explanation. The OS-autoinst system spins up a virtual machine of the test system and compares screenshots of the system to reference images to determine if tests are passed or not.

The openQA web site provides test overviews with full test results, and movies and screenshots captured during testing. The test results show how many tests passed with an "OK", "unknown", or "fail" result. Users can also view the testing scripts in the web interface to see what was tested. It's also worth noting that openQA has an IRC notification bot, so users can send test notifications to an IRC channel. The openSUSE project is using that to send test results to #opensuse-openqa on Freenode.

Limitations

The openQA project can do a lot of automated testing, but it still can't test everything. For one thing, if new artwork comes in or the resolution changes for a testing scenario, Wiedemann says that tests will come back with an "unknown" result instead of failing or passing a test. Then users will have check it manually. "This can be done over the openQA web interface in a matter of seconds." However, he notes that with openSUSE "console tests and tests in xterm hardly ever change (approximately once since 11.1 was released ~3 years ago)."

He also notes that "there is a quasi infinite number of inputs even to something as simple as 'a+b' and thus, to keep things simple, openQA just tests the core/base/default-case of important programs." In other words, there's only so much automated testing will get you before actual user-testing is required. Another limitation for openQA is that hardware and driver testing is limited to KVM-supported drivers. Wiedemann says that 99% of drivers can't be tested with openQA, so that will also be left to users to try.

openQA is also not going to work well with programs that have "things that vanish quickly" like pop-ups that close themselves. Then again, discouraging developers from having a pop-up that closes itself automatically might not be a bad thing. Programs that change themes or UI elements a lot can cause problems as well. Wiedemann singled out Firefox and LibreOffice as difficult programs to test with openQA. Finally, Wiedemann says that mouse-only input is difficult to test with openQA due to a qemu bug. Perhaps that will change if the bug is fixed.

Apart from that, Wiedemann says "nearly everything is testable."

The future

The idea, of course, is not only to use the openQA project for openSUSE, but to provide it to other projects as well. Wiedemann says that he's talked to Debian and Fedora project members, but hasn't heard of it being used by either project. He also says that the most interest shown so far, outside of openSUSE, has come from people developing antivirus software or doing QA for antivirus software. "Which surprised me at first, but they certainly do OS-level software with potential of wrecking people's lives, so they could profit from automated testing at the OS level."

Developers who want to pitch in or start using OS-autoinst for themselves can check out the code from Gitorious and consult the short user guide or install guide. The web-based interface is also on Gitorious. Some of the items on the to-do list for now include adding stress-tests for btrfs, Ext4, and XFS. Also planned is to add audio tests for Amarok and Banshee, and to add more application tests.

Overall, the openQA project looks fairly impressive. It seems to be working well for the openSUSE project, one hopes that it might find use in other projects as well.

Comments (2 posted)

Brief items

Quotes of the week

-- Matt Mackall

-- Colin McMillen et al

-- Jesús Corrius

-- Michael Meeks

Comments (15 posted)

LedgerSMB 1.3.0 released

Version 1.3.0 of the LedgerSMB accounting system has been released. "This is the most significant release to date in many ways. It is the most secure release, and it performs the best under heavy load. At the same time, it also provides the most features that many businesses rely on heavily. This means that LedgerSMB 1.3.0 is suitable for a much larger businesses and installations than previous versions." There are many new features beyond the security enhancements; see the announcement for details.

Full Story (comments: 2)

Owncloud 2 released

Owncloud is a system for the creation and management of personal cloud resources; the Owncloud 2 release is now available. It includes support for file access and sharing, network music streaming, calendar management, contact management, and more. (LWN looked at Owncloud in 2010).

Comments (2 posted)

pgwatch 1.0

Pgwatch is a monitoring tool for PostgreSQL 9.x databases; the 1.0 release is now available. See the pgwatch page for details, downloads, and lots of screenshots. (Note that the announcement claims that pgwatch is under "the Creative Commons License," but the actual license is Artistic 2.0).

Full Story (comments: none)

Plasma Active One released

The KDE project has announced the first release of the Plasma Active interface for tablets and other touchscreen-oriented devices. "Plasma Active One's touchscreen interface is more than just an application launcher. As soon as the device is turned on, rather than the traditional grid of applications, you see the Activities view showing your current project, task or idea. With Activities, you can collect all of the documents, people, web sites, media and widgets related to a topic in one place, building personalized and interactive views of your life."

Comments (9 posted)

Samba now accepts corporate-copyrighted code

The Samba project has changed its longstanding policy and will now accept code contributions owned by corporations. They have adopted a kernel-inspired "certificate of origin" which must be filed with the project; after that, corporate-owned code can go in with a simple Signed-off-by line. See the announcement (click below) for the details of how it works.

Full Story (comments: none)

Sphinx 1.1 released

Version 1.1 of the Sphinx documentation system (reviewed in LWN in 2010) has been released. Significant changes include Python 3 support, easier inline index entries, and a number of new builders (texinfo, gettext, and websupport).

Full Story (comments: none)

Subversion 1.7.0 released

Version 1.7.0 of the subversion source code management system is out. New features include centralized metadata storage, better HTTP support, a new remote dump tool, a new "svn patch" command and more; see the release notes for details.

Full Story (comments: 9)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (October 11)
• GCC 4.7.0 status report (October 11)
• Perl Weekly (October 10)
• PostgreSQL Weekly News (October 9)

Comments (none posted)

Novacut delivers distributed collaboration in HD-SLR video editing (Opensource.com)

Opensource.com has a review of the Novacut video editor. "In other words, Novacut creates a truly agile environment for the creative process. While this may not be a feature that a one-man show would need for a personal film project, it's something that only the most expensive editing solutions even begin to touch. For a free, GPLv3 video editor to be based on such a paradigm is not just unique; it's a game-changer."

Comments (14 posted)

Oracle works on Dtrace for Linux (The H)

The H reports on Oracle's plans to bring DTrace to (Oracle) Linux. "There is no word on what the Linux port of 'Dynamic Tracing Facility' (also known as Dtrace) will look like, but it is considered a major function in newer versions of Solaris. A blog post by Dtrace co-inventor Adam Leventhal indicates that few details are known, though he does say that it will be interesting to see how Oracle clears up licensing issues."

Comments (23 posted)

Time zone database shut down (The Daily Parker)

The Daily Parker reports on the shut down of the Olson time zone database (used by most Unix-like systems) due to a copyright infringement lawsuit. Evidently, an astrology company bought the rights to The American Atlas, which was used (and credited) for some of the historical time zone information, and now the company is suing for infringement of something that looks an awful lot like facts (which are not supposed to be copyrightable). "What's even stupider about this lawsuit is that comments in the database encourage people to buy the book. So even if Astrolabe owns the copyright to the facts about time zone rules—a troubling proposition—their republication in the Olson database increases the likelihood that they'll make money off it." (Thanks to Dirkjan Ochtman for the heads up.)

Comments (74 posted)

Page editor: Jonathan Corbet
Next page: Announcements>>