openSUSE announces first public release of openQA [LWN.net]

From:		Jos Poortvliet <jos-AT-opensuse.org>
To:		opensuse-announce-AT-opensuse.org
Subject:		[opensuse-announce] openSUSE announces first public release of openQA
Date:		Tue, 11 Oct 2011 16:01:05 +0200
Message-ID:		<201110111601.06048.jos@opensuse.org>
Archive-link:		Article, Thread

OpenSUSE Announces  First Public Release of openQA
The openSUSE Project announces the 1.0 release of the unique cross-
distribution-capable, fully automated testing framework openQA. openQA 
is the only comprehensive testing tool which can run tests on every 
level of the OS, from core functionality like the bootloader and booting 
the kernel up to testing applications like Firefox and LibreOffice. It 
shows the results in a convenient web interface and allows testers to 
see screenshots and even videos of the issues found. openQA is used to 
run nightly tests of the 'Factory' development repository for the 
upcoming openSUSE 12.1 release. openQA is available under the GPL 
version 2 or later.

Software Testing is Hard
Testing complex software is a crucial part of development. However, in-
depth, frequent testing is difficult, time-consuming and boring. To 
ensure that software is tested regularly and predictably, automation is 
crucial. Many in-house tools which can test specific applications have 
been developed but there was no comprehensive testing tool for a full 
operating system until openQA. openQA does its work entirely hands-off 
and offers a convenient web interface with an overview of the results. 
With openQA developers can catch and fix issues in the development 
process much faster and thus ensure a more reliable product.

openQA Makes it Easy
openQA can test any operating system which is able to run in a virtual 
machine. By taking screenshots of the process and comparing these to 
reference images it determines if tests are passed or not. The OS-
autoinst tool, part of openQA, controls the testing process by sending 
virtual keyboard and mouse events to the operating system being 
installed and run. It is able to respond to what is shown on the screen 
so it can handle a variety of issues while not having to stop the test.

openQA determines if the installation of the operating system worked 
correctly and whether a number of applications started properly. At the 
end of the test, all images are encoded into a video for consultation by 
a human tester in case this is needed. A status overview of the tests 
can be shown in a web interface, currently available for the openSUSE 
Factory development repository on openqa.opensuse.org.

Coolo, openSUSE release manager noted: "I'm thrilled to see the 1.0 
release of openQA. openQA's automated testing has been very beneficial 
to the stability of openSUSE Factory and I'm confident it can do the 
same for other Linux distributions"

openQA is Cross-platform Capable
openQA is developed as two software pieces: OS-autoinst and the web 
interface. OS-autoinst right now supports Fedora, Ubuntu, Debian, 
FreeBSD and even OpenIndiana. It can be used to test even MS DOS and 
Windows systems. A number of videos on how openQA works can be found at 
http://video.os-autoinst.zq1.de/opensuse/video/

The openQA web interface is openSUSE specific right now. It gives an 
overview of the status of all tests and allows the tester to look at 
specific screenshots of issues as well as video's of the entire testing 
process.

openQA is Free
openQA has been published under the General Public License version 2 or 
later. Installation instructions can be found on the wiki and the team 
is eager to hear feedback on openQA and use cases.

About openSUSE
The openSUSE project is a worldwide effort that promotes the use of 
Linux everywhere. openSUSE creates one of the world's best Linux 
distributions, working together in an open, transparent and friendly 
manner as part of the worldwide Free and Open Source Software community. 

The project is controlled by its community and relies on the 
contributions of individuals, working as testers, writers, translators, 
usability experts, artists and ambassadors or developers. The project 
embraces a wide variety of technology, people with different levels of 
expertise, speaking different languages and having different cultural 
backgrounds.

Important links
openQA installation & web interface on opensuse.org: 
http://openqa.opensuse.org
OS-autoinst tool: http://www.os-autoinst.org/
openSUSE 12.1 Portal: http://en.opensuse.org/Portal:12.1

(Log in to post comments)

openSUSE announces first public release of openQA

Posted Oct 12, 2011 3:12 UTC (Wed) by xxiao (subscriber, #9631) [Link]

I assume this is their new OBS?

openSUSE announces first public release of openQA

Posted Oct 12, 2011 14:58 UTC (Wed) by jospoortvliet (subscriber, #33164) [Link]

Yes, to some extend it is. On the other hand the two are quite different too. OBS is technically vastly more complex but also has a much wider audience.

Please also note that we made a bit of a mistake, not being aware of Tapper, a tool developed by AMD with a very similar scope (although it works quite different as far as I can tell):

http://developer.amd.com/zones/opensource/amdtapper/pages...

openSUSE announces first public release of openQA

Posted Oct 12, 2011 3:40 UTC (Wed) by klevin (subscriber, #36526) [Link]

To each, their own. The fact that it uses Perl is a massive plus, for me. Languages are just a covenant between the coder and the compiler/interpreter that specific syntax will have specific results. Whether the whole of the code is well-written is a responisibility resting on the coder's shoulders. Everything else is just eye candy governed by the path of least resistence.

Besides that, this has potential to be hugely useful for the sort of projects I've been working on lately.

openSUSE announces first public release of openQA

Posted Oct 12, 2011 15:06 UTC (Wed) by jospoortvliet (subscriber, #33164) [Link]

About the screenshots, you're right that this is a serious limitation. It is OK for the continuous testing openQA is meant for but more in-depth testing is far too sensitive to changes in the UI or artwork of applications to be sustainable. You'd be updating screenshot md5sums far too often...

openSUSE announces first public release of openQA

Posted Oct 13, 2011 10:06 UTC (Thu) by bmwiedemann (guest, #71319) [Link]

1) when you look at the test modules in the os-autoinst/consoletest.d/ directory, you could just imagine the perl-code describing a test to be a domain-specific-language.

Of course, one could write such a framework in ruby (which is a nice language), but for me perl was the tool that I used for 11 years - and often the most useful tools are the ones you know best.

2) It also supports comparing (rectangular parts of) screenshots pixel by pixel - one of the cool additions from Dominik Heidler.
I agree that this is much more maintainable as you can see what it is supposed to look like and in future one could add fuzzy matching based on it.
The only remaining advantage of the original md5sums is that they are incredibly fast, which is useful while the test is running.

A limitation

Posted Oct 12, 2011 11:55 UTC (Wed) by NAR (subscriber, #1313) [Link]

It depends very much on the software you're working on. In my experience most of the problems are caused by users not reading the documentation and having no system administration expertise (installing software is system administration, so it's their bosses fault that they give tasks to non-competent people).

A limitation

Posted Oct 12, 2011 15:00 UTC (Wed) by jospoortvliet (subscriber, #33164) [Link]

Yes, this is one of the main limitations of openQA: it does not test hardware.

openQA is not really very suitable for 'final' or 'only' testing of a product. You'll need far more for that. It is however incredibly good for continuous, automated testing during the development process as it will tell you right away when someone broke the bootup procedure, or some core library, or the installer, etcetera.

openSUSE announces first public release of openQA

Posted Oct 13, 2011 17:03 UTC (Thu) by edolstra (guest, #80775) [Link]

It's not quite true that openQA is the "only comprehensive testing tool which can run tests on every level of the OS". Hydra, the continuous build system used by NixOS and a bunch of other projects (including some GNU projects), does that as well. For instance, every time a NixOS developer commits a change to any component (whether it's the kernel, boot scripts, GCC, KDE, or whatever), we automatically rebuild the entire system and instantiate VMs to run tests. For instance, there is a test for KDE in NixOS. There are also various automated tests that build the NixOS installation CD and install it in a VM. A nice feature is that it can test distributed systems by instantiating networks of virtual VMs (example). The whole VM-based system testing approach is described in a paper.