| |||||||||||||
openSUSE Conference 2010: Making testing easieropenSUSE Conference 2010: Making testing easierPosted Nov 12, 2010 1:03 UTC (Fri) by gerdesj (subscriber, #5446)Parent article: openSUSE Conference 2010: Making testing easier
Rather than comparing screen shots perhaps a more formal test harness could be put into the installer.
For the installer, each screen or function paints a particular pixel or another signal which can be easily detected from the "outside". To make sure that these signatures are correct a registry could be set up and a script run against the source checking that the signatures match the section of code.
To spell it out - there is a specification that says what should happen at each stage of the process and a signal is generated at each "audit point" which is detectable from the outside of the system. Automated tools verify that the spec's requirements are represented in the code and another one checks the signals from a runtime session.
I don't wish to denigrate someone's hard work that is clearly for the benefit of the SuSE community but surely a bit of co-ordination wouldn't hurt here.
Cheers
(Log in to post comments)
openSUSE Conference 2010: Making testing easier Posted Nov 12, 2010 20:45 UTC (Fri) by joey (subscriber, #328) [Link]
I used to run automated testing of Debian Installer. I originally used a whole rack full of hardware, with serial cables and automated power cycling in order to test a lot of architectures: mips, ia64, alpha, i386, amd64, sparc, hppa. (Qemu supports some of them now; s390 was always run in emulation.)
It was a lot of work, but I think very valuable at certain points in the development of the installer and distribution.
Anyway, in the Debian Installer we may have it easier since our UI frontends are abstracted via debconf, and in 99% of cases, the same code is running whether a graphical UI or a console UI is being used. So the test harness can just boot it in text mode and use expect scripts.
openSUSE Conference 2010: Making testing easier Posted Nov 15, 2010 13:33 UTC (Mon) by jschrod (subscriber, #1646) [Link]
I would never trust the output of such a signal to be identical with "everything has been OK up to now". 30 years of writing and testing software / managing software projects taught me that this won't be the case in general. Appearance of such a signal would just mean "we have reached this point in the installation" -- but would not be any indication that no errors have happened.
Bernhard's model is good because it is robust and does not depend on correct behaviour of a part of the to-be-tested software. This would be the case with your proposal. Therefore, I think that Bernhard's approach is actually better.
(I don't know the software in question and if the implementation is robust; my comment is based on the article and your comment.)
|
|||||||||||||