openSUSE Conference 2010: Making testing easier [LWN.net]

openSUSE Conference 2010: Making testing easier

Posted Nov 14, 2010 14:22 UTC (Sun) by arief (guest, #58729)
Parent article: openSUSE Conference 2010: Making testing easier

Automated testing is excellent idea.

Although I still believe that real end-user testing is a must have for final QA.

I dont know if open-source processes can guarantee that end-user testing. Anyone care to enlight here?

All the best.
-arief

(Log in to post comments)

openSUSE Conference 2010: Making testing easier

Posted Nov 18, 2010 12:59 UTC (Thu) by bmwiedemann (guest, #71319) [Link]

Hello arief,

thanks for the feedback. You are right in that automated testing alone is not good enough for a final release for several reasons (which I also mentioned in the talk).
a) is Qemu/KVM only emulates one certain combination of hardware, so 99% of drivers will not be tested by it.
b) see Dijkstra's quote above - there is a huge number of possible code pathes in most programs and one test-run can only exercise one such path.
c) there are also a big number of packages in modern distributions and few come with their own test-suites
d) software (especially open-source) gets updated a lot and keeping tests for it in a working state can be quite some effort

The automated testing is intended to save developers/testers/users from obvious blocker bugs in the unstable-release-of-the-day (named openSUSE-Factory / Fedora-Rawhide or Debian-sid).

Ciao
Bernhard M. Wiedemann