| Weekly edition | Kernel | Security | Distributions | Search |
| Archives | Calendar | Subscribe | Write for LWN | LWN.net FAQ |
Automated testing?
Automated testing?
Posted Mar 10, 2005 16:15 UTC (Thu) by cthulhu (guest, #4776)Parent article: Is the kernel development process broken?
Hi all,
I'm a perfect candidate to be a kernel tester, but I've never "tested," per se. Lately, I've tended to resist upgrades because I finally got all the things I need working, and due to lack of time. I do want to keep going forward, and I am currently engaged in a big (for me) effort to upgrade my several home machines to 2.6.x.
My method of testing is to configure/compile/install a new kernel, then see if my favorite things still work: X, audio, CD/DVD burner, USB devices. If they do, then I keep running said kernel until something crops up. At that point, I might make the connection with the kernel upgrade, but mostly only if it's a show-stopper.
So why I'm commenting, is that I'm wondering if there's some project that collects a bunch of test scripts for most of the standard features, like the things I mentioned above, plus perhaps file system testing to make sure there's no corruption.
Perhaps there is and it's commonly used - either way, it would probably be good to mention it here in case there are others like me who would get more active in testing if there was an easy way to do a bunch of it quickly...
(Log in to post comments)
Automated testing?
Posted Mar 10, 2005 17:48 UTC (Thu) by iabervon (subscriber, #722) [Link]
The hard thing is that there's a ton of possible hardware of the sort you mention, not all of it you want to use in testing (burn another set of test DVDs each time to test the burner?), and lots of it requires user interaction to verfiy the results. People had problems with 2.6.11 where audio worked fine, except that a changed interpretation of their saved mixer settings meant that no sound actually came out of their laptop docking station's speakers. From the point of view of the user, that's a bug, but you need a sense of hearing, a particular configuration, and a particular idea of what that configuration is supposed to mean in order to notice it. Many of the audio-related bugs these days seem to cause the sound to be distorted in some way, rather than giving wrong results that are accessible to the computer.
That said, it would probably be useful to have a program which would notice when you changed kernels, and would notice when you tried a device you hadn't tried with the current kernel, and inform you that this is the first time with that kernel and ask whether it worked. Then you could compare the hardware that you'd tested with the new kernel against the hardware you'd used previously, and consider testing your USB PDA cradle in advance of the PDA's batteries being low.
Automated testing?
Posted Mar 10, 2005 18:26 UTC (Thu) by cliffman (guest, #13144) [Link]
I'm always interested in getting more test scripts, and more test examples/tools. OSDL does a bunch of automated testing, we need all the help we can get. If you have any scripts, i'll give them a home and a web page.Send to cliffw@osdl.org
Automated testing?
Posted Mar 11, 2005 5:28 UTC (Fri) by dlang (subscriber, #313) [Link]
automated testing is great for making sure that a particular feature works (useually one that you broke in the past, causeing you to write the test), but it's not very good for testing various hardware combinations (you have to HAVE that hardware combination, and you only have so much room for systems) or finding _new_ problems (becouse you haven't written a test for it yet)
the kernel does go through automated testing, and it can be improved (see the request for help above), but to think that automated testing can eliminate all problems is not understanding the problem.
Coming to a distro near you...
Posted Mar 13, 2005 20:24 UTC (Sun) by Tobu (subscriber, #24111) [Link]
Ubuntu is experimentally doing that sort of shallow, wide ranging testing:I now have a small app that asks a few questions about how well the automatic config went, did the sound work out of the box, etc... and then sends that data plus a bunch of logs and hald reports upstream.
If ever an user is left in the cold with a kernel upgrade, a bit of datamining will make sure the problem is identified and fixed, without actually having to read all the logs.