Not logged in
Log in now
Create an account
Subscribe to LWN
LWN.net Weekly Edition for December 5, 2013
Deadline scheduling: coming soon?
LWN.net Weekly Edition for November 27, 2013
ACPI for ARM?
LWN.net Weekly Edition for November 21, 2013
Testing for kernel performance regressions
Posted Aug 3, 2012 23:32 UTC (Fri) by gregkh (subscriber, #8)
Posted Aug 4, 2012 8:33 UTC (Sat) by fhuberts (subscriber, #64683)
Posted Aug 4, 2012 10:11 UTC (Sat) by Cato (subscriber, #7643)
Organising this for Linux would be harder given a bootable system is required, but it could be done.
Posted Aug 4, 2012 11:49 UTC (Sat) by jnareb (subscriber, #46500)
This works so well because by default CPAN client does tests when installing modules, and send those results to CPANtesters. So it is very easy to become CPANtesters contributors.
Perhaps a request at install / upgrade time to perform regression benchmarks of one's system before first run would be a good idea for Linux testers project?
Posted Aug 4, 2012 15:09 UTC (Sat) by Cato (subscriber, #7643)
Posted Aug 4, 2012 10:27 UTC (Sat) by email@example.com (subscriber, #72030)
Posted Aug 4, 2012 14:42 UTC (Sat) by gregkh (subscriber, #8)
That's exactly what we do, and what we expect when we do the -rc releases.
You are running them to test that nothing breaks on your machine, right?
If not, please do so.
Posted Aug 4, 2012 21:34 UTC (Sat) by smoogen (subscriber, #97)
Posted Aug 6, 2012 10:24 UTC (Mon) by geertj (subscriber, #4116)
Wrong. I am not testing -rc releases, because i have other stuff to do. And i'm not complaining that my hardware doesn't work either, which makes my behavior wholly consistent. Just pre-empting that comment.. :)
There is a lot more that could be done to make this "crowdsourced testing" more effective. Currently it is quite difficult to test out -rc releases. You have to know how to compile a kernel, and how to install and run it in your distribution. Certainly not rocket science, but not easy for the average distro user, which is who you'd need to go after for large-scale outsourced testing.
Just an idea.. What if bootable live test images could be created for -rc releases? Ideally they would need no local storage, but optionally they could use a dedicated partition. The live image could do a whole bunch of tests and send back the results, together with information the hardware the tests ran it. Those could be analyzed for problems. Also the tests could include performance tests.
Every time a new -rc would be released, you'd rebuild the live image, and ask people via G+, Facebook, Twitter, the mailing list, etc, to burn it to a CD and boot their system with it. The CD runs for a few hours overnight, sends back the results, and then says "Thank you, i'm done". I bet you that this could increase your testing base by 10x. Of course you want to be pretty sure that it is safe and put a lot of safeguards in place to make sure it is (which of course doesn't mean you don't need a pretty scary disclaimer before running the CD).
Posted Aug 6, 2012 11:14 UTC (Mon) by niner (subscriber, #26151)
In short: it would be a step forward but there's still plenty of stuff which users would have to test manually. But of course: the perfect is the enemy of the good.
Posted Aug 8, 2012 16:33 UTC (Wed) by broonie (subscriber, #7078)
Posted Feb 7, 2013 10:52 UTC (Thu) by rbrito (subscriber, #66188)
Having to compile the kernels is a burden indeed, especially for those with weaker machines.
At least for Debian-based disributions it seems that Canonical provides daily compiled kernels, which is cool to have in mind (I only remembered that when I read your comment):
Posted Aug 4, 2012 23:11 UTC (Sat) by krakensden (subscriber, #72039)
It's a little sad.
Posted Aug 5, 2012 4:43 UTC (Sun) by dirtyepic (subscriber, #30178)
Posted Aug 4, 2012 10:30 UTC (Sat) by man_ls (guest, #15091)
Posted Aug 4, 2012 12:08 UTC (Sat) by robert_s (subscriber, #42402)
Posted Aug 4, 2012 15:21 UTC (Sat) by man_ls (guest, #15091)
Some examples: mount a virtualized SATA disk, format it, create a few files, read them back and check their contents. Mount a virtualized USB disk and do the same. Create several filesystems and stress-test them. Check that the commands issued are in correct order. And so on.
This is (obviously) spoken from utter ignorance of kernel internals, just from the point of view of basic software engineering: if it is not tested and verified it is not finished. I am sure kernel devs will know how to implement the idea or ignore it if the effort is not worth it. But for me it would be a fascinating project.
Posted Aug 4, 2012 19:09 UTC (Sat) by robert_s (subscriber, #42402)
But that's the stuff that gets tested anyway, by people. The trouble _is_ with the esoteric hardware.
Posted Aug 4, 2012 22:39 UTC (Sat) by man_ls (guest, #15091)
Posted Aug 6, 2012 11:17 UTC (Mon) by niner (subscriber, #26151)
Posted Aug 6, 2012 10:27 UTC (Mon) by njd27 (subscriber, #5770)
I actually work on the Linux driver for a particular family of input devices from one manufacturer. Most of the customers are integrating devices which are targeting older kernel versions rather than mainline, so our main focus is there. But we do track patches that are going into mainline and make sure they are sane, and do some occasional testing.
Device driver testing
Posted Aug 4, 2012 10:33 UTC (Sat) by ajb (guest, #9694)
In practice, I suspect most hardware companies don't have sufficient incentive to do this, but they might for some products. Some teams also don't really have a model, preferring to develop their device using FPGAs. (I'm not counting RTL models, which are too expensive to run to use for software testing)
Posted Aug 4, 2012 21:47 UTC (Sat) by mikov (subscriber, #33179)
- Should I be testing my distro's kernel or the latest mainline? If the latter, why am I testing a kernel I am not going to use for years, if at all?
- where do I even report the bugs? Lkml? Bugzilla? My distro?
In reality, it would be a full time job for a couple of people to deal with all this. Highly paid jobs. Not every business can afford that.
A stable source level API would go a long way towards improving the situation, but that is not likely to happen :-)
Posted Aug 4, 2012 22:11 UTC (Sat) by dlang (✭ supporter ✭, #313)
you should test the mainline kernel because your distro is going to be based off of the mainline. It's because your distro is based off the mainline that you should test it.
You don't have to test every -rc kernel, but the more testing that you do, the less likely you are to run into problems when you upgrade to the latest release of your distro.
If you test only when your distro upgrades their kernel every 5 years, then finding where in the 5 years of development things went wrong is an impossible task
If you test the released kernels every 3 months, there's only 3 months of work to go through.
If you test each kernel release around the -rc3-5 range, you are even better off as the developers are currently thinking about that release, and looking for problems to fix.
> - where do I even report the bugs? Lkml? Bugzilla? My distro?
That's easy, if you are testing a mainline kernel, LKML is the best place to report bugs.
Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds