Testing for kernel performance regressions [LWN.net]

Testing for kernel performance regressions

Posted Aug 3, 2012 22:52 UTC (Fri) by mikov (guest, #33179)
Parent article: Testing for kernel performance regressions

Nobody does regression testing with hardware either. A driver present in the mainline means only that it compiles successfully with that kernel. Nobody has actually tested it. I get scared when I think about that.

to post comments

Testing for kernel performance regressions

Posted Aug 3, 2012 23:32 UTC (Fri) by gregkh (subscriber, #8) [Link] (23 responses)

Then help by testing the hardware you have, that's all any of us can do, right? What else could be expected?

Testing for kernel performance regressions

Posted Aug 4, 2012 8:33 UTC (Sat) by fhuberts (guest, #64683) [Link] (12 responses)

How about crowd-sourcing that driver testing then?
I'd be willing to donate CPU cycles on most of my machines to test the kernel, drivers, etc. if the results of that would be aggregated somewhere...

Testing for kernel performance regressions

Posted Aug 4, 2012 10:11 UTC (Sat) by Cato (guest, #7643) [Link] (2 responses)

CPAN Testers already does this for testing CPAN modules and Perl on a vast range of OSes (and hardware) - http://wiki.cpantesters.org/

Organising this for Linux would be harder given a bootable system is required, but it could be done.

https://lwn.net/Articles/446382/

Testing for kernel performance regressions

Posted Aug 4, 2012 11:49 UTC (Sat) by jnareb (subscriber, #46500) [Link] (1 responses)

> CPAN Testers already does this for testing CPAN modules and Perl on a vast range of OSes (and hardware)

This works so well because by default CPAN client does tests when installing modules, and send those results to CPANtesters. So it is very easy to become CPANtesters contributors.

Perhaps a request at install / upgrade time to perform regression benchmarks of one's system before first run would be a good idea for Linux testers project?

Testing for kernel performance regressions

Posted Aug 4, 2012 15:09 UTC (Sat) by Cato (guest, #7643) [Link]

That's a good idea - there are already some projects that do functional testing of hardware on new systems, what's missing is the feedback into central test results repository. Though it would need to be a distro-independent test approach so it can be used with the latest kernels as well as distros.

Testing for kernel performance regressions

Posted Aug 4, 2012 10:27 UTC (Sat) by siim@p6drad-teel.net (subscriber, #72030) [Link]

i'd also be willing to test on the few machines i have if there was a straightforward way to run the testsuite and to report the results so that they would be visible to interested parties.

Testing for kernel performance regressions

Posted Aug 4, 2012 14:42 UTC (Sat) by gregkh (subscriber, #8) [Link] (5 responses)

> How about crowd-sourcing that driver testing then?

That's exactly what we do, and what we expect when we do the -rc releases.

You are running them to test that nothing breaks on your machine, right?

If not, please do so.

Testing for kernel performance regressions

Posted Aug 4, 2012 21:34 UTC (Sat) by smoogen (subscriber, #97) [Link]

Well usually the machines that are going to be affected are those in production so running random rc kernels aren't being done on that. My guess is something similar to "Tragedy of the Commons"... people don't test until they have to because they have more important things to do (usually what their boss tells them is important). They then expect that the vendors and coders will do it for them (eg that somewhere there is a huge building filled with racks of machines that vendor X uses every day for every change..) and all for free too.

Testing for kernel performance regressions

Posted Aug 6, 2012 10:24 UTC (Mon) by geertj (guest, #4116) [Link] (3 responses)

> > How about crowd-sourcing that driver testing then?
>
> That's exactly what we do, and what we expect when we do the -rc releases.
> You are running them to test that nothing breaks on your machine, right?

Wrong. I am not testing -rc releases, because i have other stuff to do. And i'm not complaining that my hardware doesn't work either, which makes my behavior wholly consistent. Just pre-empting that comment.. :)

There is a lot more that could be done to make this "crowdsourced testing" more effective. Currently it is quite difficult to test out -rc releases. You have to know how to compile a kernel, and how to install and run it in your distribution. Certainly not rocket science, but not easy for the average distro user, which is who you'd need to go after for large-scale outsourced testing.

Just an idea.. What if bootable live test images could be created for -rc releases? Ideally they would need no local storage, but optionally they could use a dedicated partition. The live image could do a whole bunch of tests and send back the results, together with information the hardware the tests ran it. Those could be analyzed for problems. Also the tests could include performance tests.

Every time a new -rc would be released, you'd rebuild the live image, and ask people via G+, Facebook, Twitter, the mailing list, etc, to burn it to a CD and boot their system with it. The CD runs for a few hours overnight, sends back the results, and then says "Thank you, i'm done". I bet you that this could increase your testing base by 10x. Of course you want to be pretty sure that it is safe and put a lot of safeguards in place to make sure it is (which of course doesn't mean you don't need a pretty scary disclaimer before running the CD).

Testing for kernel performance regressions

Posted Aug 6, 2012 11:14 UTC (Mon) by niner (guest, #26151) [Link]

The idea sounds great. But this way cannot do much more than smoke tests. You can test if the kernel boots on the target's hardware. What you cannot really test is if btrfs will work on the user's drbd device which runs on top of LVM which runs on top of some special hardware RAID controller or such interesting setups. It's at least very difficult to test if suspend/resume works or if the user will experience some performance regression in his favourite game. You cannot test if the external display works automagically when the laptop is put into it's docking station.

In short: it would be a step forward but there's still plenty of stuff which users would have to test manually. But of course: the perfect is the enemy of the good.

Testing for kernel performance regressions

Posted Aug 8, 2012 16:33 UTC (Wed) by broonie (subscriber, #7078) [Link]

Lots of the distros do actually have packaged versions of latest -rc or similar kernels available for installation.

Testing for kernel performance regressions

Posted Feb 7, 2013 10:52 UTC (Thu) by rbrito (guest, #66188) [Link]

> There is a lot more that could be done to make this "crowdsourced testing" more effective. Currently it is quite difficult to test out -rc releases. You have to know how to compile a kernel, and how to install and run it in your distribution. Certainly not rocket science, but not easy for the average distro user, which is who you'd need to go after for large-scale outsourced testing.

Having to compile the kernels is a burden indeed, especially for those with weaker machines.

At least for Debian-based disributions it seems that Canonical provides daily compiled kernels, which is cool to have in mind (I only remembered that when I read your comment):

http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/

Testing for kernel performance regressions

Posted Aug 4, 2012 23:11 UTC (Sat) by krakensden (guest, #72039) [Link] (1 responses)

Phoronix (insert 5 minute hate) actually already built this. It's just that nobody who does any work on anything it tests ever looks at the results, or talks to them to try and make it more useful for developers.

It's a little sad.

Testing for kernel performance regressions

Posted Aug 5, 2012 4:43 UTC (Sun) by dirtyepic (guest, #30178) [Link]

Every time I've tried to point out flaws in their compiler benchmarks I've been ignored. I know the signal to noise ratio on their forums is very high but they've continued to make the same mistakes for years now.

Testing for kernel performance regressions

Posted Aug 4, 2012 10:30 UTC (Sat) by man_ls (guest, #15091) [Link] (6 responses)

A technical solution might be to simulate the hardware and test it automatically, perhaps in a special virtual machine. The initial effort would pay off after a few iterations.

Testing for kernel performance regressions

Posted Aug 4, 2012 12:08 UTC (Sat) by robert_s (subscriber, #42402) [Link] (5 responses)

Hardware is generally full of subtle bugs and to properly encapsulate them, you basically need a VHDL model of the hardware (that's what VHDL was originally designed for).

Testing for kernel performance regressions

Posted Aug 4, 2012 15:21 UTC (Sat) by man_ls (guest, #15091) [Link] (3 responses)

In this case, anything is better than nothing. The more esoteric specialized hardware will probably harder to emulate; but just testing the most common hardware interfaces would surely save a lot of time and eventually enable devs to change code that may be currently too brittle to touch.

Some examples: mount a virtualized SATA disk, format it, create a few files, read them back and check their contents. Mount a virtualized USB disk and do the same. Create several filesystems and stress-test them. Check that the commands issued are in correct order. And so on.

This is (obviously) spoken from utter ignorance of kernel internals, just from the point of view of basic software engineering: if it is not tested and verified it is not finished. I am sure kernel devs will know how to implement the idea or ignore it if the effort is not worth it. But for me it would be a fascinating project.

Testing for kernel performance regressions

Posted Aug 4, 2012 19:09 UTC (Sat) by robert_s (subscriber, #42402) [Link] (2 responses)

"but just testing the most common hardware interfaces would surely save a lot of time"

But that's the stuff that gets tested anyway, by people. The trouble _is_ with the esoteric hardware.

Testing for kernel performance regressions

Posted Aug 4, 2012 22:39 UTC (Sat) by man_ls (guest, #15091) [Link] (1 responses)

Yes, so it is wasting the time of those people who have to painstakingly compile -rc kernels, load them and check that everything works. Not to speak about performance regressions in the drivers and filesystems, which I imagine must be hard to find and boring work. While a test suite publicly available might be run privately and also on the public git repos after every push. It is called continuous deployment, we do it at my company and it is fun, challenging stuff!

Testing for kernel performance regressions

Posted Aug 6, 2012 11:17 UTC (Mon) by niner (guest, #26151) [Link]

Testing RC kernels is not that hard. On openSUSE I just added the Kernel_HEAD repo and new kernels are installed with the other package updates. If a kernel breaks my system I can still just boot the old one and report the bug.

Testing for kernel performance regressions

Posted Aug 6, 2012 10:27 UTC (Mon) by njd27 (subscriber, #5770) [Link]

Unfortunately simulating VHDL would in most cases be far too slow to perform a realistic test of the kernel.

I actually work on the Linux driver for a particular family of input devices from one manufacturer. Most of the customers are integrating devices which are targeting older kernel versions rather than mainline, so our main focus is there. But we do track patches that are going into mainline and make sure they are sane, and do some occasional testing.

Device driver testing

Posted Aug 4, 2012 10:33 UTC (Sat) by ajb (subscriber, #9694) [Link]

In principle, the original hardware companies could do more to help with this. It's difficult to automate tests involving physical devices, and lab space is wanted for the current generation of devices under development. But initial device drivers are often developed on 'models' of the real device before it has come back from manufacturing. The hardware company could set up a regression job, downloading the most recent kernel and booting it on a simulator connected to the model of their device.

In practice, I suspect most hardware companies don't have sufficient incentive to do this, but they might for some products. Some teams also don't really have a model, preferring to develop their device using FPGAs. (I'm not counting RTL models, which are too expensive to run to use for software testing)

Testing for kernel performance regressions

Posted Aug 4, 2012 21:47 UTC (Sat) by mikov (guest, #33179) [Link] (1 responses)

There are a couple of problems:
- there is no static set of hardware that I use. You can't buy the same hardware even if you wanted to. So, I would have to be testing all the time. Considering that the kernel changes all the time...

- Should I be testing my distro's kernel or the latest mainline? If the latter, why am I testing a kernel I am not going to use for years, if at all?

- where do I even report the bugs? Lkml? Bugzilla? My distro?

In reality, it would be a full time job for a couple of people to deal with all this. Highly paid jobs. Not every business can afford that.

A stable source level API would go a long way towards improving the situation, but that is not likely to happen :-)

Testing for kernel performance regressions

Posted Aug 4, 2012 22:11 UTC (Sat) by dlang (guest, #313) [Link]

> - Should I be testing my distro's kernel or the latest mainline? If the latter, why am I testing a kernel I am not going to use for years, if at all?

you should test the mainline kernel because your distro is going to be based off of the mainline. It's because your distro is based off the mainline that you should test it.

You don't have to test every -rc kernel, but the more testing that you do, the less likely you are to run into problems when you upgrade to the latest release of your distro.

If you test only when your distro upgrades their kernel every 5 years, then finding where in the 5 years of development things went wrong is an impossible task

If you test the released kernels every 3 months, there's only 3 months of work to go through.

If you test each kernel release around the -rc3-5 range, you are even better off as the developers are currently thinking about that release, and looking for problems to fix.

> - where do I even report the bugs? Lkml? Bugzilla? My distro?

That's easy, if you are testing a mainline kernel, LKML is the best place to report bugs.