Automated kernel testing [LWN.net]

If there is one thing that almost all kernel developers agree with, it's that more testing is a good thing - especially if the results are presented in a useful way. Martin Bligh thus got a warm reception when he announced a new kernel testing facility. As he put it:

Currently it builds and boots any mainline, -mjb, -mm kernel within about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx. Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32) PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system.

This is, indeed, a fairly wide range of coverage. The results are presented as a simple table, showing which kernels passed the tests and which did not. When a kernel fails a test, the relevant information is provided (though, often, that information is simply "did not boot," which is not entirely helpful).

These results have been augmented with benchmark results, presented in a handy graphic form. The graph shown on the right, for example, notes that kernbench performance improved significantly around 2.6.6, and has held steady since 2.6.10. The -mm trees, however, perform notably worse than the mainline, and the difference between the two has been growing. The results have already led to some investigation into what is going on; the current suspect is the (36!) scheduler patches currently living in -mm.

Numerous others have worked at testing and benchmarking kernel releases. Martin's work, however, has the advantages of being automated and presenting the results in a reasonable way. With these attributes, this project stands a good chance of helping the developers to produce better kernels in the near future.

Index entries for this article
Kernel	Benchmarking
Kernel	Development tools/Testing
Kernel	Regression testing

Automated kernel testing

Posted Jun 9, 2005 8:44 UTC (Thu) by net_bh (guest, #28735) [Link]

I couldn't find links to the scripts that might allow me to setup such an environment. Any pointers?

Automated kernel testing

Posted Jun 9, 2005 8:51 UTC (Thu) by eskild (guest, #1556) [Link] (1 responses)

Hmmm... 36! = 3.7*10^41. That's an awful lot of patches -- even for -mm.

(Sorry, couldn't resist ;-)

Automated kernel testing

Posted Jun 9, 2005 19:18 UTC (Thu) by iabervon (subscriber, #722) [Link]

You mean I'm not supposed to apply the patches in every possible order? I guess it would be take less time to build if I only applied them once, but it's so hard to choose an order...

Automated kernel testing

Posted Jun 9, 2005 13:05 UTC (Thu) by remijnj (guest, #5838) [Link]

The graph shown on the right, for example, notes that kernbench performance improved significantly around 2.6.6, and has held steady since 2.6.10. It is important to note that the graph displayed here is for a 16 CPU NUMAQ machine. This is not your average linux running computer. It's important nonetheless but not to the average linux audience yet.

Automated kernel testing

Posted Jun 10, 2005 12:26 UTC (Fri) by copsewood (subscriber, #199) [Link]

Automated testing and existence of suitable automated test suites leads to an interesting project concept. Very many Linux users would be willing to donate unused CPU time and some network bandwidth to a project to help give testing feedback to mainstream free-software developers. Patches to Linux kernels or free software applications which are capable of being automatically regression tested within a virtual machine environment could be tested more rapidly in this way. This would exclude device drivers requiring raw unvirtualised hardware access and testing user interface facilities requiring manual/visual feedback, but I suspect could allow for the distributed and automatable testing of most other things. This environment would involve:

a. running an application capable of building, initialising, running and monitoring virtual machines (e.g. user mode Linux) on a participating host.

b. synchronising patch sets and locally building binary kernels and applications to run on these in a VM node.

c. synchronising parts of a test suite which can be automatically run within the VM.

d. monitoring the test results from the VM run, and feeding these results back to the developers.

Having had some contact with running VMs, both in local test mode and as a production server, I think the most difficult part of this would be in creating the test-distribution application and standardising the languages (XML based ??) defining the patch distribution and test definitions. Does anyone know anything about the standardisation of automatable regression test definitions worth reading ?