Return to the Kernel page

Kernel prepatch 3.9-rc5

LWN.net Weekly Edition for April 4, 2013

LWN.net Weekly Edition for March 28, 2013

A kernel change breaks GlusterFS

PyCon: Evangelizing Python

Printable page

Weekly edition	Kernel	Security	Distributions	Contact Us	Search
Archives	Calendar	Subscribe	Write for LWN	LWN.net FAQ	Sponsors

Automated kernel testing

[Posted June 8, 2005 by corbet]

If there is one thing that almost all kernel developers agree with, it's that more testing is a good thing - especially if the results are presented in a useful way. Martin Bligh thus got a warm reception when he announced a new kernel testing facility. As he put it:

Currently it builds and boots any mainline, -mjb, -mm kernel within about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx. Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32) PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system.

This is, indeed, a fairly wide range of coverage. The results are presented as a simple table, showing which kernels passed the tests and which did not. When a kernel fails a test, the relevant information is provided (though, often, that information is simply "did not boot," which is not entirely helpful).

These results have been augmented with benchmark results, presented in a handy graphic form. The graph shown on the right, for example, notes that kernbench performance improved significantly around 2.6.6, and has held steady since 2.6.10. The -mm trees, however, perform notably worse than the mainline, and the difference between the two has been growing. The results have already led to some investigation into what is going on; the current suspect is the (36!) scheduler patches currently living in -mm.

Numerous others have worked at testing and benchmarking kernel releases. Martin's work, however, has the advantages of being automated and presenting the results in a reasonable way. With these attributes, this project stands a good chance of helping the developers to produce better kernels in the near future.

(Log in to post comments)

Automated kernel testing

Posted Jun 9, 2005 4:34 UTC (Thu) by mbligh (subscriber, #7720) [Link]

Re: "though, often, that information is simply "did not boot," which is not entirely helpful" .... if you click on the did not boot error message, it should take you to a boot log, which normally has the error / oops message, etc. Now if it crashes before console_init(), you're admittedly a bit hosed, unless we rerun with early_printk.

Automated kernel testing

Posted Jun 9, 2005 8:44 UTC (Thu) by net_bh (guest, #28735) [Link]

I couldn't find links to the scripts that might allow me to setup such an environment. Any pointers?

Automated kernel testing

Posted Jun 9, 2005 8:51 UTC (Thu) by eskild (subscriber, #1556) [Link]

Hmmm... 36! = 3.7*10^41. That's an awful lot of patches -- even for -mm.

(Sorry, couldn't resist ;-)

Automated kernel testing

Posted Jun 9, 2005 19:18 UTC (Thu) by iabervon (subscriber, #722) [Link]

You mean I'm not supposed to apply the patches in every possible order? I guess it would be take less time to build if I only applied them once, but it's so hard to choose an order...

Automated kernel testing

Posted Jun 9, 2005 13:05 UTC (Thu) by remijnj (subscriber, #5838) [Link]

The graph shown on the right, for example, notes that kernbench performance improved significantly around 2.6.6, and has held steady since 2.6.10. It is important to note that the graph displayed here is for a 16 CPU NUMAQ machine. This is not your average linux running computer. It's important nonetheless but not to the average linux audience yet.

Automated kernel testing

Posted Jun 10, 2005 12:26 UTC (Fri) by copsewood (subscriber, #199) [Link]

Automated testing and existence of suitable automated test suites leads to an interesting project concept. Very many Linux users would be willing to donate unused CPU time and some network bandwidth to a project to help give testing feedback to mainstream free-software developers. Patches to Linux kernels or free software applications which are capable of being automatically regression tested within a virtual machine environment could be tested more rapidly in this way. This would exclude device drivers requiring raw unvirtualised hardware access and testing user interface facilities requiring manual/visual feedback, but I suspect could allow for the distributed and automatable testing of most other things. This environment would involve:

a. running an application capable of building, initialising, running and monitoring virtual machines (e.g. user mode Linux) on a participating host.

b. synchronising patch sets and locally building binary kernels and applications to run on these in a VM node.

c. synchronising parts of a test suite which can be automatically run within the VM.

d. monitoring the test results from the VM run, and feeding these results back to the developers.

Having had some contact with running VMs, both in local test mode and as a production server, I think the most difficult part of this would be in creating the test-distribution application and standardising the languages (XML based ??) defining the patch distribution and test definitions. Does anyone know anything about the standardisation of automatable regression test definitions worth reading ?