LWN: Comments on "Toward better kernel releases"

i would like the point releases plus pre/rc clarification

alexs — Thu, 16 Dec 2004 09:16:11 +0000

let people do point releases so that the kernel users are served
until the next kernel tree has undergone a reasonable testing time.

point releases should be fixes for anything critical,
like security leaks, kernel oopses and general disabilities
of prior working components. for me when i am a user i want
to have it going out of the box with no risk because i am
not able to track down any single needed patch from lklm
which is known to have a mailing rate of 100/day.

other than that, Linux and his core crew should give a bit
more information to the outside world when the kernel is
open for feature addition and when it is only open for fixing
some period prior to changing the state because that will
help massively in getting into the user testing cycle
when needed and getting out of it when large scale testing
makes no sense for production systems but only for systems
that are test systems that were built for getting broken!

-Alex.

Test against virtual hardware?

alexs — Thu, 16 Dec 2004 09:07:11 +0000

just one word: FAUmachine

and some eplanation:
that is a usermode machine emulator
which is able to emulate misc hardware
starting from IDE devices (on top of the bus interface protocoll)
and ending with sound system emulations.
the whole thing has an optional VHDL frontend(!)
that allows rather in deep autmation
but it is useable as a pc emulator like VMware as well.

lets see what is possible in the long term.

Distributed Kernel Regression Testing

jabby — Fri, 10 Dec 2004 18:13:21 +0000

Why not do what the internet and the community do best? If OSDL developed a platform for doing distributed kernel testing, I'll bet a bunch of people would sign up for it.

If I understand correctly, one can safely run a new kernel on top of a known stable kernel with User-Mode Linux (UML). I believe the test kernel still has access to the hardware via the host kernel. Since the UML-ized kernel is just a process under the host kernel, a fatal error will not bring down the whole system. Perhaps the host environment could even monitor the UML process and do some sort of base-level reporting when something goes wrong.

This just needs to be packaged up in a nice way so that the willing can join a network of volunteer kernel testers. Put the UML folks in touch with the OSDL folks and I think we might have a solution.

Jason

Test against virtual hardware?

khim — Thu, 09 Dec 2004 22:43:59 +0000

The fact is: the first silicon is almost never sold. So there are a lot of stuff you just can not test with emulation. And kernel bugs tend to be so subtle I doubt you can do anything with emulation.

Test against virtual hardware?

JoeBuck — Thu, 09 Dec 2004 19:25:39 +0000

Electronics companies treat their simulation/emulation models for their devices as their deepest, darkest secrets; often such models can't even be shared with other employees of the same company without considerable bureaucracy.

Maybe the best thing is to equip a testing laboratory (say OSDL) with a whole lot of oddball hardware, so they can do the tests. This will cost money, but money could be raised (and we might be able to persuade hardware manufacturers to contribute some hardware).

Test against virtual hardware?

AJWM — Thu, 09 Dec 2004 16:52:01 +0000

Hardware-specific code can only be tested in the presence of the hardware in question.

Or against a (damn good) emulation of it. This is, after all, how firmware and software gets developed for hardware that is itself still in the design process.

Yes, it would no doubt be a rather large effort to develop suitable virtual hardware "devices" to be plugged into a virtual machine for testing, if those devices have to mirror exactly the idiosyncracies of real-world hardware. But it's not impossible, and it's the kind of project that can be approached in a piecemeal and distributed way that's ideal for the bazaar.

Start with one of the existing open virtual systems, make the virtual devices pluggable modules, and then tweak the virtual devices to act like specific real-world hardware rather than some idealized hardware. Once a virtual gizmo is thoroughly tested against its real counterpart in terms of bug-compatible behaviour, you can then run regression tests against that hardware on a virtual machine.

There are undoubtedly classes of bug that this won't catch, but they'll also be of the sort that are less likely to occur in the field anyway.

-- Alastair

Hardware driver testing should be done by manufacturers

hensema — Thu, 09 Dec 2004 10:47:07 +0000

Feature/bugfix releases should not be needed. -preX kernels are feature releases and -rcX kernels are bugfixes. But Linus only releases -rcX kernels so one nevers knows when a kernel is supposed to be stabilizing.

A -rcX kernel should really be an invitation to interested people to give the kernel a test drive. Those interested people might not be that interested in running kernels having new and buggy features.

Hardware driver testing should be done by manufacturers

walles — Thu, 09 Dec 2004 10:14:01 +0000

Regression testing of hardware drivers should be done by the manufacturers of said hardware. But until they start cooperating, I agree with the article that that probably won't happen.