LWN: Comments on "Driver regression testing with roadtest"

Driver regression testing with roadtest

arnout — Mon, 04 Apr 2022 14:16:21 +0000

There was a talk at FOSDEM discussing a completely different approach called EasyMock. Rather then using UML, it mocks away all of the kernel itself. It's not so great for regression testing of changes outside of the driver (because exactly those things are mocked away), but I think it has much more potential for testing the implementation of the driver itself, because it's much easier to mock corner cases and error paths (e.g. failing allocations).

With roadtest, I expect things like error paths for allocations will remain untested because they can't be mocked.

Driver regression testing with roadtest

andy_shev — Fri, 01 Apr 2022 15:31:56 +0000

`assertEqual()` against float, well done!

Driver regression testing with roadtest

qwertyface — Tue, 29 Mar 2022 11:34:22 +0000

There's a really nice property-based testing framework for Python called Hypothesis. I would imagine that in some cases using it to generate test-cases (register values etc.) would work quite nicely within Roadtest.

Driver regression testing with roadtest

riteshsarraf — Thu, 24 Mar 2022 15:56:50 +0000

Yes. That is something we've been targeting. Just that this hasn't had the urgency, given that in its current model, it works fine.

Driver regression testing with roadtest

johill — Thu, 24 Mar 2022 12:14:57 +0000

https://patchwork.ozlabs.org/project/linux-um/list/?serie...

Driver regression testing with roadtest

pabs — Thu, 24 Mar 2022 10:51:01 +0000

One other thing I was thinking; that Debian's src:user-mode-linux should be merged into src:linux (like the RT builds are) so it is always up to date with Debian's other Linux kernel packages.

https://kernel-team.pages.debian.net/kernel-handbook/

Driver regression testing with roadtest

riteshsarraf — Thu, 24 Mar 2022 10:37:39 +0000

Thank you for bringing this up. I will try to review the set of patches we're carrying in Debian, and what all can be pushed upstream.

Driver regression testing with roadtest

pabs — Tue, 22 Mar 2022 01:02:48 +0000

Sorry, I'll have to leave that to you. My Linux kernel knowledge is minimal at this point and I'm not the UML maintainer in Debian, just a bystander that looked it at one point.

Driver regression testing with roadtest

johill — Mon, 21 Mar 2022 11:23:43 +0000

Do you want to propose a couple of patches then? :) Should be pretty simple, I'd think, but if not I guess I can also just write them quickly.

Driver regression testing with roadtest

pabs — Sun, 20 Mar 2022 00:42:23 +0000

Kconfig and runtime config values sound good to me. It seems that it wasn't noticed that strrchr fix was already applied because Debian's patch modifies KERNEL_DEFINES while the upstream change was to KBUILD_CFLAGS. I don't understand enough of the Linux build system to know why both exist and which one the strrchr fix should be in though.

Also, I see there is also a DocBook based linux(1) manual page and of course the Kconfig files duplicate the upstream ones:

https://sources.debian.org/src/user-mode-linux/5.16um1/li...
https://sources.debian.org/src/user-mode-linux/5.16um1/co...
https://sources.debian.org/src/user-mode-linux/5.16um1/co...

Driver regression testing with roadtest

johill — Sat, 19 Mar 2022 20:54:07 +0000

I wasn't (and we should probably discuss this on the uml list), but:

x-terminal-emulator - that's just changing a default, not sure it makes that much sense for non-debian, but the best would probably be to make a new Kconfig variable for it, defaulting to the current value xterm
/tmp/uml.ctl - similarly, and likely it should also be made runtime-configurable?
the strrchr fix is just outdated? the bug report for it is from 2008, the fix from 2009, and in 2011 the bug was fixed upstream in commit 2c51a4bc023
WRITE_ZEROES was submitted upstream

Driver regression testing with roadtest

pabs — Sat, 19 Mar 2022 13:42:29 +0000

BTW, are you aware of the patches that Debian applies to UML?

https://sources.debian.org/src/user-mode-linux/sid/debian...
https://tracker.debian.org/pkg/user-mode-linux

Driver regression testing with roadtest

roc — Sat, 19 Mar 2022 10:46:14 +0000

This sounds like a really good approach. Bravo!

Driver regression testing with roadtest

johill — Sat, 19 Mar 2022 08:42:58 +0000

The website is largely irrelevant these days, it was most interesting back in those days when the project was out of tree (though I should say I wasn't involved back then). It should probably be largely removed and replaced by pointers to the in-kernel documentation.

Which, sadly, is also lacking, but especially Anton has been working on it, and I still have a task in my backlog to describe all the stuff they're also using here (time-travel & virtio/vhost-user).

A couple of people also use it all the time (kunit, in wifi we use it for the hostap tests [1], etc.) so it's at least constantly maintained to work for those use cases.

[1] https://w1.fi/cgit/hostap/tree/tests/hwsim/vm

Driver regression testing with roadtest

pabs — Sat, 19 Mar 2022 06:44:48 +0000

Is UML well maintained? Its website at least does not give that impression, as it references Linux 2.6.

Driver regression testing with roadtest

pmulholland — Sat, 19 Mar 2022 00:55:48 +0000

It’s great to see an open framework like this emerge for kernel testing.

I’ve designed and used proprietary frameworks like this for embedded systems testing (bare metal, RTOS and embedded Linux), and the complexity of mocking the hardware can really be chosen to be almost as shallow as the register and interrupt interface, or you can build out an entire virtual device, or something in between.

As an idea of how powerful python based regression testing can be, with some abstraction I’ve emulated I2C, USB, and even PCI devices using a few lines of python, with the kernel booting up and discovering them.