regression or not [LWN.net]

regression or not

Posted Dec 19, 2006 5:20 UTC (Tue) by roelofs (subscriber, #2599)
In reply to: regression or not by giraffedata
Parent article: Another kernel core dump security issue

What about all the test cases that were written for new features in past releases?

I believe those are more commonly referred to as unit tests. They may or may not be run as part of automated regression testing; I've seen it handled both ways. They're often at a far lower (and less interesting) level than "real" regression test cases, which, in my experience, are designed to call attention to ambiguous changes in behavior--"Is that expected/intended from the recent code changes, or did somebody break something while fixing/improving/adding something else?"

To put it another way, the practical distinction (often) is that unit tests are at the component level, while regression tests are at the system level. YMMV, of course.

Greg

(Log in to post comments)

regression or not

Posted Dec 19, 2006 16:59 UTC (Tue) by giraffedata (subscriber, #1954) [Link]

Well, I'll take your word for how "unit test" is commonly used, because I hardly ever see it. But I know that what it's supposed to mean (what it meant when coined and makes logical sense) has nothing to do with what we've been discussing.

The place I've most seen "unit test" used is IBM software development, where it was shamelessly abused to mean "testing done by the coding department, as opposed to the testing department."

Its original meaning, which, like "regression testing," you need only go to an English dictionary to understand, is testing of individual units of development as opposed to the entire product. According to my old software engineering textbook, it typically means testing a single module of code, which you do by building scaffolding for that module and possibly injecting inputs and examining outputs with debugging tools. After the unit tests are all complete, you move on to system testing. I think unit testing is rare these days because it's cheaper to build and run an entire system as your test driver than to construct single-module test drivers.

And thus, unit vs system testing is orthogonal to regression vs progression testing. If you run a test case that calls kmalloc() to make sure the new GFP_FOO kmalloc flag does what it's supposed to, that's a progression unit test. When you rerun that same test after adding GFP_BAR code to make sure you didn't break GFP_FOO, that's a regression unit test. If you just bring up Fedora, with the new kmalloc code, and browse the web for a while and verify memory doesn't leak, that's a system test, either regression or progression.