regression or not [LWN.net]

regression or not

Posted Dec 16, 2006 1:11 UTC (Sat) by giraffedata (subscriber, #1954)
Parent article: Another kernel core dump security issue

This particular bug is not technically a regression as there never was a bug that allowed this behavior until it was introduced in the patch.

The technical definition of regression doesn't require a formerly existing bug. A regression is a case where Release N+1 is worse than Release N. From the latin meaning, "to step backward." In most contexts, the definition is narrowed to: Something that works in Release N does not in Release N+1.

Regressions are separated from other bugs because a regression is a reason not to upgrade to N+1, whereas other bugs are not.

In this case, it's hard to cast the bug in any light in which it is not a regression. 2.6.18 is secure from hackers; 2.6.19 is not. A step backward. A reason to regret going to 2.6.19.

I've also heard of code regression, which might be more applicable here. That's where you accidentally overwrite new code with old code, which happens if multiple simultaneous developers aren't careful. A cursory glance it this bug might give you the impression is was caused by a code regression.

(Log in to post comments)

regression or not

Posted Dec 16, 2006 4:02 UTC (Sat) by giraffedata (subscriber, #1954) [Link]

I speak with authority on things of which I am completely certain that I know what I'm talking about. In the unlikely event that someone challenges me, I can then supply better authority or just let the challenge stand as a correction of that authority.

I am in fact surprised to hear that you heard regression and regression testing used that way, which is even weirder (hunting for a bug?) than the way it's used in the article. I guess there's more abuse of the term out there than I knew.

The best authority I can give is the definition of "regression" in an English dictionary. Software engineers don't pull words at random from the dictionary to describe their processes; they use words whose regular usage describes the process. The English "regression" doesn't fit hunting for old bugs or reintroduction of old bugs specifically. But it does describe quite nicely breaking something that used to work.

I also typed "regression testing" into Google and the first two results were dictionaries - Webopedia and Wikipedia - that defined it quite well in a way inconsistent with the definition used in the article and in your comment. The Webopedia one has an error in that in one place it suggests ALL testing of code is regression testing, but the Wikipedia is quite good. Definition of regression is implicit in the definition of regression testing.

My guess is if you looked at the next 50 Google results, at least 45 of them would use regression in a way inconsistent with limiting it to reintroduction of old bugs. I can say this because I've seen the term used in dozens of projects, and it's nearly always the same way, consistent with the plain English "regression."

I can't believe you worked somewhere where regression tests included only tests related to previously reported bugs. What about all the test cases that were written for new features in past releases? Didn't you rerun those too every time you changed code (and call it regression testing)?

Until now, the only abuse of "regression testing" I've seen is that some people think it means automated testing. That's because regression testing is virtually always automated, as a practical necessity, while new function testing often isn't.

regression or not

Posted Dec 19, 2006 5:20 UTC (Tue) by roelofs (guest, #2599) [Link]

What about all the test cases that were written for new features in past releases?

I believe those are more commonly referred to as unit tests. They may or may not be run as part of automated regression testing; I've seen it handled both ways. They're often at a far lower (and less interesting) level than "real" regression test cases, which, in my experience, are designed to call attention to ambiguous changes in behavior--"Is that expected/intended from the recent code changes, or did somebody break something while fixing/improving/adding something else?"

To put it another way, the practical distinction (often) is that unit tests are at the component level, while regression tests are at the system level. YMMV, of course.

Greg

regression or not

Posted Dec 19, 2006 16:59 UTC (Tue) by giraffedata (subscriber, #1954) [Link]

Well, I'll take your word for how "unit test" is commonly used, because I hardly ever see it. But I know that what it's supposed to mean (what it meant when coined and makes logical sense) has nothing to do with what we've been discussing.

The place I've most seen "unit test" used is IBM software development, where it was shamelessly abused to mean "testing done by the coding department, as opposed to the testing department."

Its original meaning, which, like "regression testing," you need only go to an English dictionary to understand, is testing of individual units of development as opposed to the entire product. According to my old software engineering textbook, it typically means testing a single module of code, which you do by building scaffolding for that module and possibly injecting inputs and examining outputs with debugging tools. After the unit tests are all complete, you move on to system testing. I think unit testing is rare these days because it's cheaper to build and run an entire system as your test driver than to construct single-module test drivers.

And thus, unit vs system testing is orthogonal to regression vs progression testing. If you run a test case that calls kmalloc() to make sure the new GFP_FOO kmalloc flag does what it's supposed to, that's a progression unit test. When you rerun that same test after adding GFP_BAR code to make sure you didn't break GFP_FOO, that's a regression unit test. If you just bring up Fedora, with the new kmalloc code, and browse the web for a while and verify memory doesn't leak, that's a system test, either regression or progression.