| |||||||||||||
A survey on kernel qualityA survey on kernel qualityPosted Jul 10, 2006 15:53 UTC (Mon) by jimmybgood (guest, #26142)In reply to: A survey on kernel quality by arcticwolf Parent article: A survey on kernel quality
As arcticwolf mentions the suggestion that being a paid subscriber makes one more knowledgeable is nonsense. I do have the money to subscribe and would gladly pay it, were that to be true.
The difference between subscribers and nonsubscribers is largely that subscribers identify professionally with Linux as a career and are doing well enough at it to afford the subscription. I would think this makes them _more_ biased not less. Imagine if you were to ask a group of automotive engineers if automobiles were becoming less fuel efficient or less safe. Of course, they would be concerned about the image of the industry that supports their livelihood and would have a bias to protect it by minimizing problems in their responses.
While the above claim is mere conjecture, the well-recognized psychological phenomena of cognitive consonance would, if such a bias were present, lead the subjects to convince themselves that they were not biased.
So long as Andrew Morton is aware of this bias, though, the bias should be fairly predictable and can be corrected for in a rough manner. In other words, if LWN subscribers claim that the kernel quality is staying stable, it's probably really declining.
Designing surveys that produce meaningful responses is not easy. One poster has already reported that he can't fit his observations into the survey's structure. Are the survey questions going to be released for public perusal or will they remain permanently restricted?
(Log in to post comments)
A survey on kernel quality Posted Jul 10, 2006 18:04 UTC (Mon) by Los__D (guest, #15263) [Link] No the difference is that subscribers are willing to support LWN... I'm a student, and have close to _NO_ money, I still cashed out to help LWN...
A survey on kernel quality Posted Jul 13, 2006 14:44 UTC (Thu) by lysse (guest, #3190) [Link] I gave up my widows' mites too; LWN is worth it.
However, since for the most part I regard 2.4 series kernels as those nasty newfangled things I grudgingly have to put up with, I don't see my input to this survey being that useful...
Subscribe because LWN helps the developers (me thinks) Posted Jul 10, 2006 20:03 UTC (Mon) by jstAusr (guest, #27224) [Link] There are lots of reasons to subscribe, if it helps someone who helps the community and has a long history of helping - that seems to be a good reason to me.
jstAusr
A survey on kernel quality Posted Jul 11, 2006 12:27 UTC (Tue) by vblum (guest, #1151) [Link] I agree that, here, excluding the knowledgable non-subscribers is really justified because and only because you close the door on a lot of potential noise. I am sure though that no one will mind if non-subscribers post their valuable experiences in a less formalized way in the comments section here - and all are helped.
Very much off topic: If you have the money to subscribe and would gladly pay it, then be glad that you were not around when Jon Corbet announced the end of LWN due to financial difficulties - or else you might be paying after all.
Honestly, no one on LWN's side ever demanded subscriptions - in fact, the opposite is true, Jon Corbet explicitly ruled them out for many years. It was LWN's readership who asked for a subscription option - and loudly so - when Jon Corbet announced that LWN would be closed due to lack of funding.
It is a good question whether I _have_ the money to subscribe to LWN. I do not pay for any other subscriptions whatsoever. This one I pay for; because I was there when there was no other option. Perhaps subscribing to LWN does make a good filter to find people that definitively care about Linux - at least they care enough to keep a high-quality Linux publication going which would otherwise have ceased to exist, quietly and modestly. Sure, you have lots of false negatives, people who really care but don't subscribe. But then, if you do really care, write up your kernel bugs as a comment, no?
Noise is better than bias Posted Jul 11, 2006 18:31 UTC (Tue) by jimmybgood (guest, #26142) [Link] I'm going to try real hard once more to explain my point, which seems to have been missed.
If you want to find out something with a survey, a large noisy sample is far better than a small biased sample. If Andrew Morton thinks he's going to find out something by limiting his survey to a small sample of "knowledgeable" respondents, he's making a mistake. He works in an environment where expert knowledge is highly desirable and out of habit, he imagines that experts might be able to tell him whether the kernel is getting buggier.
Even studying the kernel itself with objective code analysis tools is not a valid way to answer his questions, because those tools have recently been applied to the kernel. Many of the bugs those tools can detect have already been fixed, so the bias will tend to make the kernel appear to be less buggy than it really is. A survey is a good approach to finding out if the Linux kernel is getting buggier.
I have no proof that LWN subscribers are biased, but it is generally accepted that professional societies _are_ biased. I think LWN functions more as a professional society than as a social group.
If you want a good survey, procure as many unique responses from as wide a sample as you can get.
Noise is better than bias Posted Jul 11, 2006 19:36 UTC (Tue) by nix (subscriber, #2304) [Link] Fine, so figure out a way of preventing a single malicious attacker from poisoning an open survey by means of an auto-submission robot. Avoid penalizing multiple people behind a single proxy, and detect a single malicious attacker routing false requests via a network such as tor or a bunch of compromised hosts (he needn't be *running* said botnet: there are vast numbers of known botted hosts with open proxies running on them; he can use some of those).
Until you've done that, a scheme whereby survey responses are tied to single entities (like the registered-subscriber scheme) is needed.
Bias BS Posted Jul 11, 2006 21:48 UTC (Tue) by s_cargo (guest, #10473) [Link] I think your objections would be valid if Andrew Morton asked to survey kernel developers. He did not. This is where your earlier analogy regarding automotive engineers falls flat. I consider it a completely reasonable assumption that LWN subscribers are "serious" users with no motivation for creating any false sense of kernel quality.And to be blunt, you and I have paid nothing to keep LWN going. If everyone were "freeloading" as we are, there wouldn't be any LWN to conduct a survey in the first place. You should be grateful you only have to wait one week to have access to what subscribers have paid to access. So stop your whining.
Noise is better than bias Posted Jul 13, 2006 14:53 UTC (Thu) by lysse (guest, #3190) [Link] > If you want to find out something with a survey, a large noisy sample is far better than a small biased sample.
Not if the bias actually consists of a useful property you want to capture, not if the large noisy sample results in such a poor signal-to-noise ratio that anything meaningful descends to the level of statistical insignificance, and not if the noise itself is biased.
|
|||||||||||||