| |||||||||||||
A survey on kernel qualityA survey on kernel qualityPosted Jul 10, 2006 13:12 UTC (Mon) by arcticwolf (guest, #8341)Parent article: A survey on kernel quality
Strange - I always thought the difference between subscribers and non-subscribers was that the former pay money, nothing else. I must've missed the part where you restricted subscriptions to those who could prove that they're "sufficiently knowledgeable"...
Or maybe I should just resubscribe; I sure could use an instant boost of increased knowledge, not to mention the maturity that would make it less likely that I'd attempt skew the results.
After all, this kind of process is already successfully used by other forums, too, like the lkml, for example, so it must be a good idea to limit the ability to provide feedback to those who pay for the priviledge of being able to do so.
(N.B.: Just in case, I am definitely not opposed to LWN's subscription model in general, even though I cannot afford a subscription anymore these days. I just think that while requiring subscriptions for early access to feature articles, the weekly edition etc. makes sense, the same does not hold true when it comes to a *survey*. Not to mention that concerns about skewed results seem a bit silly when the survey will be available to everyone in the not too distant future, anyway. In any case... I know you're unlikely to change anything as a result of this comment, but consider it some unsolicited feedback from someone who at least cares enough to provide feedback, even if he's just a member of the unwashed non-subscribing Lumpenproletariat masses. :))
(Log in to post comments)
Subscriber-only Posted Jul 10, 2006 13:38 UTC (Mon) by corbet (editor, #1) [Link] Please do resubscribe - but do note that we require new subscribers to pass a kernel hacking test first :)This particular pool of people was chosen (by the relevant kernel folks, not by LWN) in an attempt to get around some of the worst problems with web surveys. Somebody who finds LWN worth paying for is reasonably likely to have at least some experience in the area of interest. But, just as importantly, they are relatively unlikely to attempt to skew the survey for reasons of their own. As soon as you open a web survey to everybody, you ask for a bunch of responses from Hank the Angry Drunken Windows User. Closing the survey to non-subscribers will exclude quite a few people who would otherwise provide good input. (And it will stay closed to them, BTW; the survey as a whole closes before the subscription period ends). That is an unfortunate result. But putting out a wide-open survey risks wiping out what little significance this exercise has.
A survey on kernel quality Posted Jul 10, 2006 15:53 UTC (Mon) by jimmybgood (guest, #26142) [Link] As arcticwolf mentions the suggestion that being a paid subscriber makes one more knowledgeable is nonsense. I do have the money to subscribe and would gladly pay it, were that to be true.
The difference between subscribers and nonsubscribers is largely that subscribers identify professionally with Linux as a career and are doing well enough at it to afford the subscription. I would think this makes them _more_ biased not less. Imagine if you were to ask a group of automotive engineers if automobiles were becoming less fuel efficient or less safe. Of course, they would be concerned about the image of the industry that supports their livelihood and would have a bias to protect it by minimizing problems in their responses.
While the above claim is mere conjecture, the well-recognized psychological phenomena of cognitive consonance would, if such a bias were present, lead the subjects to convince themselves that they were not biased.
So long as Andrew Morton is aware of this bias, though, the bias should be fairly predictable and can be corrected for in a rough manner. In other words, if LWN subscribers claim that the kernel quality is staying stable, it's probably really declining.
Designing surveys that produce meaningful responses is not easy. One poster has already reported that he can't fit his observations into the survey's structure. Are the survey questions going to be released for public perusal or will they remain permanently restricted?
A survey on kernel quality Posted Jul 10, 2006 18:04 UTC (Mon) by Los__D (guest, #15263) [Link] No the difference is that subscribers are willing to support LWN... I'm a student, and have close to _NO_ money, I still cashed out to help LWN...
A survey on kernel quality Posted Jul 13, 2006 14:44 UTC (Thu) by lysse (guest, #3190) [Link] I gave up my widows' mites too; LWN is worth it.
However, since for the most part I regard 2.4 series kernels as those nasty newfangled things I grudgingly have to put up with, I don't see my input to this survey being that useful...
Subscribe because LWN helps the developers (me thinks) Posted Jul 10, 2006 20:03 UTC (Mon) by jstAusr (guest, #27224) [Link] There are lots of reasons to subscribe, if it helps someone who helps the community and has a long history of helping - that seems to be a good reason to me.
jstAusr
A survey on kernel quality Posted Jul 11, 2006 12:27 UTC (Tue) by vblum (guest, #1151) [Link] I agree that, here, excluding the knowledgable non-subscribers is really justified because and only because you close the door on a lot of potential noise. I am sure though that no one will mind if non-subscribers post their valuable experiences in a less formalized way in the comments section here - and all are helped.
Very much off topic: If you have the money to subscribe and would gladly pay it, then be glad that you were not around when Jon Corbet announced the end of LWN due to financial difficulties - or else you might be paying after all.
Honestly, no one on LWN's side ever demanded subscriptions - in fact, the opposite is true, Jon Corbet explicitly ruled them out for many years. It was LWN's readership who asked for a subscription option - and loudly so - when Jon Corbet announced that LWN would be closed due to lack of funding.
It is a good question whether I _have_ the money to subscribe to LWN. I do not pay for any other subscriptions whatsoever. This one I pay for; because I was there when there was no other option. Perhaps subscribing to LWN does make a good filter to find people that definitively care about Linux - at least they care enough to keep a high-quality Linux publication going which would otherwise have ceased to exist, quietly and modestly. Sure, you have lots of false negatives, people who really care but don't subscribe. But then, if you do really care, write up your kernel bugs as a comment, no?
Noise is better than bias Posted Jul 11, 2006 18:31 UTC (Tue) by jimmybgood (guest, #26142) [Link] I'm going to try real hard once more to explain my point, which seems to have been missed.
If you want to find out something with a survey, a large noisy sample is far better than a small biased sample. If Andrew Morton thinks he's going to find out something by limiting his survey to a small sample of "knowledgeable" respondents, he's making a mistake. He works in an environment where expert knowledge is highly desirable and out of habit, he imagines that experts might be able to tell him whether the kernel is getting buggier.
Even studying the kernel itself with objective code analysis tools is not a valid way to answer his questions, because those tools have recently been applied to the kernel. Many of the bugs those tools can detect have already been fixed, so the bias will tend to make the kernel appear to be less buggy than it really is. A survey is a good approach to finding out if the Linux kernel is getting buggier.
I have no proof that LWN subscribers are biased, but it is generally accepted that professional societies _are_ biased. I think LWN functions more as a professional society than as a social group.
If you want a good survey, procure as many unique responses from as wide a sample as you can get.
Noise is better than bias Posted Jul 11, 2006 19:36 UTC (Tue) by nix (subscriber, #2304) [Link] Fine, so figure out a way of preventing a single malicious attacker from poisoning an open survey by means of an auto-submission robot. Avoid penalizing multiple people behind a single proxy, and detect a single malicious attacker routing false requests via a network such as tor or a bunch of compromised hosts (he needn't be *running* said botnet: there are vast numbers of known botted hosts with open proxies running on them; he can use some of those).
Until you've done that, a scheme whereby survey responses are tied to single entities (like the registered-subscriber scheme) is needed.
Bias BS Posted Jul 11, 2006 21:48 UTC (Tue) by s_cargo (guest, #10473) [Link] I think your objections would be valid if Andrew Morton asked to survey kernel developers. He did not. This is where your earlier analogy regarding automotive engineers falls flat. I consider it a completely reasonable assumption that LWN subscribers are "serious" users with no motivation for creating any false sense of kernel quality.And to be blunt, you and I have paid nothing to keep LWN going. If everyone were "freeloading" as we are, there wouldn't be any LWN to conduct a survey in the first place. You should be grateful you only have to wait one week to have access to what subscribers have paid to access. So stop your whining.
Noise is better than bias Posted Jul 13, 2006 14:53 UTC (Thu) by lysse (guest, #3190) [Link] > If you want to find out something with a survey, a large noisy sample is far better than a small biased sample.
Not if the bias actually consists of a useful property you want to capture, not if the large noisy sample results in such a poor signal-to-noise ratio that anything meaningful descends to the level of statistical insignificance, and not if the noise itself is biased.
|
|||||||||||||