LWN: Comments on "Linux wins on security in survey of 6,000+ software developers"

Representative samples: the Holy Grail

Max.Hyre — 2005-04-18T18:20:21+00:00

If I took anything away from my statistics courses, it's that the absolutely hardest part to get right is sampling. (Though figuring the right statistical analysis to use is close behind.)

It's hard because you have to

First, figure what your sample population is: Sysadmins? Developers? CIOs? A mixture of them in various proportions? Can you determine that subset which knows what they're talking about?
Then you have to figure how to generate a random sample of the members of your population---not trivial.
Next, how do you reach that set of the population? Always have Dewey vs. Truman floating in front of your eyes. (They reached their sample population [the electorate] by telephone, heavily biasing it [in 1948] toward the well-off. For gory details, google for `truman dewey poll'.)
Finally, after doing a good job of all of the above, you have to get your sample to respond to you. How many will be on vacation in Lower Slobovia? How many pick up their voicemail, or look at their e-mail, frequently, responding in time to do you any good? How many will downright refuse to have anything to do with you? Discarding these sample points, either by not counting them, or choosing someone else in their stead, puts a real dent in randomness.

So, just as you understand ``surf over here and answer some questions'', or ``dial in to tell whether you prefer Princess Di or Camilla'' polls to be nothing more than a form of entertainment, any poll like BZ Research's has to be taken with many grains of salt.

The whole thing is dubious without clear description of all the above criteria, analyzed by a knowledgeable, disinterested observer. Look at research reports in Science or Nature to see the sort of detail I mean. I'd bet a candy bar that the ``2.5 percentage points'' is nothing more than the number they looked up in a table for a sample size of 6k.

And now, for some entirely-different bias, look no further than the polls on the nightly news. They tend to be self-fulfilling prophecies: ``Well, if everyone feels like that, why should I bother to vote / call my Senator / complain to the Planning & zoning board?'' ``Hmmm, if no one's using Linux, I should hold off.''

I hope I've loosened your faith in polls somewhat. :-/

Clueful?

Zartan — 2005-04-14T21:37:31+00:00

More to the point, it's better to do:

# sync
# sync

typed in by hand, than either of the above. Why? Because the second or two that it takes to type the second sync helps compensate for the "scheduled but not necessarily written" aspect of POSIX sync(). Works on all *nixes.

Clueful?

roelofs — 2005-04-14T19:25:44+00:00

I thought his point was, the second sync has no value. At least I hope that's his point.

Tsk, newbies. ( ;-) )

Kernel-page article from three years ago:

How synchronous should sync be? Andrew Morton has posted a patch fixing a perceived problem with the sync() system call: as long as processes keep generating data, sync() will keep flushing it to disk. The result is that a sync command can take a long time to execute - as in several minutes. Andrew's patch changes sync() to just ensure that all data to be written when the call is made gets out - buffers generated thereafter may not be written immediately.
This patch, of course, changes a fundamental assumption made by many who use sync - that, upon completion, all data has been written to disk. In fact, according to the Single Unix Standard, this behavior is permissible: "The writing, although scheduled, is not necessarily complete upon return from sync()" It is, regardless, not the behavior that many expect.
There's no real consensus on what the proper behavior is. Unless Linus takes the patch, the current sync behavior will remain.

So I'm thinking the second and maybe even third sync has some value. :-)

(And I'm pretty sure I remember a followup, as well, in which further details were presented--for example, that there were already cases in which the "expected behavior" was not actually the real behavior--but I don't remember for sure. Maybe it's just my fevered imagination again...)

Greg

Clueful?

jwb — 2005-04-14T16:29:34+00:00

I thought his point was, the second sync has no value. At least I hope that's his point.

Clueful?

utidjian — 2005-04-14T14:59:21+00:00

I am an adittedly superstitious sysadmin; and I want to know why you would cringe when you see someone type sync; sync ? Would it make you cringe less if they typed sync&& sync ? If so, why?

-DU-...etc...

Clueful?

shane — 2005-04-14T08:44:12+00:00

Most sysadmins and developers I know are superstitious. By this I mean that they don't want to understand the reason why things are, but rather just get them to work.

While this makes some sense from an engineering standpoint, I still cringe when I see people typing "sync; sync".

Linux wins on security in survey of 6,000+ software developers

hppnq — 2005-04-12T19:44:29+00:00

Strange reasoning, that would apply to any exploit ever published.

Judging from my own experience I'd say that most sysadmins -- and, more importantly, management -- I have met haven't got the faintest clue about security anyway. I have seen horrific stuff you wouldn't believe, at big, big companies.

But then, I am in a bad mood today, having had to deal with this very problem all day.

Linux wins on security in survey of 6,000+ software developers

jwb — 2005-04-12T19:17:12+00:00

A developer survey reflects popular opinion versus actual experience. How about a survey of sysadmins? That would be more interesting. Given that every kernel released prior to April 4, 2005 has an exploitable SMP race, I think you'll hear a slightly different opinions. Said opinion may be of the form "Everything is crap!"