Read the followup by pagexec [LWN.net]

Read the followup by pagexec

Posted Jun 11, 2011 12:10 UTC (Sat) by spender (guest, #23067)
In reply to: Read the followup by pagexec by mingo
Parent article: Quotes of the week

The same old, tired, unrealistic argument. It sounded familiar to me, and my response still has yet to be me with a realistic solution. Here's the thread from over a year ago:
http://lwn.net/Articles/374224/

"Bugs are bugs" yet virtually everyone but the kernel developers themselves are using either a distro kernel or a -stable kernel that both involve selective backports of bugfixes. -stable even often lags months behind (or misses completely) some security backports. I know this to be fact because I backport these myself and then see months later when -stable tries to apply the very same patch, or I see that my patch still remains months later.

Are you holding on to some belief that -stable can be perfected? Do you realize that every time a CVE gets posted by Eugene or Kees or anyone else on oss-sec for a silent upstream fix it undoes your hard obfuscation work and that you're essentially just creating more work for everyone?

Even the people on this site who have been lured into the smooth talk of the "bugs are bugs" tautology I'm sure aren't running the latest -git kernel on their production systems, so I still don't understand why this stillborn idea is treated as some pinnacle of intellectual achievement.

Backports are inevitable and are here to stay. Stop complaining about the very real separation between vulnerabilities and bugs; no amount of hand-waving (as much as you think it can) can remove that separation. You're not even capable of backporting fixes for all bugs, so the whole cry of "you're missing out on bugfixes!" is just as applicable.

The "solution" supported by Linus and yourself isn't even a solution. You have no solution. Show me the users implementing your ideals on a production server. If you believe they exist, I've got a nice bridge to sell you.

"Bugs are bugs" -- it's succinct, it's clever, and syntactically persuasive. But I'm sorry, it's just ridiculous and has no place in the real world. If you believe otherwise, again, show me these users and these production machines. If you can't, then maybe you should rethink the pointless exercise of active obfuscation.

-Brad

to post comments

Read the followup by pagexec

Posted Jun 11, 2011 16:45 UTC (Sat) by mingo (subscriber, #31122) [Link] (18 responses)

"Bugs are bugs" yet virtually everyone but the kernel developers themselves are using either a distro kernel or a -stable kernel that both involve selective backports of bugfixes.

You are misrepresenting our position - nobody suggested to not do the -stable kernel, why are you claiming that? The -stable kernel is a back-port of all fixes that matter to users: data corruption bugs, boot hangs, crashes, mis-behavior and all sorts of other bugs - regardless of whether they have a known security impact or not.

Our argument and policy is to not treat security fixes differently from other bug fixes, because their identification is so imperfect in practice and because their special handling creates a false sense of security (and due to a host of other reasons - see the various discussions for a complete argument). Instead our argument is that if you want to be secure, stay updated with all bugfixes, not just the CVE list.

We are not alone in that view: see this MIT research paper on the topic.

Here's the abstract of their paper:

In this paper, we question the common practice of as- signing security impact ratings to OS updates. Specifi- cally, we present evidence that ranking updates by their perceived security importance, in order to defer applying some updates, exposes systems to significant risk. We argue that OS vendors and security groups should not focus on security updates to the detriment of other updates, but should instead seek update technologies that make it feasible to distribute updates for all disclosed OS bugs in a timely manner.

Which is part of Linus's and my argument as well.

Your attempt to distort and ridicule our position is disingenuous.

Read the followup by pagexec

Posted Jun 11, 2011 17:54 UTC (Sat) by spender (guest, #23067) [Link] (17 responses)

I'm not distorting at all. You're apparently incapable of reading what I wrote. You claim that -stable is a backport of all fixes (added qualifier here from you that it's only fixes that matter to users). I said -stable was selective backports, and gave evidence for my claim -- that even some security fixes take months to arrive in -stable, or don't appear there at all. So it's clearly *not* all bugfixes, and not even all security-relevant bugfixes.

Are you saying I'm wrong? That the latest -stable (and which -stable would that be? does -longterm count? do they all have *all* the bugfixes?) is being used by all users of Linux?

Are you saying a bug is only a bug if you deem it something that matters to users? I thought you said earlier on LKML that any bug in the kernel could be a kernel vulnerability! This is very troublesome.

If you concede (as you must) that even the very newest -stable users still aren't receiving *all* the bugfixes, then you'll agree that making it easier to spot things that need to be backported will help improve the entire process: like, say, by not intentionally obfuscating the commit message of something you know to be a security vulnerability. I'm even being kind by taking the example most closest to your ideal: if we were talking about the distro kernels i'm guessing at least 90% of people run, your own policy does nothing to change everyone else's policies -- the only effect it can have is to further delay the fixing of the vulnerabilities and bugs the distros have chosen to fix.

It's nice to quote from a paper that demonstrates the detrimental effects of your obfuscation strategy and then use it to reaffirm the use of that strategy. I don't see that it reaffirms your position at all: it only shows that your strategy contributes to the problem. You don't solve the problem of distros taking too long to push out kernel updates. BTW you also forgot to mention that the paper you cited was written entirely by Ksplice employees. Since Ksplice sells a product and service entirely built around this concept of security-through-bug-elimination, their conclusion is not surprising to me at all.

Here comes the real irony: I met some of the ksplice guys some time ago, and they asked me what bugs they should be patching. I said to them: if you can, all of them, because the developers silently patch so many security vulnerabilities that you'll be missing out on a lot of vulnerabilities that will be obvious to an attacker. If you take a look at the list of patches against an example system from http://www.ksplice.com/uptrack/using you'll see how realistic it was for even the very people whose paper you quoted to live up to the ideals you wrongly believed they shared ;)

So the real question is, will you dig your feet in the sand on this untenable position, hoping to solve some nonexistent problem through "a bug is a bug" mantras, or will you join the rest of us in reality to work on solutions to the problems people actually face?

-Brad

Read the followup by pagexec

Posted Jun 11, 2011 18:51 UTC (Sat) by mingo (subscriber, #31122) [Link]

If you concede (as you must) that even the very newest -stable users still aren't receiving *all* the bugfixes, then you'll agree that making it easier to spot things that need to be backported will help improve the entire process:

There's nothing to 'concede', it's obvious that marking bugs for -stable backports is useful. You are banging on open doors here really, we are already using such a method to help bug-fix backports and have been using it for years: the "Cc: stable" tag in commits.

We just refuse to play the CVE attention-seeking, self-promotion, reality-distorting security circus you are pushing for, for all the reasons outlined.

Read the followup by pagexec

Posted Jun 11, 2011 20:54 UTC (Sat) by Julie (guest, #66693) [Link] (15 responses)

the problems people actually face

If by 'people' you mean users, I'm pretty sure that users in general can get just as upset by a bug that results in sporadically hanging their box, stopping it from ever booting, corrupting their filesystem, breaking essential drivers or something else that results in an unuseable machine as by a bug that results in, or _might_ result in a successful attack :-)
I do accept that most bugs in the wild being fixed on a daily basis have a far less dramatic impact than the above mentioned but still, all things considered, 'a bug is a bug' works for me.

Read the followup by pagexec

Posted Jun 11, 2011 21:40 UTC (Sat) by spender (guest, #23067) [Link] (14 responses)

That's not the same as 'a bug is a bug' -- you've actually implicitly demonstrated why it's so silly: you've categorized already a subset of all bugs into "serious" bugs, and then further categorized them into boot issues, filesystem corruption, driver problems, general crashes, and potential security issues. So for you, as with everyone, you're concerned about serious bugs, but you don't care what type they are. Well, to determine if a bug is serious, you need to do the very kind of classification 'a bug is a bug' rejects. Do you think upstream avoids mentioning filesystem corruption when they know it to be true? That would be rather unnecessarily malicious on their part, would it not? So why is it acceptable to avoid mentioning security impact when they know it to be true?

So I don't think you really mean 'a bug is a bug' really works for you. Are you using a distro kernel? How often are you updating?

-Brad

Read the followup by pagexec

Posted Jun 11, 2011 21:55 UTC (Sat) by dlang (guest, #313) [Link] (9 responses)

you are misunderstanding the classification they do.

they classify bugs into 'serious' or 'not serious', but no more

what you list as categories ('boot bugs', 'filesystem corruption', ets) are not categories that things are sorted into, they are just examples of types of bugs that fall into the 'serious' category.

Read the followup by pagexec

Posted Jun 11, 2011 22:11 UTC (Sat) by spender (guest, #23067) [Link] (8 responses)

Sorry, you're right. I must have imagined all this:
http://lkml.org/lkml/2008/7/16/17
http://lkml.org/lkml/2008/4/18/154
https://lkml.org/lkml/2011/6/1/294
http://lkml.org/lkml/2009/10/1/590
http://lkml.org/lkml/2002/11/7/137
just from a cursory search.

Come on, this is ridiculous. I'm not going to argue with someone so detached from reality. Prove to me 'a bug is a bug' and show me even one example where the developers have not gone out of their way to say that a particular bug caused filesystem corruption when they knew it to be the case. Just because you say they don't mention a bug fixes booting issues, or filesystem corruption, doesn't make it the case. If your reply doesn't contain a real example as proof, don't bother replying.

-Brad

Read the followup by pagexec

Posted Jun 11, 2011 22:33 UTC (Sat) by mingo (subscriber, #31122) [Link] (7 responses)

Just because you say they don't mention a bug fixes booting issues, or filesystem corruption, doesn't make it the case.

Well, in my 15+ years of kernel maintenance experience, developers are generally very bad at writing changelogs, so yes, even if developers are aware of some serious side-effect, they more often than not do not properly mention it in the changelog.

You mention boot crashes: i'm an arch maintainer so i look after lowlevel code which is literally a magnet for boot crashes, and the majority of boot crash fixes come to me with a change-log that does not mention this fact - often i have to reconstruct it myself.

If you ever were in the position to write features and fix bugs for a big and popular OSS project you'd realize that it's a pretty natural process: developers are happy that they fixed the bug, they describe how they fixed it and what the behaviour now is - then they move on, they don't live in the past trying to figure out what other side-effects a bug might have and they don't try to find ways to break into the system.

They just spent hours or days debugging some bug and want to move on, so writing the changelog is often only an afterthought.

Linux is also very international, so often there's a language barrier that reduces the willingness of developers to write good changelogs and which degrades the quality of changelogs.

There's exceptions, but it's rare.

Come on, this is ridiculous. I'm not going to argue with someone so detached from reality.

I think you are the one detached from reality - you clearly have absolutely no idea how Linux kernel developers think when they write fixes and in this discussion you have demonstrated that you have no idea how the Linux kernel development process works.

To compound your mistake you also insult almost everyone who happens to disagree with you. I hope you behave in a different way with your loved ones.

Read the followup by pagexec

Posted Jun 12, 2011 12:18 UTC (Sun) by nix (subscriber, #2304) [Link] (6 responses)

To compound your mistake you also insult almost everyone who happens to disagree with you.

This is particularly foolish, because it goes directly against his primary goal, which is to get developers writing code with security in mind (well, OK, his *other* primary goal is to get credit come what may, but that is endemic in the security field so can be disregarded).

Unfortunately he is so unpleasant when doing it that virtually everyone who might benefit from his instruction is so turned off by his supercilious and venomous interaction style that they rapidly disregard everything he says, if they don't just killfile him and leave it at that.

I'm not a security person, just someone writing low-level userspace code some of which is network-exposed, i.e. just the sort of person he might be interested in educating, you'd think. When he started posting here I had no idea who he was. Within about fifty posts I'd realised that he obviously knew his stuff, but his interaction style was so unpleasant and so predicated on never admitting error and never giving anyone the benefit of the doubt that by that point I wouldn't believe him if he said the sky was blue. And I don't think I'm particularly unusual.

Even people who can cope with the... robust interactions on the l-k list get turned off by the spender style: the vast number of good developers who don't go near l-k because it's too unpleasant are not even going to consider paying attention to spender, because listening to his security lessons is like wading through acid. Nobody wants their lessons interspersed with commentary on how stupid and foolish they are, and how they are engaging in coverups against the public good (though that last is more PaXTeam's shtick). There's a reason decent teachers don't attack their students.

I have an aunt who communicates in a similar style to spender. Or she would, if she could -- nobody talks to her. Nobody wants to.

Read the followup by pagexec

Posted Jun 12, 2011 12:56 UTC (Sun) by spender (guest, #23067) [Link] (5 responses)

Call it whatever you like, I don't suffer fools gladly. In my view, you're like a child that walks into an advanced mathematics course and wants to talk over the professor the entire time because you foolishly think you have something immensely important to say.

You have a lot of comments about my ability as a teacher; I have some advice for you as well: a wise student knows to do his/her homework and when to be quiet and listen.

BTW while you all were busy spouting nonsense, I just backported a couple security fixes that didn't make it into -stable. In the one case, it made it into .39-stable but was left out of .32-longterm. Oh, and the fixes were all months old. But hey, forget objective evidence, here talking out of your ass about how people are running kernels that have all the bugs fixed is king, so suit yourselves.

Times like these remind me of the cognitive dissonance demonstrated here: http://lwn.net/Articles/290964/ and this post from the PaX team: http://lwn.net/Articles/290968/

See you next year.

-Brad

Read the followup by pagexec

Posted Jun 12, 2011 13:14 UTC (Sun) by mingo (subscriber, #31122) [Link]

BTW while you all were busy spouting nonsense, I just backported a couple security fixes that didn't make it into -stable. In the one case, it made it into .39-stable but was left out of .32-longterm. Oh, and the fixes were all months old. But hey, forget objective evidence, here talking out of your ass about how people are running kernels that have all the bugs fixed is king, so suit yourselves.

You make a quite elementary mistake of logic here: why do you assume that -stable backports are perfect? It's a human process and human processes are never perfect.

Our argument is that adding CVEs to change-logs and treating security bugs differently from other bugs is counter-productive, for all the reasons we outlined. You never addressed that simple argument heads on in this thread - and i submit that you cannot.

Read the followup by pagexec

Posted Jun 12, 2011 13:30 UTC (Sun) by mingo (subscriber, #31122) [Link]

I have a very simple question to people like you who seem to suffer from excessive narcissism: please name three other persons who are smarter and more capable than you, in the field you work in. (In most cases they are utterly unable to answer that question honestly.)

Read the followup by pagexec

Posted Jun 12, 2011 13:55 UTC (Sun) by nix (subscriber, #2304) [Link] (2 responses)

Call it whatever you like, I don't suffer fools gladly. In my view, you're like a child that walks into an advanced mathematics course and wants to talk over the professor the entire time because you foolishly think you have something immensely important to say.

My apologies for daring to post in any thread you have chosen to grace with your mighty presence, sire. I should have known my place, at the bottom forever, and kept silent while the adults were talking, never daring to venture comment, since that's how the free software world works.

(btw, professors really do not act like you do.)

I will shut up now, because this is verging on a flame, even if a justified one.

Read the followup by pagexec

Posted Jun 12, 2011 14:05 UTC (Sun) by nix (subscriber, #2304) [Link] (1 responses)

Apologies to Jon for lowering the tone of the discussion on LWN possibly even more than spender already had. I've filtered spender and PaXTeam out on the grounds that though they may occasionally post worthwhile things, most of their comments have me shaking with anger in seconds, so it's probably bad for my health for me to read any more of them. I will not contribute further to lowering the S/N ratio when they appear.

Because driving students away forever shaking with anger is what a good professor does.

Malicious content detected

Posted Jun 21, 2011 5:07 UTC (Tue) by gabucino (guest, #72504) [Link]

WARNING: the comment above was posted from an insecure operating system, therefore considered potentially harmful.

Read the followup by pagexec

Posted Jun 12, 2011 0:38 UTC (Sun) by Julie (guest, #66693) [Link] (3 responses)

you've categorized already a subset of all bugs into "serious" bugs, and then further categorized them into boot issues, filesystem corruption, driver problems, general crashes, and potential security issues.

Not really, with respect, _you_ just did. I just thought I'd mentioned a varied collection of bug examples (whilst not being altogether serious in my choice of 'serious' ones)- bugs can result in more than one type of offending behaviour, and sometimes might even be discovered/fixed _by accident_ when the bugfixer is fixing up something else.

Which is why I think the most sensible thing is to find bugs regardless of their effect, which to me is what 'a bug is a bug' means - it's a philosophical _starting point_, not an attempt to blur together classifications that may be artificial anyway. Some observed misbehaviour might tell you where to start looking but even then you might be wrong with any assumptions you immediately jump to and it's a good idea to be humble enough to allow the possibility. At best, you avoid a very red face later.
At the simplest level you look at suspicious code or features, find things that shouldn't be there and are or should be there and aren't, and fix it (well, sometimes this turns into development...).

Argue about whether it's a bug by all means and definitely discuss its context and the best way of fixing it in light of the intended purpose and possible side effects of the code it's in, but come to a conclusion and sort it. That does seem to be what the mailing list is for.
And don't spend all day explaining to others exactly how and why you did it, why would you want to do that, there are plenty more to fix? ;-)

Clarity from upstream is great (and I've seen maintainers make contributors go back and re-write changelogs to make them clearer) but nobody that writes code cares for explaining piecemeal to everyone what they did. You'd spend all your time documenting your changes instead of making them.
The poor documentation/lack of helpful comments in code/unhelpful config 'help' that sometimes results from this attitude irritates or inconveniences me a lot more than shoddy changelogs. Users probably won't read the changelogs and reviewers probably ought to look at the code too in any case. But from what I've seen any idea of a deliberate campaign of obfuscation is a fantasy.

So for you, as with everyone, you're concerned about serious bugs, but you don't care what type they are.

Actually, I am concerned about all bugs. I want a perfect kernel :-) It's a quality issue and if Linux can't do it with its talent and development model then no-one can.
It might be a practical impossibility, but this is due to restrictions imposed by a shortage of bug fixers/time. Not to do with aspiration or potential or even method :-)

Are you using a distro kernel? How often are you updating?

I use a distro kernel until I've set up my git kernel tree on the machine and done a hand-stripped config, then I use -next-2011whatever or -rcN or whatever I just built (including patch testing kernels I'm too lazy to reboot from :-)). Having said that I do keep an eye on updates and occasionally use the distro kernels and I've never had problems. I know that means I'm not a typical 'user' and if I was a server admin or someone I might take a different attutude - but I did say it 'works for me'.

Read the followup by pagexec

Posted Jun 12, 2011 9:07 UTC (Sun) by mingo (subscriber, #31122) [Link] (2 responses)

Btw., the primary way we back-port fixes to -stable is to simply consider whether the bug fix is relevant to the -stable tree - i.e. we try to answer the "does this fix a bug in an earlier kernel as well?" question.

That is something that bug fixers are generally capable to determine and they are willing to classify it. It's basically the same task they already did when fixing the bug, just time shifted back 3 or 6 months. It's not perfect but it works reasonably well in practice.

The "could this bug possibly be used to be a parasite in the system" question is a lot less interesting and a lot less natural to the average bug fixer (they are not thinking like parasites) and hence this information is a lot less obvious to extract and it's thus also not part of the regular flow of fixing bugs. It is also a lot more complex mathematically, because the space of potential unintended interactions between kernel bugs and the rest of the kernel and thousands of applications is very, very large. It's hard enough for bug fixers to consider all the intended interactions.

So at this stage we do not pretend to be able to answer that question, and we refuse to participate in the CVE security circus, for all the reasons outlined above.

So yes, the upstream policy is that a bug is a bug and that is applied consistently across the board, to -stable as well.

Read the followup by pagexec

Posted Jun 12, 2011 10:09 UTC (Sun) by dlang (guest, #313) [Link] (1 responses)

I'll note that bug fixers tend to only look back a short while when considering -stable fixes (if you look at the number of changesets in the various long-term and -stable supported kernels, you will see that they tend to get a lot of fixes initially, but that the rate of fixes tapers off fairly rapidly)

Read the followup by pagexec

Posted Jun 12, 2011 12:54 UTC (Sun) by mingo (subscriber, #31122) [Link]

I'll note that bug fixers tend to only look back a short while when considering -stable fixes (if you look at the number of changesets in the various long-term and -stable supported kernels, you will see that they tend to get a lot of fixes initially, but that the rate of fixes tapers off fairly rapidly)

Well, that's an expected curve: current empirical studies show that the distribution / time evolution of software defects goes along a Simon-Yule distribution - which goes down sharper than a bell curve as time goes on.

It's also roughly what you get if you use a simple bug fixing model: take the set of bugs still unfixed, and let bug fixers (developers) random walk the code, then there's a fixed probability to find a bug, with every man-hour spent on looking at the code.

As time goes on, the probability increases that a bug will be found, with a 1-1/t probability, where 't' is the time spent on the code. It will be like a bell-curve for a given difficulty of bug. The 'difficulty' of bugs is a constant parameter, so the combined bug metric would show the integral of 1-d/t ('d': difficulty, 't': time), integrated over 'd' and 't', which would still roughly be a bell curve in practice.

Thus older kernels seeing fewer fixes is natural and expected and is not in itself a sign of less attention.

There's also another aspect: the Cc: stable tag does not typically limit how far a fix is getting ported back - that is decided by the -stable maintainers. They will generally port is back as far as it will still cherry-pick cleanly - and they'll notify the patch submitter if there's a conflict with older stable kernels.

But yes, the older a stable kernel is, the higher the chance that a bug is missed and is not backported - this is a classic trade-off.