| |||||||||||||
Stable kernel 2.6.32.8Stable kernel 2.6.32.8Posted Feb 10, 2010 23:18 UTC (Wed) by dlang (✭ supporter ✭, #313)In reply to: Stable kernel 2.6.32.8 by bojan Parent article: Stable kernel 2.6.32.8
the kernel developers and stable team have decided not to try and judge which patches are security fixes and which are merely bugfixes.
They believe that if they did so, people would only apply the fixes marked as security, and so would end up being exposed to security problems fixed by other things that weren't marked that way.
in addition, they are tired of being harassed about the issue, so now they make a point not to call out particular fixes as being security fixes. This is human nature at work.
(Log in to post comments)
Stable kernel 2.6.32.8 Posted Feb 10, 2010 23:49 UTC (Wed) by PaXTeam (subscriber, #24616) [Link]
> the kernel developers and stable team have decided not to try and judge
> which patches are security fixes and which are merely bugfixes.
you're wrong. they didn't have that decision to make to begin with because they're simply not qualified to make such judgement calls due to lack of expertise. what you wanted to say is that they decided to not share what others tell them about security bugs. we call that a coverup in other areas of life.
> They believe that if they did so, people would only apply the fixes
and they have presented exactly zero evidence for such a belief. not to mention that it's outright insulting to assume that people would be that dumb.
> in addition, they are tired of being harassed about the issue, so now
they are tired of being held accountable for their treatment of security bugs. something proprietary vendors have also had to learn, mind you but they at least did. their reaction is nothing short of 'punishment' of their own userbase, a rather childish attitude at that.
Stable kernel 2.6.32.8 Posted Feb 11, 2010 2:08 UTC (Thu) by mfedyk (guest, #55303) [Link]
> > They believe that if they did so, people would only apply the fixes
> > marked as security
> and they have presented exactly zero evidence for such a belief. not
There is always someone that dumb (using your term). Look at all of the unpatched windows boxes, or even the uproar over firefox 3.7, err I mean 3.6.37. Also look at the latest hack to get debian stable working for their purpose (I used debian for over 8 years and more than 50% of debian users use debian testing or sid)
Once you come down from your lofty ivory tower, you'll see the reality of the individual that believes they're right, no matter what anyone else thinks.
Stable kernel 2.6.32.8 Posted Feb 11, 2010 10:12 UTC (Thu) by nix (subscriber, #2304) [Link]
Hell, I ran into a Linux box recently at a friend's, on the Internet, running Red Hat 5.0. That's not RH*EL* 5.0 or Fedora 5.0, note: that's Red Hat 5.0. Genuine 1997 vintage 2.0.29-ish kernel and libc5 userspace IIRC, never upgraded. Said friend wasn't even aware it *could* be upgraded. And it was in use as a firewall.
So, no, this sort of thing is not unheard of in the least.
(I dislike automatic upgrades that you can't turn off, but automatic upgrades *by default* seem like a very good idea to me. People who don't know or care about security might be secure-by-default then.)
Stable kernel 2.6.32.8 Posted Feb 12, 2010 0:35 UTC (Fri) by PaXTeam (subscriber, #24616) [Link]
> Look at all of the unpatched windows boxes[...]
it seems you're confused. unlike binary Windows updates, the patches are not useful for end users but rather distro builders, sysadmins and other people maintaining their own kernel (for the better or worse, let's not digress into the costs/benefits of not being on latest -stable). they had better be competent at what they're doing.
Stable kernel 2.6.32.8 Posted Feb 10, 2010 23:54 UTC (Wed) by bojan (subscriber, #14302) [Link]
> the kernel developers and stable team have decided not to try and judge which patches are security fixes and which are merely bugfixes.
This is not true. Read the announcement:
> combined with verifying that a security problem really was fixed and backported properly
Obviously, it was identified that a particular bug/fix was a security problem.
The only question is, if they identified it already, why can't we know what it is?
Stable kernel 2.6.32.8 Posted Feb 12, 2010 16:20 UTC (Fri) by hmh (subscriber, #3838) [Link]
Well, you could have asked Greg, but it should be really obvious if you look at the patches, and I suppose anyone who really wanted to know did just that. And probably found out there is more than one security fix patch in there.
The problem is that people are really bad at logic as a rule, and will infer that:
1. a patch was marked as a security fix
so
3. the others are not security fixes
Which is incorrect. It is also dangerous if they follow with "so, I will apply/backport only the one marked as a security fix".
There is more: while I personally mark my patches as security fixes when I am aware they fix a security problem, I certainly won't claim that a patch of mine that I have NOT marked as a security fix isn't a security fix, as I might not realize I am fixing a security hole.
Stable kernel 2.6.32.8 Posted Feb 12, 2010 21:33 UTC (Fri) by PaXTeam (subscriber, #24616) [Link]
> The problem is that people are really bad at logic as a rule, and will infer that:
have you actually seen evidence of such thinking? among the group targeted by patches (see my post above: http://lwn.net/Articles/374094/)?
> It is also dangerous if they follow with "so, I will apply/backport only the one marked as a security fix".
why is it dangerous? do they somehow *reduce* the security of their systems? sure, they could do better than that (still under the assumption that such dumb people actually exist, which i seriously question) but any improvement in system security is still improvement, whatever way you look at it.
or another perspective: if you follow your own logic, file system corruption bugs should not be announced as such either since you can't assure people that other commits don't fix such corruption as well (as a sidenote, a memory corruption bug that often lies behind a security bug is also a good candidate for file system corruption given how much data/metadata lives in kernel memory). yet such bugs are properly described in the commits. how do you explain that?
Stable kernel 2.6.32.8 Posted Feb 13, 2010 4:24 UTC (Sat) by hmh (subscriber, #3838) [Link]
>> The problem is that people are really bad at logic as a rule, and will infer that:
> have you actually seen evidence of such thinking?
Yes. And not just with kernel bugs. And certainly not just with users.
There are some rather unprepared people taking care of software for others out there. You learn a LOT when your main hobby for the last 10 years was to be a large distro downstream maintainer, and not just from the mistakes of others.
Back to your reply, yes, a partially patched system is usually better than an unpatched system, but that DOES assume the skipped patches were not shadow dependencies of the ones that were not skipped. And I know you do know better than to assume this can't happen. However, that was not the point. I consider "selective backporting" dangerous because I DO mean it is often being done on behalf of many others, without proper care.
>or another perspective: if you follow your own logic, file system corruption bugs should not be announced as such either...
Now, don't go making flawed "logic" assumptions. If you're going to use such weak "logic", let me remind you that I could as well just be advocating that "selective patching" that gives higher priority to security bugs is a very bad practice and should be severely frowned upon. Which happens to be the correct answer, but neither it or your "another perspective" can be properly derived (through logic) from what I wrote.
As far as I know, the bias against disclosing security bugs exists due to the security theater circus.
Stable kernel 2.6.32.8 Posted Feb 13, 2010 13:43 UTC (Sat) by spender (subscriber, #23067) [Link]
You are aware that every single distro's kernel, as well as the stable tree, is "selective backporting" of security fixes and other bugfixes.
So you've established that some of these people doing backporting are "unprepared" -- do you think they know that refcount overflows are exploitable? Do you think they'll know they should backport a fix for a remote DoS when the commit message is simply "Enhance frame detection support"? Remember, these are the same people you claimed were too illogical to understand that if some bugfixes are marked as security fixes, that no other security fixes possibly exist.
In other words, how is the current policy of deliberate obfuscation *better* than mentioning that one found a security bug when that was known at the time to be the case? The vendors and others that have to do backporting aren't happy with this policy. What's being done to make their job easier, being as its the result of their work that decides the kernel that is run on 99% of Linux machines? Is it responsible to stick one's head in the sand and claim no responsibility for the problem?
-Brad
Stable kernel 2.6.32.8 Posted Feb 14, 2010 21:06 UTC (Sun) by hmh (subscriber, #3838) [Link]
> You are aware that every single distro's kernel, as well as the stable tree, is "selective backporting" of security fixes and other bugfixes.
Yes. I happen to think that is a bad thing, and I said as much, so I wonder why the heck you brought that up.
>So you've established that some of these people doing backporting are "unprepared" -- do you think they know that refcount overflows are exploitable? ...
Many of the regular kernel developers won't know that (or recall that fact), so I wouldn't expect most of the distro maintainers to, either. That would be the reason why the distros are supposed to have some very security-minded people in their kernel teams.
>In other words, how is the current policy of deliberate obfuscation *better* than mentioning that one found a security bug when that was known at the time to be the case?
I believe the point is that it forces *every* -stable patch to be backported which is a fine goal by itself, at least IMHO.
That won't help with the stuff that never goes to -stable, but those are very likely to be bug fixes where the developer doing the fixing doesn't even realize he fixed a security hole, coupled to said developer either having a weak "let's keep -stable fixed" commitment, or feeling like the fix is too invasive for -stable.
That, of course, means that the case where someone would see a security fix targeted at mainline, realize that older kernels are vulnerable but the maintainer didn't notice that (maybe they're vulnerable in a slightly different way), and alert someone that -stable needs fixing becomes much harder to happen.
> The vendors and others that have to do backporting aren't happy with this policy
Well, I am still waiting for people from the RedHat, SuSE and Debian/Ubuntu kernel security teams to say that.
> What's being done to make their job easier
The -stable tree.
> Is it responsible to stick one's head in the sand and claim no responsibility for the problem?
That's not what I see happening.
However, there *is* something I do see happening. People with the skill to really help (e.g. you), aren't doing anything hard to change the status-quo, even if it bothers them enough to "not be happy with this policy".
Why can't we have an independent report from you or other kernel security specialists, published regularly, with information on the security implications of every patch hitting -stable (-mainline would be too much)? You know you can get early access to all those patches, don't you? Anyone can, they just need to ask to be in the review team. And I am pretty sure you know your report WOULD be read, and acted upon.
Why can't we see a series of articles from you or other kernel security specialist here in LWN, teaching the kernel developers about all the security issues they should know to identify at a glance, with examples of exploits for each class of pattern to get the point home?
It is completely fair if you don't do that because you have better things to do with your time, or because it doesn't suit your (or your employer's) agenda. But it is also completely fair if the kernel developers don't do what you want because they believe their way is better (or because it doesn't suit their, or their employer's agendas, whatever).
Stable kernel 2.6.32.8 Posted Feb 14, 2010 21:56 UTC (Sun) by nix (subscriber, #2304) [Link]
People don't even need to ask to be in the review team to get access
to -stable patches, do they? Just pulling stable-queue on a daily basis will do that. (IIRC. I'm not sure if things land in stable-queue before or after reviewers see them: I thought it was before...)
Stable kernel 2.6.32.8 Posted Feb 15, 2010 15:35 UTC (Mon) by BenHutchings (subscriber, #37955) [Link]
Patches are added to the queue before being sent to reviewers.
Stable kernel 2.6.32.8 Posted Feb 14, 2010 23:59 UTC (Sun) by spender (subscriber, #23067) [Link]
> Yes. I happen to think that is a bad thing, and I said as much, so I wonder why the heck you brought that up.
I brought it up because if you claim a bug is just a bug, and at the same time say that selective backporting is wrong, then you're ignoring the implications of your views in reality. You expect vendors to inspect (to see if it applies) and backport every single bugfix made to the mainline kernel? *That* is unreasonable. And just to reiterate, you say that "selective backporting" is a bad thing, but -stable is a good thing.
> > The vendors and others that have to do backporting aren't happy with this policy
> Well, I am still waiting for people from the RedHat, SuSE and Debian/Ubuntu kernel security teams to say that.
Eugene Teo's website (member of Red Hat's security team, and the main person responsible for backported kernel security fixes):
http://www.openwall.com/lists/oss-security/2010/01/20/3
Eugene says:
And don't you remember Eugene's quote that caused Linus' famous "security people are leaches [sic]"?:
Since you forgot, this was Linus' calm and measured response:
>Quite frankly, people who state things like that ARE FUCKING MORONS.
>I'm sorry, but it's true. Learn it. Think about it. Deeply, and long.
>This who security exploit is a prime example of exactly why anybody who
So yes, if you stick your head in the sand and don't listen to anything the vendors have to say, shouting expletives at voiced concerns about how things are done, things would appear to be just fine.
> However, there *is* something I do see happening. People with the skill to really help (e.g. you), aren't doing anything hard to change the status-quo, even if it bothers them enough to "not be happy with this policy".
How can anyone do anything about the problem when it originates from the top down? Linus runs the show, and when he sets his anti-security environment of "a bug is just a bug," it legitimizes it and spreads it to the other kernel developers. I report on all the silently fixed vulnerabilities on my twitter, but most don't get CVEs. I report issues directly to developers and they don't get fixed. I requested a CVE for the SELinux/mmap_min_addr vulnerability and never received a response (Red Hat months later got a CVE for it after I released 8 public exploits).
What changes at all in how Linux kernel development operates if I do any of the things you mention? Do you honestly think anyone will change Linus' stubborn mind? I can publish all day on all the silently fixed vulnerabilities in Linux, but the vendors don't care unless I start publishing exploits for them too. Why would anyone assist the kernel developers for free in the area of security when the developers are actively working against the people trying to improve security? Many of those involved in Linux kernel development (and vendor security teams) get paid to do their work. Isn't this their responsibility? The only way things will improve is from the top down, and it starts with the developers' attitudes towards security. Until then, all Linux users are the ones losing from this stupidity.
If you want to use history as a lesson on how security improves in Linux, let me give you a recap:
security@kernel.org was created some weeks after the PaX team and myself complained (and it made the news, and thus bad publicity) were unable to get any response from Linus and Andrew Morton via email (after several weeks and repeated emails) about some vulnerabilities that were reported.
mmap_min_addr was added after I published a null-ptr-dereference exploit that compromised the machine and disabled SELinux (the bugclass was known for years prior to be exploitable, there were presentations given on it at security conferences, it was published in Phrack, and PaX had already provided protection against it). Many mmap_min_addr bugs were fixed by me and the Google security team (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)
-fno-delete-null-pointer-checks was added after I released another exploit for yet another silently fixed vulnerability (again this one generating bad publicity for Linux)
Red Hat changed the default behavior of SELinux to not bypass mmap_min_addr only after 8 public exploits were released (7 wasn't enough -- only Fedora had changed at that point, but after the 8th public exploit for pipe() which affected a huge number of systems, the bad press of SELinux making exploitation easier made them change their tune).
Minor hardening of copy_from_user (using __builtin_object_size) was only added after my perf_counter() exploit was released.
ASLR added by duplicating the work of the PaX team (NIH syndrome)
exec-shield written by duplicating the work of the PaX team (NIH syndrome)
SSP support in the kernel didn't even work until the PaX team fixed it (again, the common symptom of just throwing in features to look good, but not actually caring if they work or not), this was after it was claimed that SSP would have prevented the vmsplice exploit which 1) it couldn't have as the feature was broke, and 2) SSP wouldn't have even prevented exploitation if it were working
What "hard" things have you done to improve the security of Linux?
-Brad
Stable kernel 2.6.32.8 Posted Feb 15, 2010 13:33 UTC (Mon) by eparis (subscriber, #33060) [Link]
> Many mmap_min_addr bugs were fixed by me and the Google security team (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)
While I agree with the facts (if not your interpretation) of most of your statements this one is a bold faced lie. The "original author" (me) has committed fixes for mmap_min_addr bugs as have developers from the Debian community. You have discovered a number of vulnerabilities in the mmap_min_addr implementation over time although have never, that I can recall, reported any of them to me in any form other than an md5 hash, which was obviously indecipherable. I can't recall any fixes for the code either (I might be wrong), but obviously those are the easy part which I gladly do and have done if you do the hard part of finding the problems in the first place.
Maybe you would like to change this situation? I see you say that you told a 'prominent kernel developer' about other issues, which may also affect the mmap_min_addr implementation. If you tell me, a not so prominent kernel grunt, I'll try to get to the bottom of the situation.
Stable kernel 2.6.32.8 Posted Feb 15, 2010 13:52 UTC (Mon) by spender (subscriber, #23067) [Link]
Committed the fix, yes obviously. But where did those fixes come from?
http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-07/m...
http://lkml.indiana.edu/hypermail/linux/kernel/0711.2/027...
We then even sent you additional emails in private to have you fix additional methods (a.out was one, I can paste the emails if necessary)
http://blog.cr0.org/2009/06/bypassing-linux-null-pointer....
The "Debian community" individual I believe you're talking about is Kees Cook, part of the security team for Ubuntu, not someone I would call a kernel developer. His mmap_min_addr change was to require CAP_SYS_RAWIO to modify the sysctl entry, so that it was consistent with CAP_SYS_RAWIO being required to bypass the limitation.
So remind me again which vulnerability fix for mmap_min_addr was your idea? Maybe you have faulty memory.
-Brad
Stable kernel 2.6.32.8 Posted Feb 15, 2010 15:07 UTC (Mon) by eparis (subscriber, #33060) [Link]
I think I rightly gave you credit for the harder job of finding a number of vulnerabilities. Although, often I wish that I could better interpret your reporting of issues. "3812e371986ad24ace67bab90fd07ca4" does not count as reporting a flaw, even if it was accurate. For anyone interested he really did send me an e-mail with that in it. I had no idea what it meant. A number of months later I learned it was the md5 hash of a sentence explaining how my code didn't work and was quite shitty. Could have had all that cleared up a lot faster if you just told me.
> So remind me again which vulnerability fix for mmap_min_addr was your idea? Maybe you have faulty memory.
I made no claims to having been the first person to find one of your vulnerabilities, usually I have to be aimed to find them. But you seem to be changing your story. I questioned your quote:
> Many mmap_min_addr bugs were fixed by me and the Google security team (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)
Maybe it's because I separate finding the vulnerability and fixing the problem. In many situations finding the bug is the easy part and fixing it is the hard part. In security usually finding the bug is the hard part and fixing it is the easy part. Google did both. You have not. You have found vulnerabilities and I don't think I've done anything to discredit your ability to find problems, I just question how you can stand by the above quote. Clearly the author "cares" and has worked to fix every vulnerability he has learned about rapidly (even if you are unhappy with my company's reporting at times)
How about reporting those security issues you mentioned telling a 'prominent kernel engineer' to a lowly engineer who will help get to the bottom of it?
Stable kernel 2.6.32.8 Posted Feb 15, 2010 15:56 UTC (Mon) by spender (subscriber, #23067) [Link]
Just to clarify, given your response to my question about which vulnerabilities in your mmap_min_addr were found by you, you discovered none of them.
You didn't care to try to find any bugs in the feature you developed. All the bugs were discovered by other people. It's the typical style of non-security people adding security features to the kernel. Particularly when it comes from a company, the main goal is just to be able to say "we have X" and not to care about whether "X" actually works as intended or not. That's the case here.
If you want to be congratulated for the trivial work of writing a couple lines of patches, all for bugs that were either reported to you in private or made public with lots of attention being made to them, you won't find any congratulation from me. It's called doing the job that you're paid to do.
On that note, I don't get paid to do your job, yet I keep reporting vulnerabilities and what I get back is being called a "bold face liar". I'm not sure if Linux leadership/development has caught on to this fact yet, but there is basically no relationship between the security industry and Linux. Linux is viewed as a joke stuck in 1990s security mentality, hence the constant pwnie award nominations and last year's win.
If you really want to improve security, you can take your misguided accusations and direct your attention toward upstream development. If the developers want to be treated like reasonable adults by the security community, they need to take an adult approach to security (instead of calling security people "fucking morons", "masturbating monkeys", "bold face liars", "leaches", "idiots who do not understand development", "idiots who do not understand basic facts", or whatever else the epithet of the day is). It's certainly not the way you entice anyone to do your (paid) work for free.
In case you haven't figured it out yet, I'm not interested in helping anyone who doesn't want to help themselves (any time I've reported extra vulnerabilities have been when a developer showed some initiative/interest in security, or took the handling/classification of security bugs more seriously). What I know for sure is that upstream isn't interested in taking any initiative, but they're very good at reacting to bad publicity and public exploits (at least until that exploit stops being newsworthy, then suddenly the interest in security disappears). So that's what you'll get: more exploits, more bad publicity -- and you'll continue to re-learn the lesson that preventing entire classes of exploits is more important than pretending the developers will always get things right so you can save a CPU cycle.
As for reporting those security issues to you, at least one of them has a patch out there somewhere for it. In the words of Linus, releasing a patch is disclosure, so it should be easy for you to fix it, right? Just as it's easy for any vendor to backport silent security fixes from mainline. If you have a problem with this style of disclosure, you know who to complain to.
-Brad
Stable kernel 2.6.32.8 Posted Feb 15, 2010 16:38 UTC (Mon) by eparis (subscriber, #33060) [Link]
> (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)
This is quite simply untrue and I thought by now you realized so. Call me a fool. Say that I don't have the time, knowledge, or skill to find the problems that you find and I might not argue. But saying that I don't care is a bold face lie. Claiming that I hadn't found a vulnerability in my code would also not warrant disagreement, but saying that I haven't fixed a single thing is a fabrication.
I don't think pointing out how something you said is untrue should get lumped in with the ridiculous "epithet of the day[s]" you have historically been called.
Stable kernel 2.6.32.8 Posted Feb 15, 2010 17:01 UTC (Mon) by spender (subscriber, #23067) [Link]
It's very simple, as I explained in my previous post. If you cared about it working, you would have at bare minimum found the first couple vulnerabilities yourself (because they were so juvenile -- within 30 minutes after you published the feature I had a second version of my same exploit, with 2 lines changed that worked just as fine as the original). Your claims of caring about it are very weak, as both you and I agree you didn't discover any of the vulnerabilities yourself. As I explained in the previous post, simply fixing things that are reported doesn't mean that you cared about it working correctly, it's just called doing your job.
You're getting stuck up on my use of the word "fix" when after my first reply I didn't contest that you committed the actual fixes, but the point was that those fixes would have never existed were it not for other people that actually found the vulnerabilities. You had two ways of interpreting it, knowing that 1) neither you nor any other kernel developer discovered a bug that resulted in a fix, and that 2) you committed all but two of the fixes yourself (something you know I'm aware of since you remember our emails).
Replace "fix" with "bug discovery" if you like, with the knowledge that no fixes would exist without previous bug discovery. And I stand behind what I said in general and in this specific case about not caring about added security features actually working. Fixing reported bugs after the fact doesn't explain it away.
-Brad
-Brad
Stable kernel 2.6.32.8 Posted Feb 17, 2010 20:48 UTC (Wed) by nix (subscriber, #2304) [Link]
So, you told Eric about bugs by... not telling him about them, and thus
it's his fault that he didn't fix the bugs you didn't tell him about?
Stable kernel 2.6.32.8 Posted Feb 16, 2010 11:32 UTC (Tue) by hmh (subscriber, #3838) [Link]
>> Yes. I happen to think that is a bad thing, and I said as much, so I wonder why the heck you brought that up.
>I brought it up because if you claim a bug is just a bug, and at the same time say that selective backporting is wrong, then you're ignoring the implications of your views in reality. You expect vendors to inspect (to see if it applies) and backport every single bugfix made to the mainline kernel? *That* is unreasonable. And just to reiterate, you say that "selective backporting" is a bad thing, but -stable is a good thing.
Oh, I wish vendors would do that, but it is sort of like wishing for world piece: I fully expect it to be outright impossible.
Most developers don't do a good enough job of sending all fixes they could/should to -stable, and that's the only place such an effort would have a good chance of doing a lot of good... and would still not be the same as screening every mainline commit.
I failed to make it blatantly clear that I consider selective backporting from the already too-selectively-backported set that ends up being -stable to be a very bad thing, while selectively backporting from mainline is sort of unavoidable in the short term (in the medium or long term, you can check for regressions and switch to a newer kernel).
>>> The vendors and others that have to do backporting aren't happy with this policy
>> Well, I am still waiting for people from the RedHat, SuSE and Debian/Ubuntu kernel security teams to say that.
>Eugene Teo's website (member of Red Hat's security team...
Thank you for the pointer. Looks like some people in the teams don't like it, but I still wonder why we don't see clear messages from the vendors themselves. Otherwise, RedHat would pressure the people they pay to do kernel work, to tag the security bugs very clearly in the commit logs. Well, maybe they _do_ have that policy, but I personally haven't heard of it.
It would be cool to be proven wrong on this, though.
>> However, there *is* something I do see happening. People with the skill to really help (e.g. you), aren't doing anything hard to change the status-quo, even if it bothers them enough to "not be happy with this policy".
>How can anyone do anything about the problem when it originates from the top down? Linus runs the show,
What do you take us for? If developers tag something as a security fix, that commit log is NOT censored. Not by the subsystem maintainers upstream, and certainly not by Linus.
However, there are two important factors: we (as in "your typical Linux kernel developer") usually either fail to recognize the security concerns of a fix, OR we don't want to go the early-full-disclosure route, so we have to submit the fix to someone instead of just sending it up for merging. In that case, it is the security contacts (be them your immediate upstream, kernel.org's, or a distro's) who are the ones responsible for requesting a CVE number, etc.
I worry a lot more about your claims to report stuff and they remaining unfixed.
I have no idea about what is happening in that area, but I can suggest that you could CC these reports to the kernel security teams of the distros themselves as well. AFAIK, if you do that with a good explanation of why it is exploitable, you are pretty much guaranteed to get a CVE out of it.
Or do you mean reports of far more elaborate stuff that hardens the kernel so that it reduces the damage of bugs instead of fixing specific problems (such as much of what the PaX patch is)? Those are often the stuff that would go through the (hard) peer-review and politics of LKML, and indeed would not be merged through security contacts...
I do get the points you made about NIH, I have seen a lot of that in LKML, and it seldon has far more to do with egos than specific technical reasons. That's sad, but it is also something we see everywhere in the software industry, so it is not that surprising.
>Why would anyone assist the kernel developers for free in the area of security when the developers are actively working against the people trying to improve security?
We are a far more diverse group than you take us for. A lot of developers do kernel work as a hobby in their spare time, for example.
Developer education is also the only thing that has a very good chance to help reduce the amount of crap getting into mainline at the current development pace, since even the screening done by subsystem maintainers can only scale so far.
Stable kernel 2.6.32.8 Posted Feb 16, 2010 13:34 UTC (Tue) by spender (subscriber, #23067) [Link]
> Thank you for the pointer. Looks like some people in the teams don't like it, but I still wonder why we don't see clear messages from the vendors themselves. Otherwise, RedHat would pressure the people they pay to do kernel work, to tag the security bugs very clearly in the commit logs. Well, maybe they _do_ have that policy, but I personally haven't heard of it.
Well, I think it's telling that the most direct request for a change in the policy had to occur on a private mailing list. Linus telling security people on the list to "f*ck me, shut up about idiotic things like that already" likely sent the message that any further complaints would be fruitless. We also have to consider that for the security teams, this is their career. They likely won't get a raise for getting upstream to change their ways, but they'll certainly generate unwanted attention toward their company if they upset Linus enough that another round of "masturbating monkeys" makes the news.
I hear all the time (generally in private emails to me) of people who disagree with it (and you'd be surprised by who some of them are) but that disagreement remains just that -- private. Like you, I would also like to know why the vendors aren't setting policies for the developers they employ, or why they aren't making a more concerted effort in changing policy.
> What do you take us for? If developers tag something as a security fix, that commit log is NOT censored. Not by the subsystem maintainers upstream, and certainly not by Linus.
The problem is not censorship of already-created commit messages, the problem occurs at the time of the commit message, as was established last year during much discussion (you can go back and read the LWN, LKML threads for all the evidence and admissions). It was back then in the summer of 2008 that the official policy was declared (in contradiction to the published Documentation/SecurityBugs) to be that of deliberate obfuscation of changelogs involving security fixes. For this most recent amd64 DoS, POC code was sent, the bug was known to be a security bug, and yet any mention of it was omitted from the changelogs. In fact, from the changelogs themselves, there's no indication that anything is being done other than code cleanup. This is just a fact -- Linus admitted he engages in this policy, and it's in fact the official policy: the existence of the patch is considered "disclosure," so according to him nothing actually needs to be said security-wise about it.
If you think nothing needs to be said about it, then you assume everyone that needs that security fix is capable of understanding the code to figure out that it's a security fix. Imagine if Microsoft took the approach of "we fixed it -- we don't need to tell anyone what we fixed" and what kind of backlash that would cause. It's obvious in the Windows world that security people/blackhats are fully capable of diffing binaries before a patch and after a patch to determine what vulnerabilities are fixed (a more difficult task than with Linux where the source exists). Why do people pretend this doesn't happen with Linux?
> Or do you mean reports of far more elaborate stuff that hardens the kernel so that it reduces the damage of bugs instead of fixing specific problems (such as much of what the PaX patch is)? Those are often the stuff that would go through the (hard) peer-review and politics of LKML, and indeed would not be merged through security contacts...
No, I mean direct vulnerability fixes with attached POC code to trigger them.
The reason I don't email lists (like vendor-sec for instance) with vulnerability reports is because I don't know who reads them, and I don't trust the integrity of the lists. I've pasted private emails from both vendor-sec and the security@kernel.org lists over the course of several years, obviously to the displeasure of the people on the lists (as they'd wished the kinds of things that go on in private never see the light of day). So to me, sending vulnerability reports to lists is equivalent to sending it out to everyone before any fix exists (and for the one vulnerability, a fix wasn't trivial), and long before that fix filters down to end-users.
The current situation is just silly as everyone agrees that people doing backporting don't necessarily have the knowledge they need to spot many of the vulnerabilities unless they're marked as such. I've demonstrated that security people (and blackhats) are very good at spotting these silent fixes and developing exploits for them before vendors are even aware of the flaw, and long before vendors issue updated kernels. Most of my exploits were developed within an hour of finding the commit message. If you don't omit the security information you're aware of, nothing changes on the blackhat side because they can already spot developer weasel words when the developers are committing a vulnerability fix. So who is really benefiting from the current policy?
-Brad
Stable kernel 2.6.32.8 Posted Feb 18, 2010 3:30 UTC (Thu) by eteo (guest, #36711) [Link]
> Thank you for the pointer. Looks like some people in the teams don't like
> it, but I still wonder why we don't see clear messages from the vendors
Some of my quotes Brad pasted in his previous reply are my personal opinions
Stable kernel 2.6.32.8 Posted Feb 19, 2010 8:53 UTC (Fri) by mjcox@redhat.com (subscriber, #31775) [Link]
> Deliberate obfuscation of security-relevant changelogs
> is not the right thing to do.
At Red Hat, if we know something is a security issue in any package, and we fix it, we'll get a CVE for it (if it's missing one) and label it in our updates as well as mention it in public bugs/mailing lists etc.
If something we fix later turns out to have a security relevance we didn't know about at the time, we'll retrospectively add the CVE to the relevant errata.
We definitely encourage clear and complete upstream changelogs that highlight security relevant issues.
Stable kernel 2.6.32.8 Posted Feb 15, 2010 15:50 UTC (Mon) by PaXTeam (subscriber, #24616) [Link]
>> have you actually seen evidence of such thinking?
> Yes.
is that anecdotal evidence only? just asking because my anecdotal evidence is the exact opposite.
> And not just with kernel bugs.
we are talking about *only* kernel bugs. and that's because other sw developers (learned to) understand the importance of security bugs and they do *not* in general cover them up (quite the contrary in fact).
> And certainly not just with users.
did you mean 'users' in the sense i used it above (i.e., 'patch users', not the grandma kind)? because then it's irrelevant what others think, they get to use what the 'patch users' give them. all the more important to make the life of these 'patch users' easier because any missed security fix will directly affect all their end users.
> There are some rather unprepared people taking care of software for others out there.
then go shame them in public, let the world, and more importantly, their end users know! in case you missed the past decades of the full disclosure debate 'out there', this is exactly what happened to unresponsive and/or uncooperative vendors and look at where we stand today.
> I consider "selective backporting" dangerous because I DO mean it is
so -stable is dangerous as well since it's selective backporting. i don't see why properly reporting known security issues changes any of this danger then, in other words, this is again not a reason for coverup.
> Now, don't go making flawed "logic" assumptions.
i didn't but maybe you misunderstood the 'your logic' reference. it wasn't about the bad implication per se, but what you used it to justify: that people are bad at logic therefore it's somehow dangerous to document security fixes in commits/announcements (look at bojan's question that you replied this to, if it's still not clear). now *this* logic is flawed and i gave you a counterexample where this flawed logic, quite rightly, is not applied and those file system corruption bugs are not covered up - the same should happen with security fixes as well, but for some reason doesn't.
> [...]I could as well just be advocating that "selective patching" that
actually you're wrong in the 'very bad practice' part, only because you happened to use the right term for a change: higher priority. notice how it suddenly doesn't make the whole backporting business black&white (something you and other kernel developers make it appear in their 'arguments'), suddenly it's about priorities and it is very correct (and about the only practical way) to classify fixes and order them somehow given that noone has enough resources to do everything (identify/backport/test/release/support/etc) all at once.
> As far as I know, the bias against disclosing security bugs exists due
now this is a new one and if true, the most childish of all i've heard from kernel developers. what exactly is this 'security theater circus'? and how does it prevent them from disclosing security bugs?
Stable kernel 2.6.32.8 Posted Feb 16, 2010 13:06 UTC (Tue) by hmh (subscriber, #3838) [Link]
>>> have you actually seen evidence of such thinking?
>> Yes.
>is that anecdotal evidence only? just asking because my anecdotal evidence is the exact opposite.
Then we will have to agree to disagree there, as it is indeed anecdotal. It is likely that the groups we have more contact with (and thus are the source of the anecdotal evidences) are different, though.
>> And certainly not just with users.
I did mean distro maintainers and patch users. Note that I haven't seen any evidence of skill shortcomings in the kernel maintainers of any of the larger distros, and said as much.
>> I consider "selective backporting" dangerous because I DO mean it is often being done on behalf of many others, without proper care.
>so -stable is dangerous as well since it's selective backporting.
If you trust it to have all important fixes (which is a superset of "known-to-be-security fixes" and also of "security fixes") backported in the long run? Yes, they are. However, they're the best compromise we have so far, IMO better even than getting stuck to very old distro kernels in many cases, as long as you can do your own regression testing.
>> Now, don't go making flawed "logic" assumptions.
I see. Well, I don't quite agree with that "justification" and I did say as much. But it is one of the main justifications that are put forward when this issue is brought up, and I can certainly see it as a valid reason to prefer to not tag security bugs as such when they are already going to be scheduled to -stable, regardless of whether I'd do it like that or not. And I am quite sure I don't know the entire picture and there are other reasons involved, on both sides.
>suddenly it's about priorities and it is very correct (and about the only practical way) to classify fixes and order them somehow given that noone has enough resources to do everything (identify/backport/test/release/support/etc) all at once
Well, when you see someone asking which fixes are the security ones, because he is only interested in backporting those post-haste, and you look at the commit logs and there are, e.g., RCU or list corruption fixes in there, it gets pretty clear that there are severe drawbacks to the mentality that only security fixes are of critical priority.
I did say I would rather see that issue addressed in other ways than silent security fixes, though.
>> As far as I know, the bias against disclosing security bugs exists due to the security theater circus.
>now this is a new one
No, it isn't. It is an old complain. It refers to two things: the amount of attention and priority that security issues (even non-major ones) are given when compared to non-security issues of similar damage potential, coupled to the importance given to gray-area metrics like "number of security bugs identified" (which is too easy to misuse) and the effects of those by the media.
I better make it clear that I am strongly against downplaying security issues, but I am also against overplaying them (because that, in practice, implies in downplaying data-corruption and system stability issues).
Note that I don't claim metrics are bad. There are metrics that are always useful and hard to misuse, such as "time to deploy fixes", and "fraction of available fixes effectively applied", and obviously, "fraction of issues with fixes available" and "serious regressions introduced by fixes". And even the metrics that are easily misused have their value.
Also note that I don't think security bugs and other classes of critical bugs are exactly the same thing, as by definition the security bugs have one extra component that makes their handling a lot more difficult: the fact that they are, by definition, something that can specifically be targeted and triggered in order to cause damage. Which brings the whole early-disclosure/full-disclosure versus responsible-disclosure (or whatever you want to call these, I apologize if I am not using the correct terms) deal.
|
|||||||||||||