Stable kernel 2.6.32.8
Stable kernel 2.6.32.8
Posted Feb 14, 2010 23:59 UTC (Sun) by spender (guest, #23067)In reply to: Stable kernel 2.6.32.8 by hmh
Parent article: Stable kernel 2.6.32.8
I brought it up because if you claim a bug is just a bug, and at the same time say that selective backporting is wrong, then you're ignoring the implications of your views in reality. You expect vendors to inspect (to see if it applies) and backport every single bugfix made to the mainline kernel? *That* is unreasonable. And just to reiterate, you say that "selective backporting" is a bad thing, but -stable is a good thing.
> > The vendors and others that have to do backporting aren't happy with this policy
> Well, I am still waiting for people from the RedHat, SuSE and Debian/Ubuntu kernel security teams to say that.
Eugene Teo's website (member of Red Hat's security team, and the main person responsible for backported kernel security fixes):
http://www.kernel.sg/
"Interesting dailydave/blog post about a kernel vulnerability I spotted that was fixed silently upstream. Unfortunately, this happened too frequently."
http://www.openwall.com/lists/oss-security/2010/01/20/3
> I don't think CVE ids can be assigned to this without more information. I'm
> not knowledgeable enough, nor do I have the time to properly understand
> this list.
Eugene says:
"And upstream continues to give us grief..."
And don't you remember Eugene's quote that caused Linus' famous "security people are leaches [sic]"?:
http://comments.gmane.org/gmane.comp.security.dailydave/3946
On Sun, 19 Jul 2009, Eugene Teo wrote:
>
> If the upstream development community can start doing their part by
> differentiating normal bug fixes to the security ones, I think most of
> us will benefit from it.
Since you forgot, this was Linus' calm and measured response:
>Ok, so this is a perfect example of the kind of IDIOTIC blathering that I
>hate to hear from security people.
>Quite frankly, people who state things like that ARE FUCKING MORONS.
>I'm sorry, but it's true. Learn it. Think about it. Deeply, and long.
>This who security exploit is a prime example of exactly why anybody who
>says something stupid like that is so stupid and so WRONG."
So yes, if you stick your head in the sand and don't listen to anything the vendors have to say, shouting expletives at voiced concerns about how things are done, things would appear to be just fine.
> However, there *is* something I do see happening. People with the skill to really help (e.g. you), aren't doing anything hard to change the status-quo, even if it bothers them enough to "not be happy with this policy".
How can anyone do anything about the problem when it originates from the top down? Linus runs the show, and when he sets his anti-security environment of "a bug is just a bug," it legitimizes it and spreads it to the other kernel developers. I report on all the silently fixed vulnerabilities on my twitter, but most don't get CVEs. I report issues directly to developers and they don't get fixed. I requested a CVE for the SELinux/mmap_min_addr vulnerability and never received a response (Red Hat months later got a CVE for it after I released 8 public exploits).
What changes at all in how Linux kernel development operates if I do any of the things you mention? Do you honestly think anyone will change Linus' stubborn mind? I can publish all day on all the silently fixed vulnerabilities in Linux, but the vendors don't care unless I start publishing exploits for them too. Why would anyone assist the kernel developers for free in the area of security when the developers are actively working against the people trying to improve security? Many of those involved in Linux kernel development (and vendor security teams) get paid to do their work. Isn't this their responsibility? The only way things will improve is from the top down, and it starts with the developers' attitudes towards security. Until then, all Linux users are the ones losing from this stupidity.
If you want to use history as a lesson on how security improves in Linux, let me give you a recap:
security@kernel.org was created some weeks after the PaX team and myself complained (and it made the news, and thus bad publicity) were unable to get any response from Linus and Andrew Morton via email (after several weeks and repeated emails) about some vulnerabilities that were reported.
http://lkml.org/lkml/2005/1/12/138
mmap_min_addr was added after I published a null-ptr-dereference exploit that compromised the machine and disabled SELinux (the bugclass was known for years prior to be exploitable, there were presentations given on it at security conferences, it was published in Phrack, and PaX had already provided protection against it). Many mmap_min_addr bugs were fixed by me and the Google security team (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)
-fno-delete-null-pointer-checks was added after I released another exploit for yet another silently fixed vulnerability (again this one generating bad publicity for Linux)
Red Hat changed the default behavior of SELinux to not bypass mmap_min_addr only after 8 public exploits were released (7 wasn't enough -- only Fedora had changed at that point, but after the 8th public exploit for pipe() which affected a huge number of systems, the bad press of SELinux making exploitation easier made them change their tune).
Minor hardening of copy_from_user (using __builtin_object_size) was only added after my perf_counter() exploit was released.
ASLR added by duplicating the work of the PaX team (NIH syndrome)
exec-shield written by duplicating the work of the PaX team (NIH syndrome)
SSP support in the kernel didn't even work until the PaX team fixed it (again, the common symptom of just throwing in features to look good, but not actually caring if they work or not), this was after it was claimed that SSP would have prevented the vmsplice exploit which 1) it couldn't have as the feature was broke, and 2) SSP wouldn't have even prevented exploitation if it were working
What "hard" things have you done to improve the security of Linux?
-Brad
Posted Feb 15, 2010 13:33 UTC (Mon)
by eparis (guest, #33060)
[Link] (6 responses)
While I agree with the facts (if not your interpretation) of most of your statements this one is a bold faced lie. The "original author" (me) has committed fixes for mmap_min_addr bugs as have developers from the Debian community. You have discovered a number of vulnerabilities in the mmap_min_addr implementation over time although have never, that I can recall, reported any of them to me in any form other than an md5 hash, which was obviously indecipherable. I can't recall any fixes for the code either (I might be wrong), but obviously those are the easy part which I gladly do and have done if you do the hard part of finding the problems in the first place.
Maybe you would like to change this situation? I see you say that you told a 'prominent kernel developer' about other issues, which may also affect the mmap_min_addr implementation. If you tell me, a not so prominent kernel grunt, I'll try to get to the bottom of the situation.
Posted Feb 15, 2010 13:52 UTC (Mon)
by spender (guest, #23067)
[Link] (5 responses)
http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-07/m...
http://lkml.indiana.edu/hypermail/linux/kernel/0711.2/027...
We then even sent you additional emails in private to have you fix additional methods (a.out was one, I can paste the emails if necessary)
http://blog.cr0.org/2009/06/bypassing-linux-null-pointer....
The "Debian community" individual I believe you're talking about is Kees Cook, part of the security team for Ubuntu, not someone I would call a kernel developer. His mmap_min_addr change was to require CAP_SYS_RAWIO to modify the sysctl entry, so that it was consistent with CAP_SYS_RAWIO being required to bypass the limitation.
So remind me again which vulnerability fix for mmap_min_addr was your idea? Maybe you have faulty memory.
-Brad
Posted Feb 15, 2010 15:07 UTC (Mon)
by eparis (guest, #33060)
[Link] (4 responses)
> So remind me again which vulnerability fix for mmap_min_addr was your idea? Maybe you have faulty memory.
I made no claims to having been the first person to find one of your vulnerabilities, usually I have to be aimed to find them. But you seem to be changing your story. I questioned your quote:
> Many mmap_min_addr bugs were fixed by me and the Google security team (because no one else cared if the feature worked or not -- not one fix came from the author or any other kernel developer)
Maybe it's because I separate finding the vulnerability and fixing the problem. In many situations finding the bug is the easy part and fixing it is the hard part. In security usually finding the bug is the hard part and fixing it is the easy part. Google did both. You have not. You have found vulnerabilities and I don't think I've done anything to discredit your ability to find problems, I just question how you can stand by the above quote. Clearly the author "cares" and has worked to fix every vulnerability he has learned about rapidly (even if you are unhappy with my company's reporting at times)
How about reporting those security issues you mentioned telling a 'prominent kernel engineer' to a lowly engineer who will help get to the bottom of it?
Posted Feb 15, 2010 15:56 UTC (Mon)
by spender (guest, #23067)
[Link] (3 responses)
You didn't care to try to find any bugs in the feature you developed. All the bugs were discovered by other people. It's the typical style of non-security people adding security features to the kernel. Particularly when it comes from a company, the main goal is just to be able to say "we have X" and not to care about whether "X" actually works as intended or not. That's the case here.
If you want to be congratulated for the trivial work of writing a couple lines of patches, all for bugs that were either reported to you in private or made public with lots of attention being made to them, you won't find any congratulation from me. It's called doing the job that you're paid to do.
On that note, I don't get paid to do your job, yet I keep reporting vulnerabilities and what I get back is being called a "bold face liar". I'm not sure if Linux leadership/development has caught on to this fact yet, but there is basically no relationship between the security industry and Linux. Linux is viewed as a joke stuck in 1990s security mentality, hence the constant pwnie award nominations and last year's win.
If you really want to improve security, you can take your misguided accusations and direct your attention toward upstream development. If the developers want to be treated like reasonable adults by the security community, they need to take an adult approach to security (instead of calling security people "fucking morons", "masturbating monkeys", "bold face liars", "leaches", "idiots who do not understand development", "idiots who do not understand basic facts", or whatever else the epithet of the day is). It's certainly not the way you entice anyone to do your (paid) work for free.
In case you haven't figured it out yet, I'm not interested in helping anyone who doesn't want to help themselves (any time I've reported extra vulnerabilities have been when a developer showed some initiative/interest in security, or took the handling/classification of security bugs more seriously). What I know for sure is that upstream isn't interested in taking any initiative, but they're very good at reacting to bad publicity and public exploits (at least until that exploit stops being newsworthy, then suddenly the interest in security disappears). So that's what you'll get: more exploits, more bad publicity -- and you'll continue to re-learn the lesson that preventing entire classes of exploits is more important than pretending the developers will always get things right so you can save a CPU cycle.
As for reporting those security issues to you, at least one of them has a patch out there somewhere for it. In the words of Linus, releasing a patch is disclosure, so it should be easy for you to fix it, right? Just as it's easy for any vendor to backport silent security fixes from mainline. If you have a problem with this style of disclosure, you know who to complain to.
-Brad
Posted Feb 15, 2010 16:38 UTC (Mon)
by eparis (guest, #33060)
[Link] (2 responses)
This is quite simply untrue and I thought by now you realized so. Call me a fool. Say that I don't have the time, knowledge, or skill to find the problems that you find and I might not argue. But saying that I don't care is a bold face lie. Claiming that I hadn't found a vulnerability in my code would also not warrant disagreement, but saying that I haven't fixed a single thing is a fabrication.
I don't think pointing out how something you said is untrue should get lumped in with the ridiculous "epithet of the day[s]" you have historically been called.
Posted Feb 15, 2010 17:01 UTC (Mon)
by spender (guest, #23067)
[Link] (1 responses)
You're getting stuck up on my use of the word "fix" when after my first reply I didn't contest that you committed the actual fixes, but the point was that those fixes would have never existed were it not for other people that actually found the vulnerabilities. You had two ways of interpreting it, knowing that 1) neither you nor any other kernel developer discovered a bug that resulted in a fix, and that 2) you committed all but two of the fixes yourself (something you know I'm aware of since you remember our emails).
Replace "fix" with "bug discovery" if you like, with the knowledge that no fixes would exist without previous bug discovery. And I stand behind what I said in general and in this specific case about not caring about added security features actually working. Fixing reported bugs after the fact doesn't explain it away.
-Brad
-Brad
Posted Feb 17, 2010 20:48 UTC (Wed)
by nix (subscriber, #2304)
[Link]
Posted Feb 16, 2010 11:32 UTC (Tue)
by hmh (subscriber, #3838)
[Link] (3 responses)
>I brought it up because if you claim a bug is just a bug, and at the same time say that selective backporting is wrong, then you're ignoring the implications of your views in reality. You expect vendors to inspect (to see if it applies) and backport every single bugfix made to the mainline kernel? *That* is unreasonable. And just to reiterate, you say that "selective backporting" is a bad thing, but -stable is a good thing.
Oh, I wish vendors would do that, but it is sort of like wishing for world piece: I fully expect it to be outright impossible.
Most developers don't do a good enough job of sending all fixes they could/should to -stable, and that's the only place such an effort would have a good chance of doing a lot of good... and would still not be the same as screening every mainline commit.
I failed to make it blatantly clear that I consider selective backporting from the already too-selectively-backported set that ends up being -stable to be a very bad thing, while selectively backporting from mainline is sort of unavoidable in the short term (in the medium or long term, you can check for regressions and switch to a newer kernel).
>>> The vendors and others that have to do backporting aren't happy with this policy
>> Well, I am still waiting for people from the RedHat, SuSE and Debian/Ubuntu kernel security teams to say that.
>Eugene Teo's website (member of Red Hat's security team...
Thank you for the pointer. Looks like some people in the teams don't like it, but I still wonder why we don't see clear messages from the vendors themselves. Otherwise, RedHat would pressure the people they pay to do kernel work, to tag the security bugs very clearly in the commit logs. Well, maybe they _do_ have that policy, but I personally haven't heard of it.
It would be cool to be proven wrong on this, though.
>> However, there *is* something I do see happening. People with the skill to really help (e.g. you), aren't doing anything hard to change the status-quo, even if it bothers them enough to "not be happy with this policy".
>How can anyone do anything about the problem when it originates from the top down? Linus runs the show,
What do you take us for? If developers tag something as a security fix, that commit log is NOT censored. Not by the subsystem maintainers upstream, and certainly not by Linus.
However, there are two important factors: we (as in "your typical Linux kernel developer") usually either fail to recognize the security concerns of a fix, OR we don't want to go the early-full-disclosure route, so we have to submit the fix to someone instead of just sending it up for merging. In that case, it is the security contacts (be them your immediate upstream, kernel.org's, or a distro's) who are the ones responsible for requesting a CVE number, etc.
I worry a lot more about your claims to report stuff and they remaining unfixed.
I have no idea about what is happening in that area, but I can suggest that you could CC these reports to the kernel security teams of the distros themselves as well. AFAIK, if you do that with a good explanation of why it is exploitable, you are pretty much guaranteed to get a CVE out of it.
Or do you mean reports of far more elaborate stuff that hardens the kernel so that it reduces the damage of bugs instead of fixing specific problems (such as much of what the PaX patch is)? Those are often the stuff that would go through the (hard) peer-review and politics of LKML, and indeed would not be merged through security contacts...
I do get the points you made about NIH, I have seen a lot of that in LKML, and it seldon has far more to do with egos than specific technical reasons. That's sad, but it is also something we see everywhere in the software industry, so it is not that surprising.
>Why would anyone assist the kernel developers for free in the area of security when the developers are actively working against the people trying to improve security?
We are a far more diverse group than you take us for. A lot of developers do kernel work as a hobby in their spare time, for example.
Developer education is also the only thing that has a very good chance to help reduce the amount of crap getting into mainline at the current development pace, since even the screening done by subsystem maintainers can only scale so far.
Posted Feb 16, 2010 13:34 UTC (Tue)
by spender (guest, #23067)
[Link]
Well, I think it's telling that the most direct request for a change in the policy had to occur on a private mailing list. Linus telling security people on the list to "f*ck me, shut up about idiotic things like that already" likely sent the message that any further complaints would be fruitless. We also have to consider that for the security teams, this is their career. They likely won't get a raise for getting upstream to change their ways, but they'll certainly generate unwanted attention toward their company if they upset Linus enough that another round of "masturbating monkeys" makes the news.
I hear all the time (generally in private emails to me) of people who disagree with it (and you'd be surprised by who some of them are) but that disagreement remains just that -- private. Like you, I would also like to know why the vendors aren't setting policies for the developers they employ, or why they aren't making a more concerted effort in changing policy.
> What do you take us for? If developers tag something as a security fix, that commit log is NOT censored. Not by the subsystem maintainers upstream, and certainly not by Linus.
The problem is not censorship of already-created commit messages, the problem occurs at the time of the commit message, as was established last year during much discussion (you can go back and read the LWN, LKML threads for all the evidence and admissions). It was back then in the summer of 2008 that the official policy was declared (in contradiction to the published Documentation/SecurityBugs) to be that of deliberate obfuscation of changelogs involving security fixes. For this most recent amd64 DoS, POC code was sent, the bug was known to be a security bug, and yet any mention of it was omitted from the changelogs. In fact, from the changelogs themselves, there's no indication that anything is being done other than code cleanup. This is just a fact -- Linus admitted he engages in this policy, and it's in fact the official policy: the existence of the patch is considered "disclosure," so according to him nothing actually needs to be said security-wise about it.
If you think nothing needs to be said about it, then you assume everyone that needs that security fix is capable of understanding the code to figure out that it's a security fix. Imagine if Microsoft took the approach of "we fixed it -- we don't need to tell anyone what we fixed" and what kind of backlash that would cause. It's obvious in the Windows world that security people/blackhats are fully capable of diffing binaries before a patch and after a patch to determine what vulnerabilities are fixed (a more difficult task than with Linux where the source exists). Why do people pretend this doesn't happen with Linux?
> Or do you mean reports of far more elaborate stuff that hardens the kernel so that it reduces the damage of bugs instead of fixing specific problems (such as much of what the PaX patch is)? Those are often the stuff that would go through the (hard) peer-review and politics of LKML, and indeed would not be merged through security contacts...
No, I mean direct vulnerability fixes with attached POC code to trigger them.
The reason I don't email lists (like vendor-sec for instance) with vulnerability reports is because I don't know who reads them, and I don't trust the integrity of the lists. I've pasted private emails from both vendor-sec and the security@kernel.org lists over the course of several years, obviously to the displeasure of the people on the lists (as they'd wished the kinds of things that go on in private never see the light of day). So to me, sending vulnerability reports to lists is equivalent to sending it out to everyone before any fix exists (and for the one vulnerability, a fix wasn't trivial), and long before that fix filters down to end-users.
The current situation is just silly as everyone agrees that people doing backporting don't necessarily have the knowledge they need to spot many of the vulnerabilities unless they're marked as such. I've demonstrated that security people (and blackhats) are very good at spotting these silent fixes and developing exploits for them before vendors are even aware of the flaw, and long before vendors issue updated kernels. Most of my exploits were developed within an hour of finding the commit message. If you don't omit the security information you're aware of, nothing changes on the blackhat side because they can already spot developer weasel words when the developers are committing a vulnerability fix. So who is really benefiting from the current policy?
-Brad
Posted Feb 18, 2010 3:30 UTC (Thu)
by eteo (guest, #36711)
[Link] (1 responses)
Some of my quotes Brad pasted in his previous reply are my personal opinions
Posted Feb 19, 2010 8:53 UTC (Fri)
by mjcox@redhat.com (guest, #31775)
[Link]
At Red Hat, if we know something is a security issue in any package, and we fix it, we'll get a CVE for it (if it's missing one) and label it in our updates as well as mention it in public bugs/mailing lists etc.
If something we fix later turns out to have a security relevance we didn't know about at the time, we'll retrospectively add the CVE to the relevant errata.
We definitely encourage clear and complete upstream changelogs that highlight security relevant issues.
Stable kernel 2.6.32.8
Stable kernel 2.6.32.8
SELinux/mmap_min_addr
Fix for a number of vulnerabilities because the initial mmap_min_addr check was in the wrong place completely
personality bypass from Google
http://osdir.com/ml/linux-security-module/2009-11/msg0004...
Stable kernel 2.6.32.8
Stable kernel 2.6.32.8
Stable kernel 2.6.32.8
Stable kernel 2.6.32.8
Stable kernel 2.6.32.8
it's his fault that he didn't fix the bugs you didn't tell him about?
Stable kernel 2.6.32.8
Stable kernel 2.6.32.8
Stable kernel 2.6.32.8
> it, but I still wonder why we don't see clear messages from the vendors
and not my employers. Deliberate obfuscation of security-relevant changelogs
is not the right thing to do.
Thanks, Eugene
Stable kernel 2.6.32.8
> is not the right thing to do.