Kernel release status [LWN.net]

The 2.6.32 merge window is still open as of this writing, so there is no current 2.6 development kernel. The 2.6.32-rc1 release (and the closing of the merge window) can be expected as soon as September 24.

The current stable kernel is 2.6.31. There have been no stable update releases in the last week; a series of stable updates is in the review process, but they have not been released as of this writing.

to post comments

Kernel release status

Posted Sep 24, 2009 2:43 UTC (Thu) by spender (guest, #23067) [Link] (52 responses)

FYI, the 2.6.31 kernel contains a trivially exploitable stack overflow for which my public exploit has existed for the past 2 weeks. The vulnerability is in the new perf_counter code, which is enabled automatically when using any distro's kernel configuration (they set CONFIG_PROFILING=y which causes the new perf_counter config option to be enabled).

The fix to prevent exploitation is here:
http://lkml.org/lkml/2009/9/16/587

There was some broken pointer arithmetic in the code that makes the above vuln trivially exploitable without exploiting a race being required. This was fixed by the patch:
http://lkml.org/lkml/2009/9/19/155

Of course, they called the consequences of the bug a "kernel crash".

I've demonstrated otherwise (and exploited the vulnerability as the restricted user_u SELinux user):
http://www.youtube.com/watch?v=KvREwhfQmbc

The exploit disables SELinux, AppArmor, LSM, auditing, and the recently introduced IMA (uses TPM to ensure integrity of files and modules on the system, and yet this exploit proves it cannot even do that).

"Reducing attack surface" through access control systems (take your pick) can't help when poorly-written and under-reviewed system calls like this one get added to a "stable" kernel.

-Brad

Kernel release status

Posted Sep 24, 2009 3:33 UTC (Thu) by ncm (guest, #165) [Link] (32 responses)

How old are these bugs? Are they in 2.6.30, also?

Kernel release status

Posted Sep 24, 2009 4:18 UTC (Thu) by spender (guest, #23067) [Link] (31 responses)

2.6.31-rc1 to 2.6.31 are vulnerable.

All kinds of confusion result from things not being labeled properly:

http://vigilance.fr/vulnerability/Linux-kernel-privilege-...
says it can be abused for privilege elevation, but the ability of the attacker required to exploit the vulnerability is 4/4 (expert)
(Exploiting it requires 3 lines of code!)

http://secunia.com/advisories/36763/
says it can be "exploited to cause a buffer overflow" -- whatever that's supposed to mean
It also mentions another KVM vulnerability with the wrong classification -- the vulnerability allows for (at minimum) arbitrary code execution on the guest kernel, by way of arbitrary read+write.

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-3234
was only updated once someone saw me post my exploit for the vulnerability
here: http://www.openwall.com/lists/oss-security/2009/09/17/13
Until I posted my exploit, it mentioned only a DoS (which has then been copied to a number of other sites that haven't updated to the new classification), like:
http://www.security-database.com/detail.php?alert=CVE-200...
https://launchpad.net/bugs/cve/2009-3234
http://www.us-cert.gov/cas/bulletins/SB09-264.html

I can't access the NIST site right now, but here's a cache:
http://74.125.47.132/search?q=cache:2DcOTGcDJ6EJ:web.nvd....
"Buffer overflow in the perf_copy_attr function in kernel/perf_counter.c in the Linux kernel 2.6.31-rc1 allows local users to cause a denial of service (crash) via a "big size data" to the perf_counter_open system call."

All mentions of a CVSS for the vulnerability use the original metric from when it was declared a DoS only.

Public exploits shouldn't be required for obvious vulnerabilities to be classified properly.

-Brad

fixed in v2.6.31.1

Posted Sep 24, 2009 17:54 UTC (Thu) by mingo (subscriber, #31122) [Link] (30 responses)

FYI, v2.6.31.1 (released today) has this bug fixed, and upstream -git had this bug fixed a couple of days ago via:

  b3e62e3: perf_counter: Fix buffer overflow in perf_copy_attr()

Also, v2.6.31 with SELinux (NULL pointer mitigation) should not be exploitable. (the local user DoS is still there though, so an upgrade is recommended even if you are using SELinux)

Thanks,

Ingo

fixed in v2.6.31.1

Posted Sep 24, 2009 19:36 UTC (Thu) by spender (guest, #23067) [Link] (3 responses)

Sorry, you're wrong. Apparently you didn't watch the video or look at the exploit code, or bother to look at your own buggy code apparently ;)

It is absolutely exploitable without requiring a NULL mapping. I've proven it with work, not words. Where's your proof?

-Brad

fixed in v2.6.31.1

Posted Sep 24, 2009 19:37 UTC (Thu) by spender (guest, #23067) [Link] (2 responses)

If you saw the exploit you would have recognized even that I named it after you ;)

fixed in v2.6.31.1

Posted Sep 25, 2009 9:04 UTC (Fri) by mingo (subscriber, #31122) [Link] (1 responses)

Btw., that you felt the need to name the exploit after me i found particularly telling (and somewhat amusing): it shows a basic lack of Git knowledge. 'git annotate kernel/perf_counter.c' is not hard to use at all, it shows who wrote what part of the kernel.

FYI, i did not write the code which had the bug and which we fixed, and based on which upstream fix you then distributed a working exploit for. (I reviewed and signed off on it, and i missed the bug, so no doubt i'm part of the chain of review responsibility - this example just further demonstrates exposes your irrational hatred.)

fixed in v2.6.31.1

Posted Sep 27, 2009 10:00 UTC (Sun) by nix (subscriber, #2304) [Link]

I'm not so sure about the 'irrational' part. It's not that you're hateful
in any way: it's that Brad's perspective is necessarily skewed by the work
he does. He spends his whole time looking at the kernel in an adversarial
manner, hunting for faults and ripping them open: of course the parts of
the kernel which work well will fall beneath his notice, and kernel
developers will come to be seen as a contemptible and possibly malevolent
bunch of fault-introducers. I've seen exactly this symptom in some of the
best software testers I've worked with: they find problems really well but
very soon you wish you could hire someone just to sit between them and you
and be a verbal punchbag.

Police officers are prone to the same failing: with shift work and long
hours, after a while all they see of the non-police world is victims and
criminals, and the part they concentrate on is the criminals. Of course
they'll come to see everyone around them as a potential criminal.

fixed in v2.6.31.1

Posted Sep 24, 2009 19:55 UTC (Thu) by spender (guest, #23067) [Link] (25 responses)

As an additional note, even if the zero checking
code did its pointer arithmetic correctly (which it
did not), the vulnerability would *still* be
exploitable without a NULL mapping.

-Brad

fixed in v2.6.31.1

Posted Sep 25, 2009 1:13 UTC (Fri) by busterb (subscriber, #560) [Link] (24 responses)

Brad, why don't you and PaXTeam just release an intermediate .x stable kernel patch series to fix
the problems you find? Sure, criticizing and naming exploits after people you'd like to publicly
shame is fun, but wouldn't fixing problems be even more productive? Just release 2.6.31.1.1 with all
the things you deem important fixed. Might even give you an interesting new perspective on
software development.

fixed in v2.6.31.1

Posted Sep 25, 2009 5:26 UTC (Fri) by Klavs (guest, #10563) [Link] (4 responses)

I def. wouldn't mind having people like you on the team that produces the stable (2.6.x.y) kernels ;)

fixed in v2.6.31.1

Posted Sep 25, 2009 7:10 UTC (Fri) by mingo (subscriber, #31122) [Link] (3 responses)

Agreed, it's a good idea.

Note that the fix for this bug was queued up for 2.6.31.1 before the exploit was posted. (That doesn't of course in any way mitigate the fact that this was an exploitable bug.)

Mr. Spender took the commit log of the fix that others already debugged, created, tested and submitted, and made it easier for script kiddies to vandalize Linux systems by posting an exploit.

As i expressed it earlier, i wish Mr. Spender had more empathy with the project he is trying to contribute to and if he were more mature in not harming it at the same time. Posting an almost-exploit or just saying that it's exploitable is generally enough to get immediate attention - while also avoiding a lot of immediate harm.

Btw., that posting should be immediate and as public as possible, to notify all interested parties [which includes script kiddies and black hats as well] - no matter how embarrassing it might be to the developers and maintainers involved.

Mr. Spender's choice to maximally help script kiddies while trying to maximally harm the people who are actually work on making Linux better (short of him committing a potential felony by launching attacks himself or selling the exploit) certainly qualifies his character - but it's also useful even in this form, no doubt about that.

I'm sure that a parallel or complimentary security effort with real ongoing work injected would be helpful. The sanest approach would be to also notify the -stable maintainers IMHO - the -stable maintainers are very responsive in general.

fixed in v2.6.31.1

Posted Sep 25, 2009 8:52 UTC (Fri) by drag (guest, #31333) [Link] (2 responses)

> (short of him committing a potential felony by launching attacks himself
> or selling the exploit)

For every spender I expect there are a dozen people doing exactly that.
Linux is big business now and big money is to be had by trolling the kernel
commit logs and looking for mislabeled vulnerabilities. I suppose that
there is enough money in it that a person could do pretty much nothing else
but follow along with kernel development and make a living that way.

Personally I have to maintain a handful of different kernels for personal
and professional reasons. It would be nice if I could trust the commit
logs. :)

Although it would be nice if Spender could direct his attention towards
finding exploits in rc kernels! :) Maybe one of the big commercial Linux
guys would hire him or some other group of people to concentrate entirely
on code quality in terms of security and figuring out more and more
automated checks and whatnot. Something can be done, I suppose.

fixed in v2.6.31.1

Posted Sep 25, 2009 9:34 UTC (Fri) by mingo (subscriber, #31122) [Link] (1 responses)

Although it would be nice if Spender could direct his attention towards finding exploits in rc kernels! :) Maybe one of the big commercial Linux guys would hire him or some other group of people to concentrate entirely on code quality in terms of security and figuring out more and more automated checks and whatnot. Something can be done, I suppose.

Indeed, people who find and fix bugs in Linux, with a special attention on security issues, and who try to help people and who are able to work with people are a hot commodity and get picked up by Linux companies quickly.

(Because, not the least, the very same skills and talents can also be used to fix robustness and stability problems and to design better code.)

Alas, Spender is not such a person AFAICT - or if he is, he has not demonstrated such bug finding and communication skills yet. (in a way visible to me at least)

He has not tried to seriously work with the upstream kernel, and in the Git history of Linux, spanning 4 years and over 160,000 changes/fixes, there's not a single commit/fix authored by him. There's not a single commit log entry where he was mentioned as bug reporter or tester.

It really takes a concentrated effort to stay out of the contribution zone to that level, if one is interested in security bugs. (In contrast i found a handful of 'PaX Team' commits.)

Most of the exploits i've seen from him so far were based on other people's work and generally weren't genuine security bugs he himself found. Some of the exploits show creativity (but are not always credited to Mr. Spender himself so it's unclear whether he wrote them).

One has to question the judgement (and wisdom) of someone opting to help attackers with the most ruthless yet borderline legal means, just to raise 15 minutes of attention.

I personally think it's still useful as long as there's still a net benefit to Linux (which there is here) - and the resulting embarrassment to developers that keeps us all sharper is helpful too - but the process of elaborate and very public self-destruction is always sad to observe.

The thing is, finding genuine security bugs in Linux is hard and takes a lot of skill and a lot of talent. Kenrel people and Linux companies will quickly notice if you have that skill.

fixed in v2.6.31.1

Posted Sep 25, 2009 15:03 UTC (Fri) by spender (guest, #23067) [Link]

Ingo, let's be clear about what's happening here. You waltzed into this conversation acting as an expert on the subject, having done no research whatsoever. Not only that, you were shilling for your company's security features without having done any verification that they did what you claimed.

Now, from there, having been so embarrassingly wrong, and not having had the common sense to review the links in my post to which you were replying, one would expect a "I was mistaken" response -- but not from you of course. In typical fashion, what we get instead is a series of ad-hominem attacks. Allow me to enumerate your charges:

You claim that I haven't found or fixed bugs in Linux.
You claim that my work is plagiarized (funny concept coming from you -- exec-shield, CFS, perf counters, etc).
You claim that I'm choosing to help attackers to gain attention for myself.
You claim that I'm engaging in borderline-felonious activity.
You claim that I harmed security by not immediately issuing a release to full-disclosure/dailydave/LKML about the vulnerability.

So let's take these claims one by one.

You claimed that I harmed security by not immediately notifying everyone about the vulnerability. This is a curious claim, because it doesn't look good for you or for Linux kernel development in general. And in fact it negates one of your other claims (that observing an OOPs and submitting a report for a "kernel crash" and writing one line of code is more difficult than actually determining the vulnerability class and exploiting it.)

So here is my question to you. Is it the case that yourself, Paul Mackerras, Xiao Guangrong, Greg Kroah-Hartman, Peter Zijlstra, Eugene Teo and the other members of oss-security *ALL* can not plainly see that trivial, classic stack overflows, like the one signed-off on the fix for, are exploitable for arbitrary code execution? Did you all really believe that it was purely a "kernel crash" as noted in the commit message you all reviewed and signed off on?

If that is the case, I feel sorry for Linux kernel development. Perhaps your respective companies could pay to send you to an entry-level security crash course to learn what gets done with trivial stack overflows.

If that's not the case, and you were aware that the vulnerability was exploitable for arbitrary code execution, why is the onus on me to inform the stable maintainers or the world at large of the vulnerability? Isn't that your job in the commit messages that you write and review? Is it somehow my fault that people are being knowingly mislead (including the security teams of the various vendors, who republished the "kernel crash" claim and had the CVE marked as a denial of service issue)? Is it my fault that nobody in the Linux kernel development community seems to think twice about "buffer overflow" and "kernel crash" being in the same sentence in a commit message? We (in conjunction with the rest of the security industry -- Linus was awarded with a Pwnie this year for being voted the worst at handling security issues) have been arguing for a long time now for you guys to end your silly practice of obfuscation and lies.

Put plainly, you're arguing with the wrong person. We keep demonstrating the harm of these silly commit messages; you'd be better served by growing a spine and launching your complaints at the people who are actually responsible for this current situation. I'd be happily surprised to see that happen, but suspect that it won't.

You claim that I'm engaged in borderline-felonious activity. Do you make the same claims about any other security researcher that posts exploits? About all the people in the security industry writing modules for Metasploit?

Public exploits are very effective in making a point. Here's some of the lessons you should have learned from this (but apparently were asleep during class):
1) An exploit not being made public isn't an excuse to pretend one doesn't exist
2) All of the exploits I've written have been developed and fully-weaponized in under an hour.
3) Given how quickly these can be developed, it's been made clear that the stable maintainers and Linux vendors are ill-equipped to respond quickly enough.
4) Don't rest on the laurels of your ineffective security measures.
5) Obfuscating commit messages doesn't stop someone who wants to write an exploit from figuring out what the issue is.
6) Even your most prized security features can be defeated.

If it wasn't clear to people before, it should be plainly clear now what kind of security "guarantees" can be provided when a framework exists that rewrites the SELinux code to pretend that it's in enforcing mode or completely defeats IMA. Linux was benefiting in appearance only by not having something like this published -- since it's nothing any private exploit writer hadn't written themselves. Now that it is public, the correct response should be "what can we do about this problem?" Fixing bugs isn't the answer (you'll just add more in the next kernel), neither is attacking the person who demonstrated the techniques to you. Did you forget what caused mmap_min_addr to be added to the kernel in the first place?

You know very well that I've found and fixed bugs and vulnerabilities in Linux. Have you forgotten our private conversations about exec-shield vulnerabilities I reported to you that you fixed silently? I still have the emails, if necessary. What about the missing NULL pointer checks issue? The SELinux vulnerability? The exploitability of NULL pointer dereferences in general? How quickly we forget, and what an obvious example of why I would have no interest in cooperating with you in the future. As many can attest (yourself included), I contact them privately about vulnerabilities in their code. I'm not always given credit (like with the SELinux issue). Very few of the things I've reported privately that have been fixed in the kernel I have been given credit for, so it's no surprise that you didn't find anything in your (rather contrived) search (much like your benchmarks, I wager). Given that I don't believe I've ever sent a single message to LKML, I don't imagine I would be listed as the author of any commit. Being listed as a bug reporter is something at the discretion of the committer, so again, no surprise there.

Furthermore, had it occurred to you that I'm not interested in finding and fixing your bugs? For each one I fix, you'll add two more. Finding and patching bugs doesn't improve security. Just like userland security was a cesspool until PaX arrived on the scene (where any memory corruption bug could be exploited by someone who read some simple Phrack articles), kernel security is the same way without any defensive techniques. It's developing those techniques, as I have been doing, that I'm interested in -- preventing exploit vectors, stopping exploitation of entire bug classes, not in wasting time fixing your individual bugs so that you can add more next week. Oh and you guessed it, it's my dream in life to one day work for a Linux vendor. Writing up 30 advisories every month for all the packages that use webkit, I can see it now!

You claim that my exploits are plagiarized, due to the fact that I don't put my name on some of them. Had it occurred to you that maybe I don't care so much about putting my name in it if it's obvious I wrote it? You can follow my postings on twitter where I discuss additions to the framework I developed. If you have questions about the authenticity of the code, those questions should exist whether I put my name in it or not.

You claim that I'm choosing to help attackers just to gain some attention for myself. I don't think posting messages only to my twitter account that only a handful of people read is an attempt to get attention. As I mentioned earlier, if I were to do what you suggested and let the entire world know, you would accuse me of the same attention seeking, so it's impossible for any researcher to escape your misdirected criticism. I know that members of the security teams for several vendors, including that of your own company, follow my twitter and so are immediately notified. In fact, you can see several examples of it on oss-security where they link to me explicitly. The attention I want to bring is to the mislabeling of security vulnerabilities caused by the kernel developers' official policy of obfuscating commit messages. The lies contained in them propagate into the CVE, which then give everyone a false representation of their risk. And you are complicit in this behavior.

So in summary, Ingo, I have a bit of advice for you that will avoid such embarrassment in the future, and then you won't have to waste your time on misguided and misinformed ad-hominem attacks against one of the few people who are actually improving the security of Linux: if you're about to enter a discussion and speak authoritatively on a topic, do your research first. How can you really expect to be an expert on something you know nothing about? I don't come on LKML and engage in scheduler debates with you, acting as an expert on the subject who has done his research -- what is it about security that makes it OK for you to pose as an armchair expert about any arbitrary topic? Your time would be better spent learning instead of talking.

A simple "my mistake, I didn't do my research before opening my mouth" would have saved us both a lot of time. Pissing off me and the PaX team isn't exactly productive -- after all, where will your company get its new (effective) security features from?

-Brad

fixed in v2.6.31.1

Posted Sep 25, 2009 6:43 UTC (Fri) by PaXTeam (guest, #24616) [Link] (18 responses)

> criticizing and naming exploits after people you'd like to publicly shame is fun

it's not fun but it's what works for kernel developers. this is what Linus himself said in the past at least:

"I really _despise_ people who think security is an issue of hiding
bugs. If they then try to make themselves look good ("no zero-day
exploit, we fixed it immediately"), they're worse than low.

The only thing that seems to work for security is public shaming.

And yeah, I get personally embarrassed by some of the things we've
had too, and some of that public shaming from the bug can well fall
on me. I've had cases where I've simply _forgotten_ about some bug
that was reported to me, or more commpnly [sic] just overlooked it.
Shame on me. That's ok."

> but wouldn't fixing problems be even more productive?

but we've always done that. what we find and fix ourselves is released in our patches (without the fanfare).

fixed in v2.6.31.1

Posted Sep 25, 2009 8:45 UTC (Fri) by hppnq (guest, #14462) [Link] (17 responses)

what we find and fix ourselves is released in our patches

You do not submit a patch for the Linux kernel. Instead, an exploit is released, with a lot of fanfare indeed. Selectively quoting Linus about the greatness of shame is not going to convince people that it is Linux security you are after. In fact, I am not even so convinced anymore that you are just plugging your own software at the expense of someone else's.

fixed in v2.6.31.1

Posted Sep 25, 2009 12:17 UTC (Fri) by spender (guest, #23067) [Link] (7 responses)

Why would we submit a patch for the Linux kernel when a patch already existed for the bug and was queued for the 2.6.31.1 kernel? We can't control the fact that the stable team sat on the release for 2 weeks while public exploits circulated (I wasn't the only one to write one). And what fanfare are you talking about? Ingo complained that there wasn't enough fanfare apparently, that I should have emailed full-disclosure/dailydave like I usually do and gotten the information out to everyone, and then surely again I would be insulted by people like you (and Ingo) for being an attention seeker by doing so.

-Brad

fixed in v2.6.31.1

Posted Sep 25, 2009 18:09 UTC (Fri) by dlang (guest, #313) [Link] (6 responses)

if the fix was already in the release process, why release an exploit at all?

also, it's very common for the -stable release to wait until after the initial merge window before releasing 2.6.x.1 the same developers who do the -stable work also work on the main kernel and that is a very busy two weeks.

fixed in v2.6.31.1

Posted Sep 25, 2009 18:20 UTC (Fri) by foom (subscriber, #14868) [Link] (5 responses)

> if the fix was already in the release process, why release an exploit at all?

Apparently it takes the existence of a public exploit for a kernel bug's severity to be properly
described as an arbitrary code execution exploit, instead of just a denial-of-service crash.

fixed in v2.6.31.1, also caught by StackProtector

Posted Sep 26, 2009 9:55 UTC (Sat) by mingo (subscriber, #31122) [Link] (4 responses)

It simply was not known as an exploitable bug. When we know something is exploitable a -stable kernel is released immediately, within hours.

(Let me know about a labeling method that actually works in practice - i'm not aware of any. Post facto labeling does not actually prevent bugs from getting mislabeled - it mostly only increases the security theatre - i.e. rewards the wrong kind of behaviour and the people who leech off of that.)

Also, note that if you had StackProtector enabled (CONFIG_CC_STACKPROTECTOR_ALL=y), the bug is not exploitable into a local root hole but gets caught and the break-in attempt gets exposed:

[   66.928106] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffff810c49d2
[   66.928109] 
[   66.939059] Pid: 2628, comm: exploit Not tainted 2.6.31 #17228
[   66.944947] Call Trace:
[   66.947423]  [ffffffff81622ba6] panic+0x84/0x12f
[   66.952238]  [ffffffff810c49d2] ? sys_perf_counter_open+0x660/0x672
[   66.958729]  [ffffffff810500cd] __stack_chk_fail+0x27/0x57
[   66.964419]  [ffffffff810c49d2] sys_perf_counter_open+0x660/0x672

It's a good thing we revived this feature in recent kernels, it already caught real kernel bugs and also catches exploit attempts.

fixed in v2.6.31.1, also caught by StackProtector

Posted Sep 26, 2009 12:47 UTC (Sat) by spender (guest, #23067) [Link] (1 responses)

So I have the answer to my question above then: "Is it the case that yourself, Paul Mackerras, Xiao Guangrong, Greg Kroah-Hartman, Peter Zijlstra, Eugene Teo and the other members of oss-security *ALL* can not plainly see that trivial, classic stack overflows, like the one signed-off on the fix for, are exploitable for arbitrary code execution? Did you all really believe that it was purely a "kernel crash" as noted in the commit message you all reviewed and signed off on?"

Let the record show that the Linux kernel development community and oss-security can't spot the exploitability of a trivial stack overflow.

It would be funny if it weren't so pathetic.

-Brad

fixed in v2.6.31.1, also caught by StackProtector

Posted Sep 26, 2009 13:16 UTC (Sat) by mingo (subscriber, #31122) [Link]

Yes, it indeed happens.

I take responsibility for this having slipped through (i should have caught it), and let me defend those other hard working people, who take an active part in the upstream kernel development process. Yes, nobody is infallible, and no, i dont think your repeated attempts to ridicule them is fair or justified.

If you think you could do better then i'd invite you to take part in the process instead of taking pot shots at it externally. Otherwise you'll never know whether you could do better than those whom you attack so viciously.

The fundamental question there (if you care): are you able to participate in a constructive community? Judging by your messages your life seems to be defined by hatred.

fixed in v2.6.31.1, also caught by StackProtector

Posted Sep 26, 2009 13:20 UTC (Sat) by spender (guest, #23067) [Link] (1 responses)

I seem to recall someone talking about another exploit being unexploitable from stackprotector. Right, it was you:
http://www.pubbs.net/kernel/200904/62088/
"the vmsplice exploit would only have been caught by the -ALL variant."

How quickly you forgot this:
http://copilotconsulting.com/mail-archives/linux-kernel.2...

and the fact that you actually committed his fix (and so were involved in the discussion above):
http://copilotconsulting.com/mail-archives/linux-kernel.2...

Did you have any proof for that one? That SSP stops exploitation of a vuln that doesn't even involve overwriting a return address?

-Brad

fixed in v2.6.31.1, also caught by StackProtector

Posted Sep 26, 2009 13:45 UTC (Sat) by mingo (subscriber, #31122) [Link]

Did you have any proof for that one? That SSP stops exploitation of a vuln that doesn't even involve overwriting a return address?

Indeed, i was wrong about that in the changelog, mea culpa. StackProtector has its place, but it would not have stopped the vmsplice exploit.

Thanks,

Ingo

fixed in v2.6.31.1

Posted Sep 25, 2009 19:25 UTC (Fri) by PaXTeam (guest, #24616) [Link] (8 responses)

> You do not submit a patch for the Linux kernel. Instead, an exploit is released, with a lot of fanfare indeed.

wow, lots of accusations there. let's see.

first of all, i didn't release let alone write this exploit, spender did, so you'll have to take that up with him, not me.

second, since you're seemingly unfamiliar with the Linux kernel development process, let me explain the situation briefly. your first task is to read Documentation/SubmittingPatches which is a file in the kernel source tree. in there you will find something called the Developer's Certificate of Origin (in section 12 as of .31). careful reading will reveal the following bits:

| then you just add a line saying
|
| Signed-off-by: Random J Developer <random@developer.example.org>
|
| using your real name (sorry, no pseudonyms or anonymous contributions.)

notice that anonymous contributions are explicitly not allowed (those words were actually added because of a patch of mine a few years ago). so there is the answer to your accusation.

> Selectively quoting Linus about the greatness of shame is not going to convince people that it is Linux security you are after.

why was that selective again? but more to the point, it seemingly works and makes people take a second look and tell something closer to the truth in turn. that can only be good for security, or do you believe that covering up security bugs is better?

> In fact, I am not even so convinced anymore that you are just plugging your own software at the expense of someone else's.

did you even read what i responded to? it was a suggestion about releasing our own security fixes as an alternative to the so-called 'stable' series (apparently that implies neither timely nor honest). obviously when we've already done that work for many years now, i'll not withhold the fact. as for the expense part, it sounds like as if you didn't like the taste of your own medicine.

fixed in v2.6.31.1

Posted Sep 26, 2009 10:45 UTC (Sat) by mingo (subscriber, #31122) [Link] (2 responses)

notice that anonymous contributions are explicitly not allowed (those words were actually added because of a patch of mine a few years ago). so there is the answer to your accusation.

I don't question your honesty, but FYI, your level of paranoia is reaching clinical levels.

If you don't trust others to know your real name, why do you expect millions of others to trust your anonymous word that the code you have contributed is indeed yours, with no legal strings attached? (such as an employer you work for currently and who thus does not know (and cannot know, due to the pseudonym) that you contribute in corporate time, etc.)

As far as it being added to the SOB requirements based on your case - that might simply be because in 99.99% of the cases people are actually proud of being mentioned in the Linux kernel source (it's even a good point for your career so generally considered stupid if you forfeit it), so this situation was simply not contemplated before?

Nevertheless i do always look at your fix patches and accept them (for subsystems i maintain) if they are good. I add my own signoff so i take responsibility for your patches, and credit you for the fix. I do think you are genuinely interested in (and worried about) Linux security, you are competent and capable, and I think you are doing good fixes.

As an example look at this upstream fix that you wrote:

e003208: x86: fix stackprotector canary updates during context switches

So there's absolutely no impediment to you contributing fixes to the Linux kernel, if you chose to do so. All it takes is to find someone with a real name who trusts you enough to take fixes from you.

(bigger patches are still a problem of course.)

the so-called 'stable' series (apparently that implies neither timely nor honest).

Do you have proof of (or even an indication of) that -stable is dishonest? I don't have such an indication, and i take honesty and security seriously.

Have you considered alternative possibilities, such as honest mistakes or lack of information? Or the possibility of maintainers being genuinely worried about and interested in a far wider spectrum of system stability than the important but (technically) narrow sub-set of security related bugs?

fixed in v2.6.31.1

Posted Sep 27, 2009 10:34 UTC (Sun) by nix (subscriber, #2304) [Link] (1 responses)

We've had this flamewar before. From those past flamewars, 'dishonest' in
PaXspeak appears to mean 'does not research every single patch to
determine if it fixes an exploitable hole, then pause the release while a
CVE is assigned and rewrite the changelog to include the CVE number and
exploit'. If a single hole lacks a CVE we get flames on LWN (why pick on
LWN, which is just reporting them? Search me. Maybe he wants as many
people as possible to dislike him. LWN used to have a pleasant collegial
comment atmosphere before these PaXTeam and Brad Spengler came along. Now
it often feels like alt.flame: I certainly want a killfile again.)

So we have PaXTeam on one side damning the stable team for not doing all
*that* and Brad on the other damning them for not releasing quickly. You
can't win.

fixed in v2.6.31.1

Posted Sep 27, 2009 10:35 UTC (Sun) by nix (subscriber, #2304) [Link]

Allow me to thank Greg et al for continuing with stable maintenance
despite getting 'thanks' and accusations of incompetence, lying, and
malice with virtually every release. I know I'd have given up long ago and
told these horrible people to do the releases themselves if they cared so
much about them. Greg is more mature than that thankfully. :)

fixed in v2.6.31.1

Posted Sep 27, 2009 7:21 UTC (Sun) by hppnq (guest, #14462) [Link] (4 responses)

did you even read what i responded to? it was a suggestion about releasing our own security fixes as an alternative to the so-called 'stable' series

busterb suggested:

Brad, why don't you and PaXTeam just release an intermediate .x stable kernel patch series to fix the problems you find?

You could set up a git repository with the fixes you find from which the -stable maintainers could pull, instead of wasting your time and energy on arguing with people over things that have nothing to do with security.

As for real security: I ran the perf_counter exploit through Brad's framework on my pristine, default Ubuntu distribution and it fails right after it killed pulseaudio, so it was only a bit annoying.

fixed in v2.6.31.1

Posted Sep 27, 2009 12:32 UTC (Sun) by spender (guest, #23067) [Link] (3 responses)

You ran the wrong exploit then. I made two versions of the exploit, one that requires a NULL mapping, and one that doesn't. The one that doesn't won't do anything at all with pulseaudio.
./run_nonnull_exploits.sh and chose "Ingo m0wnar"

Run the wrong exploit, obviously get the wrong results ;)

-Brad

fixed in v2.6.31.1

Posted Sep 27, 2009 14:06 UTC (Sun) by hppnq (guest, #14462) [Link] (2 responses)

I was just pulling your leg, my system -- pristine, default Ubuntu -- is not running 2.6.31 in the first place. ;-) You may think this is silly, but I think that this observation makes a lot of sense when it comes to "real" security. The first question, when doing a vulnerability assessment, is not "Is it remotely exploitable?", it is "Are we running that stuff?". So in this case, anyone who is running 2.6.31 with perf counters on a system that handles untrusted user data is likely to be vulnerable. Not too many people, I should think, fall in that category without knowing it.

That said, these bugs need to be fixed. But your sense of urgency and your tone are easily ignored -- sometimes much too easily -- and therefore, the question whether you plan to ever cooperate better with the kernel developers remains valid and to the point. If you would strip all communication of its unneeded emotion, and for instance, simply set up a repository that contains the fixes you find during your research, this could prove to be very fruitful to all parties.

fixed in v2.6.31.1

Posted Sep 27, 2009 14:19 UTC (Sun) by spender (guest, #23067) [Link] (1 responses)

As I mentioned elsewhere, anyone using a distro config will have perf counters automatically enabled without knowing it (since it gets turned on if PROFILING=y, which the vendors all enable).

As for the repository:
http://grsecurity.net/test/grsecurity-2.1.14-2.6.31.1-200...

our code has always been open for anyone to pull fixes from.

PS: what system doesn't handle untrusted user data?

-Brad

fixed in v2.6.31.1

Posted Sep 27, 2009 20:32 UTC (Sun) by hppnq (guest, #14462) [Link]

As I mentioned elsewhere, anyone using a distro config will have perf counters automatically enabled without knowing it (since it gets turned on if PROFILING=y, which the vendors all enable).

You mean, in the case where users download 2.6.31 and blindly reuse a non-matching distribution-supplied config? That is not a safe practice. Otherwise, I have to assume that either the distribution or the user knows what is turned on, with what impact.

our code has always been open for anyone to pull fixes from.

Incorporating fixes in your own patches is not the same as fixing bugs in the vanilla kernel. (The link doesn't work, so I am assuming it points to a full-blown grsecurity patch, which is not part of the vanilla kernel.) That said, you seem to write excellent code, and I wish more of your efforts would end up in the kernel in a way that satisfies you, the kernel developers, and Linux users alike. A git repository seems like a good idea. It's not so much the code contribution that's important (crucial fixes will always be picked up), but the cooperation it would help shape.

PS: what system doesn't handle untrusted user data?

Well, in the office we of course work with data classifications (and there is no "completely trusted" one), but here I would say a home or test system that is not connected to a public network fits the bill. When referring to systems that do handle untrusted data, I was thinking of Internet facing systems that, for instance, allow FTP or shell access, and I couldn't think of a reason why an admin would turn on profiling on them, or why the admin would not have a security policy based on the assumption that these systems will be hacked to pieces sooner or later.

Kernel release status

Posted Sep 24, 2009 4:05 UTC (Thu) by foom (subscriber, #14868) [Link] (18 responses)

> "Reducing attack surface" through access control systems (take your pick) can't help when
> poorly-written and under-reviewed system calls like this one get added to a "stable" kernel.

If the access control system reduced the attack surface such that it didn't include this buggy system
call, I'd expect that to be pretty effective at preventing this exploit.

Kernel release status

Posted Sep 24, 2009 4:29 UTC (Thu) by spender (guest, #23067) [Link] (17 responses)

If such an access control system existed -- yes. I believe seccomp is the only one (by virtue of the fact that it only allows a handful of system calls and can only be used on a tiny subset of specific applications).

Any other mainline access control system will do nothing, as they control access based on defined LSM hooks that won't exist for new system calls. Greater scrutiny should be applied to new system calls before they're added especially for this reason (that there will be no mitigation).

-Brad

Kernel release status

Posted Sep 26, 2009 10:08 UTC (Sat) by mingo (subscriber, #31122) [Link] (16 responses)

Reality is that StackProtector (CONFIG_CC_STACKPROTECTOR=y) makes the bug unexploitable.

Here's what the vanilla v2.6.31 kernel gives if i try to run the exploit you are distributing:

[   66.928106] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffff810c49d2
[   66.928109] 
[   66.939059] Pid: 2628, comm: exploit Not tainted 2.6.31 #17228
[   66.944947] Call Trace:
[   66.947423]  [ffffffff81622ba6] panic+0x84/0x12f
[   66.952238]  [ffffffff810c49d2] ? sys_perf_counter_open+0x660/0x672
[   66.958729]  [ffffffff810500cd] __stack_chk_fail+0x27/0x57
[   66.964419]  [ffffffff810c49d2] sys_perf_counter_open+0x660/0x672

It reduces a local root hole to a DoS. (Which is still a bug of course - just with a different severity.)

Kernel release status

Posted Sep 26, 2009 12:38 UTC (Sat) by spender (guest, #23067) [Link] (5 responses)

Ingo, always the one to come up with the contrived examples to save face.

First, tell me how long that config option has been in the kernel. Then, please tell me, for how long has that feature you mention actually worked? (This part might be difficult to figure out because no one at your company bothered to make sure it actually worked after they threw the code in the kernel). Also please tell me how many distro configurations actually use it. My videos should give you a hint.

PS: didn't a whole ton of stack infoleaks just get fixed? And isn't that cookie per-process (allowing me to perform repeated infoleaks in the same process as my exploit to determine the stack cookie)? I'm sure after finding a dozen of those in the past month, there aren't any more laying around to exploit. Oh, right.

-Brad

Kernel release status, StackProtector

Posted Sep 26, 2009 12:57 UTC (Sat) by mingo (subscriber, #31122) [Link] (4 responses)

Ingo, always the one to come up with the contrived examples to save face.

There's no need for me to save face - my face is blushing deep red from having let that hole slip through - i should do better than that.

I take your reply as a confirmation that CONFIG_CC_STACKPROTECTOR=Y indeed stops your exploit dead in its tracks - good to hear independent confirmation of such things.

FYI, a number of distributions that care about security features have CONFIG_CC_STACKPROTECTOR=Y enabled by default, such as Fedora:

 spirit:~> grep STACKPROT /boot/config-2.6.31-0.204.rc9.fc12.x86_64
 CONFIG_CC_STACKPROTECTOR_ALL=y
 CONFIG_CC_STACKPROTECTOR=y

So the bug is not exploitable on those distros.

Whether distributions are exploitable is a relevant piece of information: it influences how urgently a system administrator upgrades a kernel. A local root exploit is treated at a different severity level from a local DoS in most places. You probably know that.

That does not make it in any way better or worse kind of bug - it simply adds to the pool of information about this bug and about the exploit - a piece of information you either did not know about or which you chose not to disclose.

Thanks,

Ingo

Kernel release status, StackProtector

Posted Sep 26, 2009 13:26 UTC (Sat) by spender (guest, #23067) [Link] (1 responses)

Here we go again with the contrived (and misleading) examples. You quote the config of FC12, a distro that hasn't even shipped yet.

How about something like FC11, a distro that does actually exist, which I exploited in my video with a fully patched system using their config?

How about the latest version of Ubuntu x64, a distro that does actually exist, which I exploited in my video with a fully patched system using their config?

Can you name *any* distro that isn't a contrived example of a beta distro no-one is using that actually enables this feature?

-Brad

Kernel release status, StackProtector

Posted Sep 26, 2009 13:41 UTC (Sat) by mingo (subscriber, #31122) [Link]

How about something like FC11, a distro that does actually exist, which I exploited in my video with a fully patched system using their config?

FYI, the latest update kernel for Fedora 11 is v2.6.30 - while the bug only exists in v2.6.31.

If you wanted bleeding edge v2.6.31 you could have gone to rawhide (like i did). There the bug was present but not exploitable as a local root hole due to StackProtector catching it.

Instead you chose to build your own v2.6.31 kernel for Fedora 11, using the v2.6.30 config - not the v2.6.31 rawhide config (which has stackprotector enabled).

That's how you were able to exploit it. Had you used the F11 kernel, or had you used the v2.6.31 rawhide kernel you'd not have been able to exploit it.

(Again - this does not remove anything from my (serious) responsibility over this bug. It just adds to the pool of information.)

Thanks,

Ingo

Kernel release status, StackProtector

Posted Sep 26, 2009 13:45 UTC (Sat) by spender (guest, #23067) [Link]

"Unexploitable" isn't a term I would use. As I mentioned, it just takes an infoleak on the stack or per-cpu data to determine the cookie and then use this in the exploit payload. That a particular exploit is stopped doesn't logically imply that a working exploit cannot be written.

I can write such an exploit just for you, but since I'd be writing it just to prove a point to you (which I shouldn't have to do, see above re: taking some security training courses), you'd have to do something more for me than just stop using the word "unexploitable."

I'm open to ideas.

-Brad

Kernel release status, StackProtector

Posted Sep 27, 2009 10:37 UTC (Sun) by nix (subscriber, #2304) [Link]

Does this feature actually work on x86 these days? Last time I looked at
it it only worked on x86-64, which ruled it out for pretty much every
external-facing system I run (as they tend to be old cheap x86-32s).

Kernel release status

Posted Sep 26, 2009 17:03 UTC (Sat) by spender (guest, #23067) [Link] (9 responses)

PS: Using the same exploit I distributed I was still able to exploit the vulnerability on my Ubuntu 9.04 box with CONFIG_CC_STACKPROTECTOR=y and CONFIG_CC_STACKPROTECTOR_ALL=y. At first I simply enabled the option and recompiled, saw that the exploit still worked -- so I thought perhaps it was a fluke and recompiled the entire kernel with the new configuration. I ran the exploit and still got root. I'm using gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3

a grepping of /proc/kallsyms shows that __stack_chk_fail exists
A disassembly of perf_counter.o shows that perf_counter_mmap and perf_counter_comm get instrumented, but not sys_perf_counter_open.

There goes your theory ;) Now you have something to work on this weekend.

-Brad

Kernel release status

Posted Sep 26, 2009 17:17 UTC (Sat) by spender (guest, #23067) [Link] (6 responses)

I can make public any binaries, configs, video, etc in case you don't trust my analysis above (for whatever reason). Just ask.

-Brad

Kernel release status

Posted Sep 26, 2009 18:19 UTC (Sat) by spender (guest, #23067) [Link] (5 responses)

I added -Wstack-protector to the list of flags, but it didn't warn about anything in perf_counter.c missing protection (though it did on many other functions). I then verified again that the syscall in question wasn't being protected.

When you figure out the problem, don't forget my "bug reporter" line, since I've never reported a bug in Linux before, I'm told.

-Brad

Kernel release status

Posted Sep 29, 2009 12:38 UTC (Tue) by spender (guest, #23067) [Link] (4 responses)

Paging Dr. Molnar..

You had a lot to say before, but have been silent for days since I brought up stack-protector not doing what you claimed. I think it should be cleared up, since you spammed your same comment about stack-protector making the bug unexploitable in at least four different threads, and in light of what I posted (stack-protector not protecting the syscall in question with CONFIG_CC_STACKPROTECTOR_ALL), that doesn't seem to be true.

Obviously you're capable of responding:
http://lkml.org/lkml/2009/9/29/41

Do you have any updates for the readers?

-Brad

Kernel release status

Posted Oct 1, 2009 9:44 UTC (Thu) by mingo (subscriber, #31122) [Link] (3 responses)

You had a lot to say before, but have been silent for days since I brought up stack-protector not doing what you claimed.

I was away attending a real-time conference. (There was an LWN.net article about that conference.)

At a quick glance (i'm still not up to speed after my trip) my guess is that you couldn't reproduce my results because you used an older GCC version. I used GCC 4.4. Earlier GCCs not catching all overflows is of course a problem - but it would be nice if you could check if you can reproduce the problem with a recent version of GCC.

I also noticed that you apparently avoided Fedora rawhide for your tests - the only distribution which actually had the 2.6.31 kernel.

So if you have intellectual curiosity you might want to check it on rawhide - when producing exploit videos you are using Fedora so it should be easy enough.

Another update is that we are working on various measures to harden the Linux kernel against similar exploits in the future. See LKML for details. Again, feel free to contribute to Linux security efforts if you are interested.

Thanks,

Ingo

Kernel release status

Posted Oct 1, 2009 13:05 UTC (Thu) by spender (guest, #23067) [Link] (2 responses)

I could check that, but don't you have a security team there at Red Hat whose job it is to investigate things like these? Or shouldn't the people who introduced the feature into the kernel be responsible for it working as they claim? I have no interest in installing rawhide. When it's released officially, then I'll likely upgrade to it.

You mentioned various measures, could you link me to the ones I haven't found? All I've been able to find so far is the copy_from_user changes which use __builtin_object_size (and fix up cases where gcc couldn't determine the size for copies into stack objects (where that stack object didn't exist in another file)). But there are caveats for this, some of which are mentioned and others that are not.

Here are the links I found for the benefit of other readers (all involving the same basic feature):
http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-09/m...
http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-09/m...
http://linux.derkeiler.com/Mailing-Lists/Kernel/2009-09/m...

I happen to have some experience with this particular subject because I wrote up a full FORTIFY_SOURCE implementation for the kernel:
http://grsecurity.net/~spender/fortify_source_30_percent_...

You can tell by the name how impressed I was with the amount of coverage.
It was so ineffective, in fact, that I haven't included it in grsecurity. I went through and analyzed the disassembly of my own code (all the code in the grsecurity directory) and some other kernel code to see how it was being protected by the extra __builtin_object_size checks for memory/string operations, as well as copy_to/from_user. What I found was that it was basically only "proving"/protecting trivial/unexploitable cases. I even added in attributes for the kernel allocators so that the size of the objects they returned could be determined (in some cases). I think the addition of those attributes added 10% coverage (from 20% initially).

I also noticed __builtin_object_size couldn't figure out the size of "blah" in the simple example below:

typedef struct {
int test;
char blah[10];
int test2;
} teststruct;

int blah(char *buf, int len)
{
teststruct wee;

memcpy(&wee.blah, buf, len);
}

Basically, any array within a structure (and there are tons of examples of this within the kernel) gcc couldn't determine the size of. Nobody apparently had investigated this the entire time __builtin_object_size had been around, or nobody cared. I reported the problem and received no credit for it (further proof for you that I don't find bugs in Linux), but the fix for it should be in some new version of gcc (I don't know if it's made it into a version anyone's using yet).

I found 2 or 3 unexploitable overflows detected at compile-time with the above patch (or perhaps only after I added some enhancements of my own that would have mimicked the gcc fix) and reported those as well (more bugs I don't report/fix).

So now here you have implemented a feature that implements "provable" security, yet what it actually protects is wildly dependent upon the version of gcc the person is using. I noticed a warning was added (by configure option) to show the cases where gcc can't figure out the size. Judging from my own experience above, this will most likely be useless given that gcc can't determine the size in 70% of the cases, and so cases that the developer could fix easily would be thrown in with 100 other cases that he/she couldn't fix easily or at all.

It's an improvement, but a very minor one. A better improvement would be to make -fstack-protector-all actually do what it claims to do.

-Brad

Kernel release status

Posted Oct 1, 2009 14:23 UTC (Thu) by spender (guest, #23067) [Link]

Two more quick things:

I recently noticed:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-...

Which looks eerily similar to this (from July):
http://grsecurity.net/~spender/oooo_fancy.diff

Mentioned last month here:
http://lwn.net/Articles/346801/

And just as a demonstration of the limitations of __builtin_object_size, you show how it would have prevented the perf_counter exploit (which is more a statement of how utterly trivial the vulnerability was). It would indeed have prevented it (even with an older 4.X compiler I believe). However, making one small change of having perf_copy_attr() become a global function, and __builtin_object_size suddenly becomes useless. There's tons of real kernel code like that to be found.

-Brad

Kernel release status

Posted Oct 1, 2009 14:38 UTC (Thu) by spender (guest, #23067) [Link]

Also just noticed this: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-...

It's a useful change -- I hope we see more kernel-wide changes like this in the future.

With some extra compiler flags you could then go through and remove the (now) useless < 0 checks.

-Brad

Kernel release status

Posted Sep 27, 2009 10:49 UTC (Sun) by nix (subscriber, #2304) [Link] (1 responses)

That's interesting. I'd expect the behaviour you see with only
CONFIG_CC_STACKPROTECTOR enabled, as this only protects functions
containing large char arrays (not arrays of other types, not aggregates,
not pointers to them). But _ALL should change this, and the presence of
the aggregate 'struct perf_counter_attr attr' should then enable guard
generation for sys_perf_counter_open(). Yet it doesn't.

I wonder if there's something about SYSCALL_DEFINE*() that is blocking
stack protector... doesn't look like it, asmlinkage shouldn't do it. I'll
have to beat up GCC a bit and figure out what's going on...

Kernel release status

Posted Sep 27, 2009 12:52 UTC (Sun) by spender (guest, #23067) [Link]

CONFIG_CC_STACKPROTECTOR cannot be enabled alone anymore. See the link I pasted above where Ingo was talking about stackprotector_all being on by default because otherwise it wouldn't have stopped vmsplice (which it had nothing to do with).

I don't believe it has to do with SYSCALL_DEFINE, because I've found other syscalls that were protected (disassemble kernel/sys.o for example and you'll see that sys_prctl is protected).

But yes, there is definitely an important weakness/vulnerability in the stackprotector implementation (which must have been silently fixed in a newer gcc perhaps? for Ingo to have gotten his results). Nevertheless, this makes it quite a poor protection, and some mention should be made in the configure help about the ambiguous protection it provides depending on your compiler version (and give a recommended minimum compiler version that does what the feature actually claims).

-Brad