Posted Sep 25, 2009 7:10 UTC (Fri) by mingo (subscriber, #31122)
[Link]
Agreed, it's a good idea.
Note that the fix for this bug was queued up for 2.6.31.1 before the exploit was posted. (That doesn't of course in any way mitigate the fact that this was an exploitable bug.)
Mr. Spender took the commit log of the fix that others already debugged, created, tested and submitted, and made it easier for script kiddies to vandalize Linux systems by posting an exploit.
As i expressed it earlier, i wish Mr. Spender had more empathy with the project he is trying to contribute to and if he were more mature in not harming it at the same time. Posting an almost-exploit or just saying that it's exploitable is generally enough to get immediate attention - while also avoiding a lot of immediate harm.
Btw., that posting should be immediate and as public as possible, to notify all interested parties [which includes script kiddies and black hats as well] - no matter how embarrassing it might be to the developers and maintainers involved.
Mr. Spender's choice to maximally help script kiddies while trying to maximally harm the people who are actually work on making Linux better (short of him committing a potential felony by launching attacks himself or selling the exploit) certainly qualifies his character - but it's also useful even in this form, no doubt about that.
I'm sure that a parallel or complimentary security effort with real ongoing work injected would be helpful. The sanest approach would be to also notify the -stable maintainers IMHO - the -stable maintainers are very responsive in general.
fixed in v2.6.31.1
Posted Sep 25, 2009 8:52 UTC (Fri) by drag (subscriber, #31333)
[Link]
> (short of him committing a potential felony by launching attacks himself
> or selling the exploit)
For every spender I expect there are a dozen people doing exactly that.
Linux is big business now and big money is to be had by trolling the kernel
commit logs and looking for mislabeled vulnerabilities. I suppose that
there is enough money in it that a person could do pretty much nothing else
but follow along with kernel development and make a living that way.
Personally I have to maintain a handful of different kernels for personal
and professional reasons. It would be nice if I could trust the commit
logs. :)
Although it would be nice if Spender could direct his attention towards
finding exploits in rc kernels! :) Maybe one of the big commercial Linux
guys would hire him or some other group of people to concentrate entirely
on code quality in terms of security and figuring out more and more
automated checks and whatnot. Something can be done, I suppose.
fixed in v2.6.31.1
Posted Sep 25, 2009 9:34 UTC (Fri) by mingo (subscriber, #31122)
[Link]
Although it would be nice if Spender could direct his attention towards
finding exploits in rc kernels! :) Maybe one of the big commercial Linux
guys would hire him or some other group of people to concentrate entirely
on code quality in terms of security and figuring out more and more
automated checks and whatnot. Something can be done, I suppose.
Indeed, people who find and fix bugs in Linux, with a special attention on security issues, and who try to help people and who are able to work with people are a hot commodity and get picked up by Linux companies quickly.
(Because, not the least, the very same skills and talents can also be used to fix robustness and stability problems and to design better code.)
Alas, Spender is not such a person AFAICT - or if he is, he has not demonstrated such bug finding and communication skills yet. (in a way visible to me at least)
He has not tried to seriously work with the upstream kernel, and in the Git history of Linux, spanning 4 years and over 160,000 changes/fixes, there's not a single commit/fix authored by him. There's not a single commit log entry where he was mentioned as bug reporter or tester.
It really takes a concentrated effort to stay out of the contribution zone to that level, if one is interested in security bugs. (In contrast i found a handful of 'PaX Team' commits.)
Most of the exploits i've seen from him so far were based on other people's work and generally weren't genuine security bugs he himself found. Some of the exploits show creativity (but are not always credited to Mr. Spender himself so it's unclear whether he wrote them).
One has to question the judgement (and wisdom) of someone opting to help attackers with the most ruthless yet borderline legal means, just to raise 15 minutes of attention.
I personally think it's still useful as long as there's still a net benefit to Linux (which there is here) - and the resulting embarrassment to developers that keeps us all sharper is helpful too - but the process of elaborate and very public self-destruction is always sad to observe.
The thing is, finding genuine security bugs in Linux is hard and takes a lot of skill and a lot of talent. Kenrel people and Linux companies will quickly notice if you have that skill.
fixed in v2.6.31.1
Posted Sep 25, 2009 15:03 UTC (Fri) by spender (subscriber, #23067)
[Link]
Ingo, let's be clear about what's happening here. You waltzed into this conversation acting as an expert on the subject, having done no research whatsoever. Not only that, you were shilling for your company's security features without having done any verification that they did what you claimed.
Now, from there, having been so embarrassingly wrong, and not having had the common sense to review the links in my post to which you were replying, one would expect a "I was mistaken" response -- but not from you of course. In typical fashion, what we get instead is a series of ad-hominem attacks. Allow me to enumerate your charges:
You claim that I haven't found or fixed bugs in Linux.
You claim that my work is plagiarized (funny concept coming from you -- exec-shield, CFS, perf counters, etc).
You claim that I'm choosing to help attackers to gain attention for myself.
You claim that I'm engaging in borderline-felonious activity.
You claim that I harmed security by not immediately issuing a release to full-disclosure/dailydave/LKML about the vulnerability.
So let's take these claims one by one.
You claimed that I harmed security by not immediately notifying everyone about the vulnerability. This is a curious claim, because it doesn't look good for you or for Linux kernel development in general. And in fact it negates one of your other claims (that observing an OOPs and submitting a report for a "kernel crash" and writing one line of code is more difficult than actually determining the vulnerability class and exploiting it.)
So here is my question to you. Is it the case that yourself, Paul Mackerras, Xiao Guangrong, Greg Kroah-Hartman, Peter Zijlstra, Eugene Teo and the other members of oss-security *ALL* can not plainly see that trivial, classic stack overflows, like the one signed-off on the fix for, are exploitable for arbitrary code execution? Did you all really believe that it was purely a "kernel crash" as noted in the commit message you all reviewed and signed off on?
If that is the case, I feel sorry for Linux kernel development. Perhaps your respective companies could pay to send you to an entry-level security crash course to learn what gets done with trivial stack overflows.
If that's not the case, and you were aware that the vulnerability was exploitable for arbitrary code execution, why is the onus on me to inform the stable maintainers or the world at large of the vulnerability? Isn't that your job in the commit messages that you write and review? Is it somehow my fault that people are being knowingly mislead (including the security teams of the various vendors, who republished the "kernel crash" claim and had the CVE marked as a denial of service issue)? Is it my fault that nobody in the Linux kernel development community seems to think twice about "buffer overflow" and "kernel crash" being in the same sentence in a commit message? We (in conjunction with the rest of the security industry -- Linus was awarded with a Pwnie this year for being voted the worst at handling security issues) have been arguing for a long time now for you guys to end your silly practice of obfuscation and lies.
Put plainly, you're arguing with the wrong person. We keep demonstrating the harm of these silly commit messages; you'd be better served by growing a spine and launching your complaints at the people who are actually responsible for this current situation. I'd be happily surprised to see that happen, but suspect that it won't.
You claim that I'm engaged in borderline-felonious activity. Do you make the same claims about any other security researcher that posts exploits? About all the people in the security industry writing modules for Metasploit?
Public exploits are very effective in making a point. Here's some of the lessons you should have learned from this (but apparently were asleep during class):
1) An exploit not being made public isn't an excuse to pretend one doesn't exist
2) All of the exploits I've written have been developed and fully-weaponized in under an hour.
3) Given how quickly these can be developed, it's been made clear that the stable maintainers and Linux vendors are ill-equipped to respond quickly enough.
4) Don't rest on the laurels of your ineffective security measures.
5) Obfuscating commit messages doesn't stop someone who wants to write an exploit from figuring out what the issue is.
6) Even your most prized security features can be defeated.
If it wasn't clear to people before, it should be plainly clear now what kind of security "guarantees" can be provided when a framework exists that rewrites the SELinux code to pretend that it's in enforcing mode or completely defeats IMA. Linux was benefiting in appearance only by not having something like this published -- since it's nothing any private exploit writer hadn't written themselves. Now that it is public, the correct response should be "what can we do about this problem?" Fixing bugs isn't the answer (you'll just add more in the next kernel), neither is attacking the person who demonstrated the techniques to you. Did you forget what caused mmap_min_addr to be added to the kernel in the first place?
You know very well that I've found and fixed bugs and vulnerabilities in Linux. Have you forgotten our private conversations about exec-shield vulnerabilities I reported to you that you fixed silently? I still have the emails, if necessary. What about the missing NULL pointer checks issue? The SELinux vulnerability? The exploitability of NULL pointer dereferences in general? How quickly we forget, and what an obvious example of why I would have no interest in cooperating with you in the future. As many can attest (yourself included), I contact them privately about vulnerabilities in their code. I'm not always given credit (like with the SELinux issue). Very few of the things I've reported privately that have been fixed in the kernel I have been given credit for, so it's no surprise that you didn't find anything in your (rather contrived) search (much like your benchmarks, I wager). Given that I don't believe I've ever sent a single message to LKML, I don't imagine I would be listed as the author of any commit. Being listed as a bug reporter is something at the discretion of the committer, so again, no surprise there.
Furthermore, had it occurred to you that I'm not interested in finding and fixing your bugs? For each one I fix, you'll add two more. Finding and patching bugs doesn't improve security. Just like userland security was a cesspool until PaX arrived on the scene (where any memory corruption bug could be exploited by someone who read some simple Phrack articles), kernel security is the same way without any defensive techniques. It's developing those techniques, as I have been doing, that I'm interested in -- preventing exploit vectors, stopping exploitation of entire bug classes, not in wasting time fixing your individual bugs so that you can add more next week. Oh and you guessed it, it's my dream in life to one day work for a Linux vendor. Writing up 30 advisories every month for all the packages that use webkit, I can see it now!
You claim that my exploits are plagiarized, due to the fact that I don't put my name on some of them. Had it occurred to you that maybe I don't care so much about putting my name in it if it's obvious I wrote it? You can follow my postings on twitter where I discuss additions to the framework I developed. If you have questions about the authenticity of the code, those questions should exist whether I put my name in it or not.
You claim that I'm choosing to help attackers just to gain some attention for myself. I don't think posting messages only to my twitter account that only a handful of people read is an attempt to get attention. As I mentioned earlier, if I were to do what you suggested and let the entire world know, you would accuse me of the same attention seeking, so it's impossible for any researcher to escape your misdirected criticism. I know that members of the security teams for several vendors, including that of your own company, follow my twitter and so are immediately notified. In fact, you can see several examples of it on oss-security where they link to me explicitly. The attention I want to bring is to the mislabeling of security vulnerabilities caused by the kernel developers' official policy of obfuscating commit messages. The lies contained in them propagate into the CVE, which then give everyone a false representation of their risk. And you are complicit in this behavior.
So in summary, Ingo, I have a bit of advice for you that will avoid such embarrassment in the future, and then you won't have to waste your time on misguided and misinformed ad-hominem attacks against one of the few people who are actually improving the security of Linux: if you're about to enter a discussion and speak authoritatively on a topic, do your research first. How can you really expect to be an expert on something you know nothing about? I don't come on LKML and engage in scheduler debates with you, acting as an expert on the subject who has done his research -- what is it about security that makes it OK for you to pose as an armchair expert about any arbitrary topic? Your time would be better spent learning instead of talking.
A simple "my mistake, I didn't do my research before opening my mouth" would have saved us both a lot of time. Pissing off me and the PaX team isn't exactly productive -- after all, where will your company get its new (effective) security features from?