| |||||||||||||
Kernel release statusKernel release statusPosted Sep 24, 2009 4:05 UTC (Thu) by foom (subscriber, #14868)In reply to: Kernel release status by spender Parent article: Kernel release status
> "Reducing attack surface" through access control systems (take your pick) can't help when
> poorly-written and under-reviewed system calls like this one get added to a "stable" kernel.
If the access control system reduced the attack surface such that it didn't include this buggy system
(Log in to post comments)
Kernel release status Posted Sep 24, 2009 4:29 UTC (Thu) by spender (subscriber, #23067) [Link]
If such an access control system existed -- yes. I believe seccomp is the only one (by virtue of the fact that it only allows a handful of system calls and can only be used on a tiny subset of specific applications).
Any other mainline access control system will do nothing, as they control access based on defined LSM hooks that won't exist for new system calls. Greater scrutiny should be applied to new system calls before they're added especially for this reason (that there will be no mitigation).
-Brad
Kernel release status Posted Sep 26, 2009 10:08 UTC (Sat) by mingo (subscriber, #31122) [Link] If such an access control system existed -- yes. I believe seccomp is the only one (by virtue of the fact that it only allows a handful of system calls and can only be used on a tiny subset of specific applications). Reality is that StackProtector (CONFIG_CC_STACKPROTECTOR=y) makes the bug unexploitable. Here's what the vanilla v2.6.31 kernel gives if i try to run the exploit you are distributing:
[ 66.928106] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffff810c49d2 [ 66.928109] [ 66.939059] Pid: 2628, comm: exploit Not tainted 2.6.31 #17228 [ 66.944947] Call Trace: [ 66.947423] [ffffffff81622ba6] panic+0x84/0x12f [ 66.952238] [ffffffff810c49d2] ? sys_perf_counter_open+0x660/0x672 [ 66.958729] [ffffffff810500cd] __stack_chk_fail+0x27/0x57 [ 66.964419] [ffffffff810c49d2] sys_perf_counter_open+0x660/0x672 It reduces a local root hole to a DoS. (Which is still a bug of course - just with a different severity.)
Kernel release status Posted Sep 26, 2009 12:38 UTC (Sat) by spender (subscriber, #23067) [Link]
Ingo, always the one to come up with the contrived examples to save face.
First, tell me how long that config option has been in the kernel. Then, please tell me, for how long has that feature you mention actually worked? (This part might be difficult to figure out because no one at your company bothered to make sure it actually worked after they threw the code in the kernel). Also please tell me how many distro configurations actually use it. My videos should give you a hint.
PS: didn't a whole ton of stack infoleaks just get fixed? And isn't that cookie per-process (allowing me to perform repeated infoleaks in the same process as my exploit to determine the stack cookie)? I'm sure after finding a dozen of those in the past month, there aren't any more laying around to exploit. Oh, right.
-Brad
Kernel release status, StackProtector Posted Sep 26, 2009 12:57 UTC (Sat) by mingo (subscriber, #31122) [Link] Ingo, always the one to come up with the contrived examples to save face. There's no need for me to save face - my face is blushing deep red from having let that hole slip through - i should do better than that. I take your reply as a confirmation that CONFIG_CC_STACKPROTECTOR=Y indeed stops your exploit dead in its tracks - good to hear independent confirmation of such things. FYI, a number of distributions that care about security features have CONFIG_CC_STACKPROTECTOR=Y enabled by default, such as Fedora:
spirit:~> grep STACKPROT /boot/config-2.6.31-0.204.rc9.fc12.x86_64 CONFIG_CC_STACKPROTECTOR_ALL=y CONFIG_CC_STACKPROTECTOR=y So the bug is not exploitable on those distros. Whether distributions are exploitable is a relevant piece of information: it influences how urgently a system administrator upgrades a kernel. A local root exploit is treated at a different severity level from a local DoS in most places. You probably know that. That does not make it in any way better or worse kind of bug - it simply adds to the pool of information about this bug and about the exploit - a piece of information you either did not know about or which you chose not to disclose. Thanks, Ingo
Kernel release status, StackProtector Posted Sep 26, 2009 13:26 UTC (Sat) by spender (subscriber, #23067) [Link]
Here we go again with the contrived (and misleading) examples. You quote the config of FC12, a distro that hasn't even shipped yet.
How about something like FC11, a distro that does actually exist, which I exploited in my video with a fully patched system using their config?
How about the latest version of Ubuntu x64, a distro that does actually exist, which I exploited in my video with a fully patched system using their config?
Can you name *any* distro that isn't a contrived example of a beta distro no-one is using that actually enables this feature?
-Brad
Kernel release status, StackProtector Posted Sep 26, 2009 13:41 UTC (Sat) by mingo (subscriber, #31122) [Link] How about something like FC11, a distro that does actually exist, which I exploited in my video with a fully patched system using their config? FYI, the latest update kernel for Fedora 11 is v2.6.30 - while the bug only exists in v2.6.31. If you wanted bleeding edge v2.6.31 you could have gone to rawhide (like i did). There the bug was present but not exploitable as a local root hole due to StackProtector catching it. Instead you chose to build your own v2.6.31 kernel for Fedora 11, using the v2.6.30 config - not the v2.6.31 rawhide config (which has stackprotector enabled). That's how you were able to exploit it. Had you used the F11 kernel, or had you used the v2.6.31 rawhide kernel you'd not have been able to exploit it. (Again - this does not remove anything from my (serious) responsibility over this bug. It just adds to the pool of information.) Thanks, Ingo
Kernel release status, StackProtector Posted Sep 26, 2009 13:45 UTC (Sat) by spender (subscriber, #23067) [Link]
"Unexploitable" isn't a term I would use. As I mentioned, it just takes an infoleak on the stack or per-cpu data to determine the cookie and then use this in the exploit payload. That a particular exploit is stopped doesn't logically imply that a working exploit cannot be written.
I can write such an exploit just for you, but since I'd be writing it just to prove a point to you (which I shouldn't have to do, see above re: taking some security training courses), you'd have to do something more for me than just stop using the word "unexploitable."
I'm open to ideas.
-Brad
Kernel release status, StackProtector Posted Sep 27, 2009 10:37 UTC (Sun) by nix (subscriber, #2304) [Link]
Does this feature actually work on x86 these days? Last time I looked at
it it only worked on x86-64, which ruled it out for pretty much every external-facing system I run (as they tend to be old cheap x86-32s).
Kernel release status Posted Sep 26, 2009 17:03 UTC (Sat) by spender (subscriber, #23067) [Link]
PS: Using the same exploit I distributed I was still able to exploit the vulnerability on my Ubuntu 9.04 box with CONFIG_CC_STACKPROTECTOR=y and CONFIG_CC_STACKPROTECTOR_ALL=y. At first I simply enabled the option and recompiled, saw that the exploit still worked -- so I thought perhaps it was a fluke and recompiled the entire kernel with the new configuration. I ran the exploit and still got root. I'm using gcc (Ubuntu 4.3.3-5ubuntu4) 4.3.3
a grepping of /proc/kallsyms shows that __stack_chk_fail exists
There goes your theory ;) Now you have something to work on this weekend.
-Brad
Kernel release status Posted Sep 26, 2009 17:17 UTC (Sat) by spender (subscriber, #23067) [Link]
I can make public any binaries, configs, video, etc in case you don't trust my analysis above (for whatever reason). Just ask.
-Brad
Kernel release status Posted Sep 26, 2009 18:19 UTC (Sat) by spender (subscriber, #23067) [Link]
I added -Wstack-protector to the list of flags, but it didn't warn about anything in perf_counter.c missing protection (though it did on many other functions). I then verified again that the syscall in question wasn't being protected.
When you figure out the problem, don't forget my "bug reporter" line, since I've never reported a bug in Linux before, I'm told.
-Brad
Kernel release status Posted Sep 29, 2009 12:38 UTC (Tue) by spender (subscriber, #23067) [Link]
Paging Dr. Molnar..
You had a lot to say before, but have been silent for days since I brought up stack-protector not doing what you claimed. I think it should be cleared up, since you spammed your same comment about stack-protector making the bug unexploitable in at least four different threads, and in light of what I posted (stack-protector not protecting the syscall in question with CONFIG_CC_STACKPROTECTOR_ALL), that doesn't seem to be true.
Obviously you're capable of responding:
Do you have any updates for the readers?
-Brad
Kernel release status Posted Oct 1, 2009 9:44 UTC (Thu) by mingo (subscriber, #31122) [Link] You had a lot to say before, but have been silent for days since I brought up stack-protector not doing what you claimed. I was away attending a real-time conference. (There was an LWN.net article about that conference.) At a quick glance (i'm still not up to speed after my trip) my guess is that you couldn't reproduce my results because you used an older GCC version. I used GCC 4.4. Earlier GCCs not catching all overflows is of course a problem - but it would be nice if you could check if you can reproduce the problem with a recent version of GCC. I also noticed that you apparently avoided Fedora rawhide for your tests - the only distribution which actually had the 2.6.31 kernel. So if you have intellectual curiosity you might want to check it on rawhide - when producing exploit videos you are using Fedora so it should be easy enough. Another update is that we are working on various measures to harden the Linux kernel against similar exploits in the future. See LKML for details. Again, feel free to contribute to Linux security efforts if you are interested. Thanks, Ingo
Kernel release status Posted Oct 1, 2009 13:05 UTC (Thu) by spender (subscriber, #23067) [Link]
I could check that, but don't you have a security team there at Red Hat whose job it is to investigate things like these? Or shouldn't the people who introduced the feature into the kernel be responsible for it working as they claim? I have no interest in installing rawhide. When it's released officially, then I'll likely upgrade to it.
You mentioned various measures, could you link me to the ones I haven't found? All I've been able to find so far is the copy_from_user changes which use __builtin_object_size (and fix up cases where gcc couldn't determine the size for copies into stack objects (where that stack object didn't exist in another file)). But there are caveats for this, some of which are mentioned and others that are not.
Here are the links I found for the benefit of other readers (all involving the same basic feature):
I happen to have some experience with this particular subject because I wrote up a full FORTIFY_SOURCE implementation for the kernel:
You can tell by the name how impressed I was with the amount of coverage.
I also noticed __builtin_object_size couldn't figure out the size of "blah" in the simple example below:
typedef struct {
int blah(char *buf, int len)
memcpy(&wee.blah, buf, len);
Basically, any array within a structure (and there are tons of examples of this within the kernel) gcc couldn't determine the size of. Nobody apparently had investigated this the entire time __builtin_object_size had been around, or nobody cared. I reported the problem and received no credit for it (further proof for you that I don't find bugs in Linux), but the fix for it should be in some new version of gcc (I don't know if it's made it into a version anyone's using yet).
I found 2 or 3 unexploitable overflows detected at compile-time with the above patch (or perhaps only after I added some enhancements of my own that would have mimicked the gcc fix) and reported those as well (more bugs I don't report/fix).
So now here you have implemented a feature that implements "provable" security, yet what it actually protects is wildly dependent upon the version of gcc the person is using. I noticed a warning was added (by configure option) to show the cases where gcc can't figure out the size. Judging from my own experience above, this will most likely be useless given that gcc can't determine the size in 70% of the cases, and so cases that the developer could fix easily would be thrown in with 100 other cases that he/she couldn't fix easily or at all.
It's an improvement, but a very minor one. A better improvement would be to make -fstack-protector-all actually do what it claims to do.
-Brad
Kernel release status Posted Oct 1, 2009 14:23 UTC (Thu) by spender (subscriber, #23067) [Link]
Two more quick things:
I recently noticed:
Which looks eerily similar to this (from July):
Mentioned last month here:
And just as a demonstration of the limitations of __builtin_object_size, you show how it would have prevented the perf_counter exploit (which is more a statement of how utterly trivial the vulnerability was). It would indeed have prevented it (even with an older 4.X compiler I believe). However, making one small change of having perf_copy_attr() become a global function, and __builtin_object_size suddenly becomes useless. There's tons of real kernel code like that to be found.
-Brad
Kernel release status Posted Oct 1, 2009 14:38 UTC (Thu) by spender (subscriber, #23067) [Link]
Also just noticed this: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-...
It's a useful change -- I hope we see more kernel-wide changes like this in the future.
With some extra compiler flags you could then go through and remove the (now) useless < 0 checks.
-Brad
Kernel release status Posted Sep 27, 2009 10:49 UTC (Sun) by nix (subscriber, #2304) [Link]
That's interesting. I'd expect the behaviour you see with only
CONFIG_CC_STACKPROTECTOR enabled, as this only protects functions containing large char arrays (not arrays of other types, not aggregates, not pointers to them). But _ALL should change this, and the presence of the aggregate 'struct perf_counter_attr attr' should then enable guard generation for sys_perf_counter_open(). Yet it doesn't.
I wonder if there's something about SYSCALL_DEFINE*() that is blocking
Kernel release status Posted Sep 27, 2009 12:52 UTC (Sun) by spender (subscriber, #23067) [Link]
CONFIG_CC_STACKPROTECTOR cannot be enabled alone anymore. See the link I pasted above where Ingo was talking about stackprotector_all being on by default because otherwise it wouldn't have stopped vmsplice (which it had nothing to do with).
I don't believe it has to do with SYSCALL_DEFINE, because I've found other syscalls that were protected (disassemble kernel/sys.o for example and you'll see that sys_prctl is protected).
But yes, there is definitely an important weakness/vulnerability in the stackprotector implementation (which must have been silently fixed in a newer gcc perhaps? for Ingo to have gotten his results). Nevertheless, this makes it quite a poor protection, and some mention should be made in the configure help about the ambiguous protection it provides depending on your compiler version (and give a recommended minimum compiler version that does what the feature actually claims).
-Brad
|
|||||||||||||