Weekly Edition
Return to the Kernel page
| |||||||||||||
Another old security problem
In August, a longstanding kernel security hole related to overflowing the
stack area was closed. But
it turns out there are other problems in this area, at least one of which
has been known about since late last year. Fixes are in the
works, but it's hard not to wonder if we are not handling security issues as
well as we should be.
Once again, the problem was reported by Brad Spengler, who posted a short program demonstrating how easily things can be made to go wrong. The program allocates a single 128KB array, which is filled as a long C string. Then, an array of over 24,000 char * pointers is allocated, with each entry pointing to the large string. The final step is to call execv(), using this array as the arguments to the program to be run. In other words, the exploit is telling the kernel to run a program with as many huge arguments as it can. Once upon a time, the kernel had a limit on the maximum number of pages which could be used by a new program's arguments. This limit would have prevented any problems resulting from the sort of abuse shown by Brad's program, but it was removed for 2.6.23; it seems that any sort of limit made life difficult for Google. In its place, a new check was put in which looks like this (from fs/exec.c):
/*
* Limit to 1/4-th the stack size for the argv+env strings.
* This ensures that:
* - the remaining binfmt code will not run out of stack space,
* - the program will have a reasonable amount of stack left
* to work from.
*/
rlim = current->signal->rlim;
if (size > ACCESS_ONCE(rlim[RLIMIT_STACK].rlim_cur) / 4) {
put_page(page);
return NULL;
}
The reasoning was clear: if the arguments cannot exceed one quarter of the allowed size for the process's stack, they cannot get completely out of control. It turns out that there's a fundamental flaw in that reasoning: the stack size may well not be subject to a limit at all. In that case, the value of the limit is -1 (all ones, in other words), and the size check becomes meaningless. The end result is that, in some situations, there is no real limit on the amount of stack space which can be consumed by arguments to exec(). And, unfortunately, the consequences are not limited to the offending process. At a minimum, Brad's exploit is able to oops the system once the stack tries to expand too far. He mentioned the possibility of expanding the stack down to address zero - thus reopening the threat of null-pointer exploits - but has not been able to figure out a way to make such exploits work. The copying of all those arguments will, naturally, consume large amounts of system memory; due to another glitch, that memory use is not properly accounted for, so, if the out-of-memory killer is brought in to straighten things out, it will not target the process which is actually causing the problem. And, as if that were not enough, the counting and copying of the argument strings is not preemptible or killable; given that it can run for a very long time, it can be very hard on the performance of the rest of the system. Brad says that he first reported this problem in December, 2009, but got no response. More recently, he sent a note to Kees Cook, who posted a partial fix in response. That fix had some technical problems and was not applied, but Roland McGrath has posted a new set of fixes which gets closer. Roland has taken a minimal approach, not wanting to limit argument sizes more than absolutely necessary. So his patch just ensures that the stack will not grow below the minimum allowed user-space memory address (mmap_min_addr). That check, combined with the guard page added to the stack region by the August fix, should prevent the stack from growing into harmful areas. Roland has also added a preemption point to the argument-copying code to improve interactivity in the rest of the system, and a signal check allowing the process to be killed if necessary. He has not addressed the OOM killer issue, which will need to be fixed separately. Roland's patch seems likely to fix the worst problems, though some commenters feel that it does not go far enough. One assumes that fixes will be headed toward distribution kernels in the near future. But there are a couple of discouraging things to note from this episode:
The problem may indicate a key shortcoming in how Linux kernel development is supported. There are thousands of developers who are funded to spend at least some of their time doing kernel work. Some of those are paid to work in security-related areas like SELinux or AppArmor. But it's not at all clear that anybody is funded simply to make sure that the core kernel is secure. That may make it easier for security problems to slip into the kernel, and it may slow down the response when somebody points out problems in the code. There is a strong (and increasing) economic interest in exploiting security issues in the kernel; perhaps we need to find a way to increase the level of interest in preventing these issues in the first place. (Log in to post comments)
Another old security problem Posted Sep 9, 2010 11:20 UTC (Thu) by error27 (subscriber, #8346) [Link]
It's hard to blame all the rest of us for not doing anything when only Ted knew about this bug for the longest time... *grumble* *grumble*
Brad, I know you read these pages. How did you find this bug in the first place? Was it through a code audit, manual testing or was there a stack trace reported on some bugzilla?
I'm pretty sure that Eugene and Kees track bugzilla entries for security related stack traces. It seems that way from the CVEs they issue.
Another old security problem Posted Sep 9, 2010 19:52 UTC (Thu) by spender (subscriber, #23067) [Link]
During an audit of ways to bypass mmap_min_addr.
-Brad
Nobody working on security Posted Sep 9, 2010 12:11 UTC (Thu) by Trou.fr (subscriber, #26289) [Link]
I find it completely stunning that nobody is working on a full-time paid job to handle the security of the kernel.
Microsoft, having suffered a lot a few years ago, got it right and dedicates a lot of ressources on security, internal testing/fuzzing and all (no trolling indented, just a fact).
RedHat or the Linux Foundation could certainly pay someone to improve kernel security...
Nobody working on security Posted Sep 14, 2010 13:50 UTC (Tue) by kbad (subscriber, #61983) [Link]
Like Eugene Teo? :P
Another old security problem Posted Sep 9, 2010 15:30 UTC (Thu) by alonz (subscriber, #815) [Link] Unfortunately, several high-profile kernel developers have shown a distinct antipathy towards anyt issues raised by “crack-addled security people”.Which attitude serves quite well to reduce community involvement of anyone with security awareness.
Another old security problem Posted Sep 9, 2010 15:53 UTC (Thu) by dgm (subscriber, #49227) [Link]
I think it's more like those developers have shown antipathy towards some _people_ in the "security circus", more than towards the _issues_ themselves. Linus himself being the archetypal one, he agrees that most kernel bugs can have security implications.
Another old security problem Posted Sep 9, 2010 18:05 UTC (Thu) by clugstj (subscriber, #4020) [Link]
Tried the exploit on my machine. The DOS aspect is bad, but doesn't completely lock up the machine. Of course you could run multiple instances.
What's wrong with the obvious workaround of setting a stack size limit? The exploit program then outputs a boring "execve failed" message and nothing bad happens.
Another old security problem Posted Sep 9, 2010 20:30 UTC (Thu) by martinfick (subscriber, #4455) [Link]
"What's wrong with the obvious workaround of setting a stack size limit? The exploit program then outputs a boring "execve failed" message and nothing bad happens."
I suspect that the obvious problem is the limit. Who wants arbitrary limits that are not based on available resources and that affect potentially valid use cases (non exploits)? As for the nothing bad happens, did you forget that the program (and now likely others that should) didn't run? By many people's standards, that would be something bad happening. :(
valid? Posted Sep 9, 2010 22:38 UTC (Thu) by marcH (subscriber, #57642) [Link]
> Who wants arbitrary limits that are not based on available resources and that affect potentially valid use cases (non exploits)?
Calling a megabytes argv a "valid use case" is a bit of a stretch.
valid? Posted Sep 9, 2010 23:02 UTC (Thu) by bronson (subscriber, #4806) [Link]
I totally disagree. I love no longer needing to worry about find and xargs all the time. Does a dir have tens of thousands of files? Who cares! Just glob away, it just works.
A gigabytes argv would be a stretch. But megabytes? That seems pretty reasonable to me.
valid? Posted Sep 15, 2010 0:51 UTC (Wed) by roelofs (guest, #2599) [Link] A gigabytes argv would be a stretch. But megabytes? That seems pretty reasonable to me.Absolutely. A minor side project of mine involves the generation of 35000 time-series images per year, each with a name of the form "fubar-XX-yyyymmdd-hhmm-UTC.png". As a same-dir glob, that works out to just over a megabyte; add a "yyyy/" directory prefix and multiple years, and you're easily into the 10MB range. Increase the time resolution by a factor of 3 to 5, and you're well on your way to 100MB. (And yes, it's very cool to watch a full-year sequence animate, particularly on a fast machine; a 5- or 10-year sequence would be even better, assuming I could hit 60fps on the decode.) Of course, at some point it becomes a database-driven custom app, but 10MB command lines are not out of the question with the trivial hack I have so far. Greg
valid? Posted Sep 9, 2010 23:04 UTC (Thu) by martinfick (subscriber, #4455) [Link]
Right, and who would ever need more than 640K of RAM? Let's constrain the arbitrary limit thinking for DOS(windows) users and devs, not for linux. ;)
valid? Posted Sep 13, 2010 19:18 UTC (Mon) by nix (subscriber, #2304) [Link] Well, actually, stupid arbitrary limits have long been part of the Unix experience. They're part that GNU set itself against, and I'm glad to say that it hasn't been part of the Linux experience heretofore, and Linux is all the better for it.
valid? Posted Sep 13, 2010 19:25 UTC (Mon) by martinfick (subscriber, #4455) [Link]
Very true. I really should used "legacy OS" instead of "DOS/Windows" in my complaint.
Another old security problem Posted Sep 9, 2010 23:23 UTC (Thu) by gregkh (subscriber, #8) [Link]
> Brad says that he first reported this problem in December, 2009,
> but got no response.
Was this sent to the security contact for the kernel? I just searched
If it was not sent there, where was it sent?
Another old security problem Posted Sep 10, 2010 0:00 UTC (Fri) by spender (subscriber, #23067) [Link]
The comment in 64bit_dos.c says where I sent it and when. Apparently that's not enough, so here are the full mails. The context was I was already working with Ted on the ext4 bug, and since he was being responsive, chose to share some additional vulnerabilities with him. You'll note my reply was within 2 hours of the response I received. With the general upstream attitude and handling of security bugs, on principle I don't email vendor-sec or security@.
I hope it's not a complaint you're making, as every time we discover one of these vulnerabilities it's fixed within grsecurity (like this 64kb heap infoleak I actually reported and wasn't credited for:
-Brad
2694 r + Dec 11 12:12PM tytso@mit.edu Re: ext4 bug
X-Spam-ASN: AS14742 69.25.192.0/20
On Fri, Dec 11, 2009 at 01:51:32PM -0500, Brad Spengler wrote:
Sure, send them my way!
- Ted
842 sF Dec 11 11:57AM To tytso@mit.ed ext4 bug
To: tytso@mit.edu
[-- PGP output follows (current time: Thu 09 Sep 2010 07:33:02 PM EDT) --]
[-- The following data is signed --]
> Sure, send them my way!
Ok, the second issue should be clear with the following PoC:
#include <stdio.h>
#define NUM_ARGS 9000
int main(void)
str = malloc(128 * 1024);
execv("/bin/sh", args);
return 0;
It requires very little memory on the part of the program doing the
first issue is fs/exec.c:shift_arg_pages()
The reason this is possible on 64bit machines is because the resource
-Brad
[-- End of signed data --]
To: tytso@mit.edu
[-- PGP output follows (current time: Thu 09 Sep 2010 07:33:33 PM EDT) --]
[-- The following data is signed --]
Sorry, NUM_ARGS should be set larger -- around 24550 or so, to fill up
-Brad
[-- End of signed data --]
Another old security problem Posted Sep 10, 2010 20:23 UTC (Fri) by gregkh (subscriber, #8) [Link]
> With the general upstream attitude and handling of security bugs, on
> principle I don't email vendor-sec or security@.
That's sad, as security@kernel.org is the only group that will guarantee
> Is it only a problem when the shoe's on the other foot?
No, I was worried that something got reported to security@kernel.org
If you notify random kernel developers, one of whom is not part of the
In the future, it would be great if you could notify security@kernel.org
thanks.
Another old security problem Posted Sep 10, 2010 20:31 UTC (Fri) by Trelane (guest, #56877) [Link] since he was being responsive, chose to share some additional vulnerabilities with him.Mmm, this sounds like you sit on vulnerabilities, choosing to notify people who can fix it upstream if and only if they're "being responsive". Is this accurate? Are you sitting on any now? With the general upstream attitude and handling of security bugs, on principle I don't email vendor-sec or security@.What principle is this that you're acting on?
Another old security problem Posted Sep 10, 2010 21:25 UTC (Fri) by rilder (guest, #59804) [Link]
I am not sure about the unlimited stack size. Most (or all) distro kernels restrict stack size. In my case it is 8 MB which I think is quite sufficient even for large systems.
Are there applications which would *normally* require stack greater than this ?
Another old security problem Posted Sep 10, 2010 23:14 UTC (Fri) by daglwn (subscriber, #65432) [Link]
Absolutely. Many scientific codes regularly exceed this amount of stack.
Another old security problem Posted Sep 11, 2010 0:02 UTC (Sat) by jspaleta (subscriber, #50639) [Link]
I would not call scientific codes... normal.. nor the people who run them.
-jef
Another old security problem Posted Sep 11, 2010 4:29 UTC (Sat) by Trelane (guest, #56877) [Link]
Hey! I represent that remark.
Another old security problem Posted Sep 11, 2010 10:50 UTC (Sat) by rilder (guest, #59804) [Link]
What I implied through that comments is this - in regular desktop environments it won't happen(cat /proc/**/status | grep -i vmstk didn't reveal more than 100-200 KB) . However, higher stack usage may be visible in applications in enterprise or as you said scientific establishments. Though, in such conditions the applications/packages installed on the host remain stable over time and are carefully chosen with lot of testing, so any of these exploits may not find their place there.
Another old security problem Posted Sep 11, 2010 14:01 UTC (Sat) by spender (subscriber, #23067) [Link]
I'm not sure you understand the meaning of 'exploit' or the difference between soft and hard limits.
-Brad
Another old security problem Posted Sep 15, 2010 9:10 UTC (Wed) by nix (subscriber, #2304) [Link]
Rilder made a perfectly good point: applications do not generally need vast stacks (not even Emacs!). Thus it is reasonable to impose a hard limit.
That distros generally impose soft limits instead (for no particularly obvious reason in the case of stack size) is no barrier to imposing a hard limit yourself.
Another old security problem Posted Sep 17, 2010 20:34 UTC (Fri) by oak (guest, #2786) [Link]
> I am not sure about the unlimited stack size. Most (or all) distro kernels restrict stack size. In my case it is 8 MB which I think is quite sufficient even for large systems.
The *default* for *thread* stacks (which are of fixed size instead of automatically growing like main process stack) is commonly set to 8MB.
|
|||||||||||||