True enough [LWN.net]

Which is which

Posted Jul 24, 2009 6:38 UTC (Fri) by man_ls (subscriber, #15091) [Link]

You are right, Brad is well known around here and has exposed a lot of ideas so it's better to qualify.

What I agree with is with full disclosure. Kernel developers seem to think that security issues have to be discussed quietly and in private. And working exploits against released kernels are generally not welcome ("irresponsible" is the word). However, it seems that having security holes exposed gaping in public may have temporary disadvantages like public opinion or insecure versions, but makes developers care more about security in the long run. As always, code speaks louder than words.

Labeling bugs as "DoS", or "potentially exploitable", or not applying any security label (and thus apparently "hiding" the problem), is another contentious point of his. Here I can only say that having some security culture will harm no developer.

Harsh manners by Brad were employed not in publicizing this particular exploit, but in his (and PaXTeam's) long rants here on LWN. That I disagree with.

Which is which

Posted Jul 24, 2009 9:52 UTC (Fri) by dlang (✭ supporter ✭, #313) [Link]

now you are mis-stating the position of the kernel developers.

yes, they are opposed to people releasing exploits against released kernels.

that is because they recognise that it's frequently not trivial for people to change their kernels, so by releasing exploits you are doing the bad guys work for them.

and I don't believe that they deliberately downplay a known exploitable scenerio by classifying it as a DOS, I do believe that they fix many bugs that they believe are only a DOS at the time they fix them that are later determined to be exploitable.

Brad believes that every fix should be examined and it's security impact determined prior to the fix being submitted (that's the only way the commit message could explain the severity of the bug being fixed)

the kernel developers are more interested in fixing more bugs (and probably introducing a few with the new changes) as a better use of their time.

the kernel developers also fear that if they do start categorizing bugs as 'security' fixes or not, it will be worse in the long run because the fixes that are not classed as security fixes are far less likely to get deployed.

Which is which

Posted Jul 25, 2009 0:16 UTC (Sat) by spender (subscriber, #23067) [Link]

BTW as we've mentioned repeatedly before, the only thing we called for in the past was for security-relevant information *known at the time of the commit* to not be omitted from the changelog. It would be silly to hold a fix from being published until its full impact could be determined.

Similar to how static checking is done after the fact, vendors should be employing people trained in security to spot bugs with security implications. The information could be posted on a blog similar to xorl's but with simple summaries of impact included as well. Xorl can push out several quality posts like these a day, and he does it for free in his spare time; there's no reason why a company with 600 million in revenue can't do the same.

-Brad

Which is which

Posted Jul 25, 2009 0:21 UTC (Sat) by spender (subscriber, #23067) [Link]

And I know it's fashionable to hate Microsoft, but I think this blog is very useful and a good example of what Linux vendors could/should collaborate on:
http://blogs.technet.com/srd/default.aspx

-Brad

Speaking with code

Posted Jul 25, 2009 0:27 UTC (Sat) by man_ls (subscriber, #15091) [Link]

yes, they are opposed to people releasing exploits against released kernels. that is because they recognise that it's frequently not trivial for people to change their kernels, so by releasing exploits you are doing the bad guys work for them.

What we have seen here is that, while Brad's sometimes shrill voice has in the past got him nowhere, a working exploit has mobilized a lot of developers to work on several attacked areas. Not working for the bad guys but making the good guys work for us. As I said before code speaks much louder than words.

Speaking with code

Posted Jul 25, 2009 0:56 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]

Brad's shrill criticisms have mostly been about how badly the kernel devs have handled things after the fact. this is the first one that I've seen where he has presented a bug instead of complaining abut how other bugs are handled.

there are a _lot_ of bugs that get reported, some security bugs, most not. and they do get worked, it doesn't take publishing an exploit to get them to jump on it and fix it. it may take telling a developer what you are seeing and how you think it could be exploited (as a security person myself I see a lot of times when I see something that I think is an obvious hole, but it takes me explaining for a while before even other security people see the same thing)

now, if this bug had been presented and explained and the kernel devs blew him off, further steps are nessasary, but as I understand it in this case, the bug was reported, fixes were already in the works when the exploit was released.

Speaking with code

Posted Jul 25, 2009 1:07 UTC (Sat) by man_ls (subscriber, #15091) [Link]

Combining your message with nix's you get precisely to my point: after the exploit was released kernel devs started not just closing that bug, but dealing with whole categories of bugs. As explained by Jon at length in this very weekly edition:

But this exploit suggests that there could be a whole class of related problems in the kernel; there is a definite chance that similar vulnerabilities could be discovered - if, indeed, they have not already been found.

and

All told, the posting of this exploit has served as a sort of wakeup call for the kernel community; it will, with luck, result in the cleaning up of a lot of code and the closing of a number of security problems.

Reporting each bug would probably have served to just close a few tiny bugs, but the exploit made kernel devs think beyond that. Which is exactly what we need.

Speaking with code

Posted Jul 25, 2009 1:07 UTC (Sat) by spender (subscriber, #23067) [Link]

Then you missed my similar exploit in 2007 that introduced NULL ptr dereferences as an exploitable bug class. It was the entire reason mmap_min_addr was introduced in the first place.

-Brad

Speaking with code

Posted Jul 25, 2009 1:36 UTC (Sat) by dlang (✭ supporter ✭, #313) [Link]

but if you pointed out a problem and they implemented a fix, doesn't that indicate that they care and are interested in fixing the problems?

the fact that that fix didn't solve all possible aspects of this problem is unfortunate, but not more than that (you didn't see the implications at that time or you would have made more of a fuss then)

by the way, I do think that there are times when it's necessary to develop exploits, I don't remember exactly which thing, but within the last few months there was some exploit (I think DNS) where there were reports of people looking at the flaw and saying 'they managed to make an exploit of _that_???'

and I do think that if a vendor does not respond in a reasonable time there is a need to shame them (the bad guys aren't going to remain ignorant of the exploit forever, especially if the good guys are talking about it)

however I strenuously disagree with the people who believe that full public disclosure with exploit code should be the first step

Speaking with code

Posted Jul 25, 2009 1:22 UTC (Sat) by spender (subscriber, #23067) [Link]

The only thing that was fixed when the exploit was written was the tun.c bug, which was fixed with no mention of any security impact (understandable since I had opened up yet another bug class as being exploitable). Everything else was fixed either in response to obvious hints/statements in videos I released a week prior to the exploit, or in response to the exploit itself.

Still waiting on anyone from Red Hat/Fedora to report a CVE for the SELinux issue and tell their users how long they've been vulnerable to null ptr dereference kernel exploits because of it.

-Brad

Speaking with code

Posted Jul 25, 2009 12:40 UTC (Sat) by nix (subscriber, #2304) [Link]

I'm curious: since they've said they don't think it's a bug (more an
expected tradeoff: if you run SELinux without useful policy this is what
you get, and no, I don't agree with that either), why would *they* give it
a CVE?

Speaking with code

Posted Jul 25, 2009 12:53 UTC (Sat) by spender (subscriber, #23067) [Link]

Well, perhaps because they asked me to cancel the CVE request I put in (which I still haven't gotten a response from) so that they would submit their own:

Brad Spengler wrote:
>> Let me get back to you on this.
>
> I've contacted the relevant people to request a CVE for the issue, as
> the previous bypass of mmap_min_addr was given a CVE back in 2007; this
> should be no different.

Thanks Brad.

Can you cancel the request? I will assign one (faster), and provide you
with the proper credits in the errata.

-Brad

Which is which

Posted Jul 24, 2009 9:59 UTC (Fri) by nix (subscriber, #2304) [Link]

The problem is that modern systems are so horribly complicated and have a high enough security hole density that I fear that whack-a-mole exploit squashing will never get us to a secure state. Shaming developers only helps to kill exploits in the same way that shaming developers works to eliminate all bugs: it might reduce the density but only to a point, and it doesn't really matter if you have one security hole on your system or six, the bad guys will still get in. And even if the universally-used core of the kernel *does* get all its security bugs squashed, the vast mass of drivers run by only a few people (but which many people run at least one of) will still have holes, as will the vast tower of userspace above it, much of which is run by almost everyone.

So attacking kernel devs for 'covering up' holes is pointless: most security threats are in the layers above it, particularly as most cracks these days are financially-motivated and don't care about getting root so much as spying on you, making you a botnet node, and sending out spam, all of which can be done without root. I don't notice Brad howling about FF even though its security record is much worse than the kernel's of late (but they have a webpage saying which security bugs they fixed! that solves everything! oh wait, it doesn't). You surely can't accuse the kernel devs of *not fixing* security holes when they are pointed out: they just don't make a great song-and-dance about it (it's probably wisest to assume that any bug in something all-powerful like the kernel, or something network-facing like FF, is a potential security hole).

Now kernel bugs *are* dangerous --- it's easier to write a seriously infective worm using a kernel remote root exploit than an FF bug --- but are they worth paying so much more attention to than the higher layers?

Perhaps capabilities would have saved us from the insecurity of modern systems, but it's very non-POSIX and Shapiro has been eaten by Microsoft now so we won't see anything from that until a new caps deity emerges. Certainly any number of paranoid bug-hunters aren't going to solve the problem.

Perhaps writing the higher-level stuff in languages better than C would save us, although that would probably just mean attackers finding some other way to force programs to do unexpected things. That'll happen as long as machines can make choices in response to external input, but we could probably get their density down a long way. Then maybe focusing on the kernel to the exclusion of all else would make sense.

If we must use C, randomly perturbing the ABI on each installed system might help a bit (there's been work on this in the past): but it would be difficult, wouldn't stop all exploits by any means, and would break distribution of binaries.

I have no solutions, but what we're doing now surely isn't going to fix anything. Holding back the tide is very praiseworthy, but in the long run gets you nowhere.

Which is which

Posted Jul 24, 2009 12:28 UTC (Fri) by spender (subscriber, #23067) [Link]

The reason you don't notice me howling about FF is the same reason you don't notice me howling about filesystem code. I talk only about what I know (you should try it sometime).

BTW you are aware that protections implemented in the kernel affect the exploitability of some issues in all software, including FF right?

-Brad

Which is which

Posted Jul 24, 2009 20:37 UTC (Fri) by nix (subscriber, #2304) [Link]

Well, yes, of course, but most attacks on higher-level stuff don't bother
trying to exploit a kernel bug in order to get at, say, Firefox. They
buffer-overrun some image library instead (FF does things like ship its
own copy of that known security-wobbly library libpng, always, because
some patch of theirs was rejected from the upstream tree, sigh). I'm not
sure I've ever heard of userspace network-facing software such as web
browsers being successfully attacked by way of a kernel hole. I suppose it
*could* happen, and perhaps it has, but is it common enough to make the
kernel a more worthwhile thing to fix than the higher-level stuff? It
seems more likely they'd stick to attacking the higher layer, and thus get
an exploit that works no matter what version of the kernel you're running.

(Damn good interview, btw.)

Which is which

Posted Jul 24, 2009 22:36 UTC (Fri) by spender (subscriber, #23067) [Link]

My mention of "kernel protections" referred to NX/ASLR/etc, which are implemented in the kernel ;)

-Brad

Which is which

Posted Jul 25, 2009 12:34 UTC (Sat) by nix (subscriber, #2304) [Link]

Ah, yes, obviously they're worthwhile. Still there seems to be a
depressing frequency of vulns (exploitable even with these protections) in
the higher levels: the days when the kernel had more holes than everything
else put together seem to be behind us.

Killing bugs

Posted Jul 25, 2009 0:37 UTC (Sat) by man_ls (subscriber, #15091) [Link]

Of course we (kernel users) don't want kernel developers to just fix a few exploited bugs as they come up. What we really want (and have got a taste of in this situation) is developers to close gaping holes and eliminate whole categories of errors. People are integrating protections against Brad's clever exploits at all levels; and as it happens the main bug exploited here had been reported by Coverity, so maybe kernel devs will listen to these reports. It might take a few more exploits but Brad has got their attention now.

We all make mistakes; good engineering should prevent known mistakes from happening again, or at least from taking down the whole system with them. Two buffer overflows is one too much. This probably means doing work at several different levels (language, compiler, memory libraries, code checkers, audit tools, security modules), but we will all be better off for the next round of attacks, which may come from less benign sources. This is the fundamental truth which we probably all knew, but had forgotten; we have to be reminded every now and then.

Killing bugs

Posted Jul 27, 2009 12:32 UTC (Mon) by hppnq (guest, #14462) [Link]

Of course we (kernel users) don't want kernel developers to just fix a few exploited bugs as they come up. [ ... ] We all make mistakes; good engineering should prevent known mistakes from happening again, or at least from taking down the whole system with them. [ ... ] This is the fundamental truth which we probably all knew, but had forgotten; we have to be reminded every now and then.

I think the fundamental truths here are: 1) any usable system can be abused, and 2) all resources are limited. Laws of nature.

This then means that, if you care about actual security, it is far more important to do things like kicking pulseaudio off of your precious servers and monitoring what gets run by which user. What we need are not perfectly engineered systems -- of course it helps if developers try a bit -- but fault tolerant systems.

And then, of course, there remains plenty of time and energy to discuss full disclosure, bug class reporting, static code analysis and auditing, and how useful these are or would be. That entire discussion also revolves around laws 1 and 2, and these are also the points that people will either forget or, worse, ridicule.

Oh, and make regular backups. ;-)

Which is which

Posted Jul 24, 2009 20:32 UTC (Fri) by nix (subscriber, #2304) [Link]

Kernel developers seem to think that security issues have to be discussed quietly and in private.

I'm sure I saw Al Viro complaining about vendor-sec's 'discuss it in private and embargo it until the Sun grows cold' attitude... oh yes, here.