LWN: Comments on "Kernel vulnerabilities: old or new?"

Kernel vulnerabilities: Promoting protection

jageorge — Fri, 29 Oct 2010 20:43:02 +0000

The biggest problem in this domain is one of egos. Influential kernel developers who don't care about stuff that they don't own, and security hackers who want to see their names in lights.

In the case of kernel hackers there is a tendency to use excuses which boil down to shooting down solutions which don't have a perfect "model" for resolving an exploit or class of exploits. (AppArmor) Also, kernel hackers tend to hate large patches which actually address a bunch of conceptually separate issues to the point where the aggregate gets blown off. (GrSec)

Security hackers on the other hand will legitimately discover bugs, but don't like Linus's attitude that their presumably security compromising exploit bugs are no more worthy of special recognition than any other kernel bug. Furthermore, there is a tendency to provide "fixes" which simply shift the point of exploitable failure from a current failure to a future or even concurrent failure point. This is sometimes combined with with creating solutions to a bunch of small kernel vulnerabilities in combination with larger frameworks which make them harder to apply to the root cause problems and to audit. There is sometimes a lack of consideration of balancing of performance, scalability, and complexity considerations against quick and dirty or overly complex framework solutions. Finally, there is often a desire to re-invent the wheel rather than understanding and working with existing robustness models.

The Solution (tm):
1. Preview solutions with key kernel developers who are interested in security.
2. Request reviews from other key kernel developers who can help with feedback.
3. Treat different conceptual problems with different patch sets. Debugging, tracing, simple patches, class of exploit preventions, etc.
4. Performance matters.
5. Compatibility matter even more. Breaking X.org, FTP, SSH, procfs, udev, or Apache is not OK. :-)
6. We already have SELinux and AppArmor - don't reinvent the wheel - invent the tire instead.
7. Document your work... When has this been a problem? Why should people care now? What is the design of the solution? Why couldn't it be simpler? Are there benchmarks? What issues were identified in review and how were they addressed?

I really wish that security (and performance) were addressed in a more rigorous fashion, but the best we can hope for is learning how to work together and maybe getting a corporate sponsor. :-)

Kernel vulnerabilities: old or new?

spender — Tue, 26 Oct 2010 21:38:31 +0000

For future reference (since this post seems to be linked to from elsewhere) the CVEs I mentioned in the previous post that Dan Rosenberg was still waiting to have allocated were finally provided yesterday on oss-sec:
http://seclists.org/oss-sec/2010/q4/78

It's 12 new CVEs, bringing the count to 92, 15% higher, from just one guy with grep and a few days in his spare time (and some of the CVEs cover multiple vulnerabilities).

-Brad

Rate of introduction vs removal

man_ls — Mon, 25 Oct 2010 21:28:15 +0000

For starters, how about concocting a different metric which shows what our esteemed editor was trying to discover here: what is the rate of introduction and removal of security holes in the kernel. His own approach for this article was arguably thin. I am sure spender must have a couple of ideas.

Kernel vulnerabilities: old or new?

dany — Mon, 25 Oct 2010 09:01:50 +0000

Yes, Brad's own security articles would be for LWN fresh water (or complete shock therapy :)

Kernel vulnerabilities: old or new?

vonbrand — Sun, 24 Oct 2010 12:09:48 +0000

Sorry, but "widely used" isn't really an argument... the saying here goes "10,000,000,000,000 flies eat sh*t, so should you"

Is it possible to track before 2.6.12?

smurf — Sun, 24 Oct 2010 08:50:47 +0000

Not quite complete, but close enough.

Is it possible to track before 2.6.12?

bfields — Sat, 23 Oct 2010 20:47:40 +0000

Neat-o, thanks. So you can git a repo with complete history something like just:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
cd linux-2.6.git
git remote add -f post-2.4 git://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
git remote add -f pre-2.4 git://git.kernel.org/pub/scm/linux/kernel/git/davej/history.git

cat >.git/info/grafts <<EOF
1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 e7e173af42dbf37b1d946f9ee00219cb3b2bea6a
7a2deb32924142696b8174cdf9b38cd72a11fc96 379a6be1eedb84ae0d476afbc4b4070383681178
EOF

Spender writing for LWN.net

promotion-account — Fri, 22 Oct 2010 19:57:09 +0000

Completely agree. All these released exploits (and the heated discussions about them) made me think more seriously about security of the code I write.

Kernel vulnerabilities: old or new?

tzafrir — Thu, 21 Oct 2010 17:27:53 +0000

Ring 1? Isn't this x86-specific?

What would happen in a VM?

And why are you using a crappy out-of-tree driver anyway? (yeah, I know, I shouldn't be writing that specific comment).

Kernel vulnerabilities: old or new?

ccurtis — Thu, 21 Oct 2010 16:51:19 +0000

I'm speaking from an uninformed position here so I may just be regurgitating marketing I've heard, but it seems like Microsoft has really taken a leadership position in securing the NT kernel.

Their biggest problem (in recent history) has always been in 3rd party drivers, but I hear less and less about these being exploited lately. Have they sufficiently hardened their DDK to prevent the majority of these problems? Is there something about their driver architecture - some microkernel-esque fault isolation perhaps - that prevents wider system compromise?

I don't see how you could prevent a driver from, say, dereferencing a NULL pointer, but it seems like it would be a nice feature to have a lower barrier to entry for device driver writers. Specifically: can the Linux driver model be crafted such that a hardware manufacturer can provide a driver (in source code form or otherwise) that is sloppily written yet not expose the kernel to further breach? Would running drivers in ring1 instead of ring0 achieve this?

Requiring perfect code in order to support a device, while ideal, seems like a long-term losing proposition. One could make the same argument for network protocols or file systems.

Is this a reasonable goal or is it overkill? Is it anything other than a microkernel architecture that's been dismissed out of hand since Linux 0.01? And perhaps philosophically, is it reasonable to even consider killing and respawning a misbehaving driver?

It seems like what I'm really thinking of is a [fixed] kernel API for drivers, but I expect this has already been discussed and rejected many times over. But for example, when descending the drivers/ directory, the 'kmalloc' symbol can be #defined to something invalid, forcing drivers to use ddk_malloc() which has grsec-style checking built-in. (This "kernel API" would be a family of ddk_xxxx() calls.)

Alternately, for code paths where it's deemed more important to be fast than safe (scheduler, SL*B allocators, other "core" code), there can be an audited/ tree as the only place where code has access to versions of the system calls that aren't encumbered by security restrictions. (The ddk_xxxx() calls would naturally live here.)

Or is this just not seen as a problem?

Kernel vulnerabilities: old or new?

nix — Thu, 21 Oct 2010 12:29:23 +0000

Strongly agreed. The more people learn about how to apply Spengler-style paranoia to the code they themselves write, the better our security gets.

Kernel vulnerabilities: old or new?

mjthayer — Thu, 21 Oct 2010 11:34:57 +0000

> > Additionally, many of the protections are done in a way that mainline folks do not like for various technical reasons.

> Would you be able elaborate on that, or provide a couple of pointers? (Sorry for lazily asking rather than researching myself!)

Replying to myself - probably http://userweb.kernel.org/~jmorris/lss2010_slides/out-of-... will be one of the pointers.

Kernel vulnerabilities: old or new?

mjthayer — Thu, 21 Oct 2010 11:30:00 +0000

> Additionally, many of the protections are done in a way that mainline folks do not like for various technical reasons.

Would you be able elaborate on that, or provide a couple of pointers? (Sorry for lazily asking rather than researching myself!)

Kernel vulnerabilities: old or new?

patrick_g — Thu, 21 Oct 2010 09:15:56 +0000

>>> Another possibility is for a company with a strong involvement in Linux to hire someone

Why not the Linux foundation ? Perhaps they can sponsors a few security experts ?
There is a "Linux fellow program" here => https://www.linuxfoundation.org/programs/developer/fellowship

Kernel vulnerabilities: old or new?

jamesmrh — Wed, 20 Oct 2010 23:42:01 +0000

Kees listened to what upstream folk said, but was lead in circles by them, and ultimately his Yama code was vetoed.

I'm not sure what the solution is.

Users can request the protections be added to their distros, which will at least get them better protected, and possibly help make a stronger case for upstream inclusion.

Another possibility is for a company with a strong involvement in Linux to hire someone with the traits a & b above, to act as a system-wide security coordinator / advocate. i.e. make it someone's paid job to work on kernel hardening.

Kernel vulnerabilities: old or new?

clugstj — Wed, 20 Oct 2010 21:14:05 +0000

How about
d) are not so hardheaded that they will actually listen to what other people say.

Kernel vulnerabilities: old or new?

spender — Wed, 20 Oct 2010 18:36:56 +0000

We don't necessarily prevent the bugs themselves, though it is true we sometimes have fixes for things that aren't fixed in mainline. I imagine what you mean is that exploitation of the bugs is prevented in grsec kernels. That's been true in many cases at different levels.

Often, we turn vulnerabilities that can be exploited for privilege elevation into DoSes. The reason is that we've detected something that shouldn't have happened that we know can be abused, so it's not safe to allow the system to continue running.

Some vulnerabilities are prevented completely, like those of the form:
buf = kmalloc(attacker_controlled_size * something, GFP_ANYTHING);
instead of overflowing and allowing exploitation, we detect the overflow and cause buf to be set to NULL. We also add additional bounds checking to copy_*_user calls, both against the kernel stack and kernel heap.

Other features of grsecurity prevent exploitation of certain bugs by reducing attack surface. For example, GRKERNSEC_MODHARDEN prevents attackers from auto-loading an unused kernel module just so that it can be exploited. This reduces the risk from numerous obscure driver vulnerabilities (like the recent RDS vulnerability).

Finally, a number of features of grsecurity and PaX work together to make exploitation of the remaining vulnerabilities quite difficult in many cases. Symbol hiding, PAX_KERNEXEC, and PAX_UDEREF all work together to make the kernel a hostile environment to attackers. In general, memory corruption exploits should require an arbitrary read bug in addition to whatever vulnerability is being exploited. Any exploits have to be written considerably different as the ring-0 payload to execute can no longer be located in userland (as it is with every public exploit in existence).

Since we have a couple more people now writing exploits (Kees, Tavis, Dan) I'd actually like to see a memory corruption exploit that works against a grsec kernel with a writeup about the process involved.

-Brad

Kernel vulnerabilities: old or new?

kees — Wed, 20 Oct 2010 18:35:03 +0000

Not only did those protections stop many of the vulnerabilities from being exploited, some of the vulnerabilities were discovered _because_ of the protections (see the history on how CVE-2010-2955 got reported).

The main difficulty with getting PaX and grsec protections into mainline is that the patchset is hard to separate into individual pieces. Additionally, many of the protections are done in a way that mainline folks do not like for various technical reasons. I suspect the only way to get these protections into mainline is to have someone (or better yet, a group of people) fighting for them that:
a) understands the code
b) understands why it is important
c) have the time to advocate for it and revise patches until they get in

So far, no one has really had all 3. Plenty of people get "b" (even if they don't have "a"), and "c" is insanely frustrating given the attitudes of the mainline maintainers towards security.

Kernel vulnerabilities: old or new?

bronson — Wed, 20 Oct 2010 17:16:16 +0000

Agreed. They might be impassioned and even inflammatory, but I bet they'd be awesome reading.

What about 2.4 security

spender — Wed, 20 Oct 2010 15:01:05 +0000

Between September of last year and September of this year, these were all the bugs fixed:
http://www.kernel.org/pub/linux/kernel/v2.4/ChangeLog-2.4...
http://www.kernel.org/pub/linux/kernel/v2.4/ChangeLog-2.4...
http://www.kernel.org/pub/linux/kernel/v2.4/ChangeLog-2.4...
http://www.kernel.org/pub/linux/kernel/v2.4/ChangeLog-2.4...
http://www.kernel.org/pub/linux/kernel/v2.4/ChangeLog-2.4...

I count about 24 vulnerabilities fixed, counting each line involving a vulnerability in the changelog as a separate vulnerability (even if multiple things were fixed at once in the same file, see: netlink). I also counted some of the infoleaks that weren't worthy of CVEs in 2.6. At least half of the vulnerabilities were information leaks.

Most of these vulnerabilities were also present in 2.6 kernel (and deemed important enough to check how far the vulns went back so that they could be backported). So it too is a biased dataset, but it's also a much more stable codebase that avoids all the issues introduced in 2.6.

It's not possible to run a 2.4 kernel with any modern distro I'm aware of (you'll get a "kernel too old" death on boot, even for running old 2.6 kernels). I imagine it would have to be some custom supported/maintained distro. The users would have to be updating their userland as well if they wanted to match their security intent in upgrading to newer versions of 2.4 kernels, so it's unclear who would be bothering to do that and how they're doing it.

-Brad

Kernel vulnerabilities: old or new?

ballombe — Wed, 20 Oct 2010 09:19:26 +0000

I am afraid the dataset is too limited to be useful.
What would be informative is to take an old kernel (says 2.6.18) and to
look at the report date of all reported vulnerabilities that affect it.
This way we would know how much time it takes to find vulnerabilities.

Kernel vulnerabilities: old or new?

nix — Wed, 20 Oct 2010 08:36:17 +0000

Quite so. You'd get the same results if there was one very numerous class of security holes that almost always took at least ten years to track down, and other less numerous classes that were normally found faster. The same results, but the rate of hole introduction would be going *up* since 2.6.12 because the rate of kernel growth shot up since the introduction of git: we just haven't found any of those bugs yet.

(for the record this is a rather unlikely scenario -- 'digital kuru' if you will -- but it is a valid interpretation of the data, I think.)

Kernel vulnerabilities: old or new?

intgr — Wed, 20 Oct 2010 08:33:19 +0000

According to several posts I've seen here, some of these bugs were already prevented by grsec kernels prior to their discovery in mainline.

Is this true?

Why aren't relevant grsec patches moving to mainline? It can't be just that nobody bothers to submit them to mainline. One would think that even if grsec developers aren't interested, then other security-conscious devs would review and cherry-pick patches from the grsec tree and submit them for the mainline. But I get the impression that no cooperation is happening at all.

Kernel vulnerabilities: old or new?

mjthayer — Wed, 20 Oct 2010 08:28:36 +0000

Brad, ever thought of submitting a couple of guest articles to the security section of LWN?

Is it possible to track before 2.6.12?

smurf — Wed, 20 Oct 2010 08:17:07 +0000

Note that you can patch up the GIT history to link these three archives into one coherent whole.
$ cat .git/info/grafts
1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 e7e173af42dbf37b1d946f9ee00219cb3b2bea6a
7a2deb32924142696b8174cdf9b38cd72a11fc96 379a6be1eedb84ae0d476afbc4b4070383681178
$

Incidentally, the first history doesn't contain the rest of the v2.4 data (the part that came after the v2.5 branchpoint).

Anyway, does anybody remember what happened to the history of 2.4.25? Pre-patch 1 thru 3 are missing from kernel.org.

What about 2.4 security

Alterego — Wed, 20 Oct 2010 07:25:30 +0000

I am wondering about the 2.4 kernel, which has been maintained for a very long time.

1/ How many security vulnerabilties were discovered and fixed during deep maintenance period? when?

2/ who is (was) using it?

I hope our editor can enlight us on this topic too.

Kernel vulnerabilities: old or new?

error27 — Wed, 20 Oct 2010 05:18:32 +0000

I've used Smatch to fix a bunch of buffer range checking bugs (at least 50 since January). But I didn't get any CVEs. Actually most of them weren't exploitable.

But yeah. I don't think the fixes on this list were found with static analysis tools. Vasiliy Kulikov just posted a list of eight information leaks and people assumed he used a tool but he did it with grep. It seems to me like you could find a bunch of information leaks automatically but no one has done that yet.

Kernel vulnerabilities: old or new?

martinfick — Wed, 20 Oct 2010 03:27:04 +0000

> So one way to phrase your criticism is: the sample chosen (of kernel vulnerabilities found this year) is biased towards older bugs, because it takes time for kernel bugs to be found.

No, my criticism is that the sample is biased towards the known bugs. It is not very useful to attempt to compare the known to the unknown. There could be an infinite amount of bugs being introduced, we have no idea. Each "fix" could even introduce more bugs than it fixes. The discovery rate is unrelated to the introductory rate! Looking at a subset of the possible bugs tells you nothing conclusive about the total except that the total includes the subset. It's like trying to determine when we will have mastered intergalactic space travel from the rate of scientific papers published during the 20th century. :)

Kernel vulnerabilities: old or new?

spender — Wed, 20 Oct 2010 02:18:02 +0000

I think it's just the same case as what's always improved Linux security: outside contributors actually auditing the code for security issues, which apparently isn't done otherwise. Other than coccinelle (http://coccinelle.lip6.fr/impact_linux.php) which isn't exactly new (it's been used for patches since 2006) I don't know what "new static analysis tools" are being used. Dan Rosenberg, who's discovered a ridiculous number of vulnerabilities this year, has been using grep. Tavis' auditing has been manual AFAIK.

It's worse than just 80 for this year. You called 80 the "known issues", but really it's just the problems for which CVEs have been allocated. Dan Rosenberg had a huge collection of information leak vulnerabilities that still haven't received CVEs despite numerous requests. If you follow oss-sec (I know Kees follows it, but this is directed to the other readers), you'll notice Eugene often sends kernel vulnerability information to the list with a message of "I'm not requesting a CVE for this as it does not affect any Red Hat-supported kernels". Often this means the vulnerability was introduced in newer kernels. This mostly explains why there are suspiciously no CVEs for recent kernels: simply that nobody bothers to ask for CVEs for recently introduced vulnerabilities.

And again it demonstrates the problem of silent vulnerability fixing. The editor is left to gauge vulnerability risk by allocated CVEs -- a biased metric which ultimately biases any conclusions derived purely from the dataset. How different would it look if Dan Rosenberg didn't mail repeatedly to make sure the vulnerabilities he discovered were assigned CVEs? It might end up like my http://grsecurity.net/~spender/64bit_dos.c which after over 2 months now still has no CVEs or committed fixes.

The sad thing is nobody has any clue how bad it really is. The scary thing is we're finding evidence that blackhats have developed exploits for vulns months or years ahead of the reactive vuln fixing going on. Look at how Microsoft changed strategy years ago in response to these systemic problems, or how Adobe recently has been taking some defensive steps (hardening their JIT and sandboxing the next version of Reader). Linux is in dire need of a similar change, but it's unlikely to ever occur in mainline unless starts with the leadership.

-Brad

Kernel vulnerabilities: old or new?

bfields — Wed, 20 Oct 2010 01:49:49 +0000

Could it not simply mean that we are doing a really poor job of finding the new bugs (and have thus not found them yet)?

So, say we want to determine the age distribution of kernel bugs. Given infinite time, we could find every kernel bug, then determine the age of each bug we find, and get an exact answer to our question. But of course we can only afford to investigate a sample of the kernel bugs.

So one way to phrase your criticism is: the sample chosen (of kernel vulnerabilities found this year) is biased towards older bugs, because it takes time for kernel bugs to be found.

So the problem is to find a sample that we think is more representative.

One approach might be to look just at bugs discovered by one new technique. If we believe the technique is sufficiently novel that very few of the bugs it discovers would have been discovered without it, then we could hope that set of bugs it discovers might have the characteristics of a random sample. (And perhaps we could test the novelty of the technique by looking through previously discovered bugs to see if any of them would have been caught by the new technique.)

Is it possible to track before 2.6.12?

bfields — Wed, 20 Oct 2010 01:33:25 +0000

Two that I know of: from 2.4.0 to 2.6.12-rc2:

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git

Before 2.4.0:

git://git.kernel.org/pub/scm/linux/kernel/git/davej/history.git

Kernel vulnerabilities: old or new?

BenHutchings — Wed, 20 Oct 2010 00:56:38 +0000

Some more for you, all fixed post-2.6.32:

drm/i915: Sanity check pread/pwrite (CVE-2010-2962)
sctp: Fix out-of-bounds reading in sctp_asoc_get_hmac() (CVE-2010-3705)
RDS sockets: remove unsafe kmap_atomic optimization (CVE-2010-3904)
v4l: disable dangerous buggy compat function (CVE-2010-2963)
sctp: Do not reset the packet during sctp_packet_config() (CVE-2010-3432)
x86-64, compat: Retruncate rax after ia32 syscall entry tracing; Test %rax for the syscall number, not %eax (CVE-2010-3301)
irda: Correctly clean up self->ias_obj on irda_bind() failure. (CVE-2010-2954)
GFS2: Fix up system xattrs (CVE-2010-2525)
tipc: Fix oops on send prior to entering networked mode (CVE-2010-1187)
KGDB: don't needlessly skip PAGE_USER test for Fsl booke (CVE-2010-1446)
futex: Handle futex value corruption gracefully. (CVE-2010-0623)
KVM: PIT: control word is write-only (CVE-2010-0309)
KVM: Add IOPL/CPL checks to emulator, to prevent privilege escalation within a guest. (CVE-2010-0298, CVE-2010-0306)
untangle the do_mremap() mess (CVE-2010-0291)
kernel/signal.c: fix kernel information leak with print-fatal-signals=1 (CVE-2010-0003)
netfilter: ebtables: enforce CAP_NET_ADMIN (CVE-2010-0007)
ipv6: skb_dst() can be NULL in ipv6_hop_jumbo(). (CVE-2010-0006)

Kernel vulnerabilities: old or new?

timn — Wed, 20 Oct 2010 00:46:42 +0000

It would be nice to see a graph extend by a second column per revision denoting the number that were fixed in that particular release. Just from the data it seems that 2.6.36 has fixed quite a few problems. It would be nice to see if the amount of fixes grew over time.
Have there been more efforts recently to look out for security related bugs?

Is it possible to track before 2.6.12?

smoogen — Wed, 20 Oct 2010 00:44:40 +0000

I was wondering if there is a tree that could help the editor look further back than 2.6.12... this is the 3rd article where I have seen a large number of things (be they changes, bugs, etc) falling to the 5 year boundary. It would be interesting if we could track things down further... if for just 'oh so this came from 2.2.. wow' factor.

Kernel vulnerabilities: old or new?

kees — Wed, 20 Oct 2010 00:30:12 +0000

Based on http://lwn.net/Articles/297366/ it would seem that more and more vulnerabilities are being reported. Using Mitre, the numbers for 2008 are 76, 2009 are 112, and 2010 is at 80 with 3 months to go. And that is just the known issues.

It may be a while before the tide turns. Since there are so many vulnerabilities, and their exposure frequency is increasing, it would make sense for the kernel to adopt some more defensive behaviors to make vulnerability exploitation harder. As a start, if we assume an unprivileged user will eventually have arbitrary write access to kernel memory, we can make choices about what is important to defend against within the kernel itself.

Kernel vulnerabilities: old or new?

martinfick — Wed, 20 Oct 2010 00:24:34 +0000

> There may some comfort in knowing that a large proportion of 2010's known security vulnerabilities are not a product of 2010's development.

Hmm, I am not sure that is comforting. Could it not simply mean that we are doing a really poor job of finding the new bugs (and have thus not found them yet)?

This whole article while interesting, just seems a little bit silly. It really is impossible to make any valuable conclusions whatsoever about the rate of introduction vs fixing from any of this data since it ignores all the potentially unknown/unfixed bugs. The only thing that could be concluded is that many bugs lurk for a LONG time unfixed.