LWN: Comments on "Kernel security: beyond bug fixing"

userspace drivers

Cyberax — Fri, 18 Dec 2015 04:29:33 +0000

> There's this thing called "Direct Memory Access" in modern computers. Drivers can use that to access any sort of main memory or other devices in the same bus.
Not anymore: https://en.wikipedia.org/wiki/Input%E2%80%93output_memory...

userspace drivers

Rudd-O — Fri, 18 Dec 2015 04:22:33 +0000

There's this thing called "Direct Memory Access" in modern computers. Drivers can use that to access any sort of main memory or other devices in the same bus.

Thus, while you think your (possibly compromised) network driver is oblivious to your password keystrokes because your connection to this site is SSL, your (possibly compromised) network driver is in fact stealing your keystrokes as you go.

(I say possibly compromised, but with DMA, it's a juicy target for a compromise. There are videos of people doing this sort of thing, by the way. It's not something esoteric.)

Kernel security: beyond bug fixing

PaXTeam — Tue, 24 Nov 2015 16:25:03 +0000

https://pax.grsecurity.net/docs/PaXTeam-H2HC15-RAP-RIP-RO... (there's some small errata in there that i'll fix eventually so better check the doc page for the updated version)

Kernel security: beyond bug fixing

hummassa — Tue, 24 Nov 2015 13:43:46 +0000

I couldn't find the slides. Link, please? :D

von Neumann extractor is not useful

cladisch — Mon, 16 Nov 2015 05:55:03 +0000

0101
(You cannot use the same bit for two decisions; that would break the output, too.)

von Neumann extractor is not useful

kevinm — Mon, 16 Nov 2015 04:10:16 +0000

How exactly do you get two consecutive 01 transitions without an intervening 10 transition?

Kernel security: beyond bug fixing

robbe — Wed, 11 Nov 2015 21:20:36 +0000

I don’t think TeX being largely bug-free owns much to its implementation language.

A better explanation is that it’s in rigid bugfix-only mode for more than 20 years now.

Kernel security: beyond bug fixing

PaXTeam — Tue, 10 Nov 2015 17:29:53 +0000

there's a whole bunch of public research on control flow integrity, some of which also addresses protecting the return address despite its being writable by the attacker. i also recently presented my ideas on this at H2HC, see the slides for more details. in short, all these defenses boil down to restricting the 'address of my choosing' to the point that at least privilege escalation is no longer possible.

Kernel security: beyond bug fixing

hummassa — Tue, 10 Nov 2015 16:39:27 +0000

Care to elaborate? If any instruction can write to the return stack, it's always possible to point the return address to an address of my choosing, and that way calling anything the current function's security context has the right to call. Which is mostly everything, because proper security contexts are a pain in the arse.

Kernel security: beyond bug fixing

xman — Mon, 09 Nov 2015 02:16:31 +0000

> Because it really seems possible to write a bad program in any language. And some people seem pretty good at that, whatever the programming environment cleverness.

While undeniably people are great at writing bad code, there are languages/interfaces/apis/designs/whatever that are less error prone than others, and make it easier to see and correct bugs once you find them. Sparse demonstrates that even relatively subtle enhancements to the expressiveness of the kernel's code, you can significantly reduce the overhead lost to bugs.

That allows for a much, much more targeted approach to addressing security.

Kernel security: beyond bug fixing

xman — Mon, 09 Nov 2015 02:10:36 +0000

It may seem like developers are insane about this, but when you contemplate just how much performance is left on the table with our security measures and abstractions right now, there is actually a LOT developers already have sacrificed on the pillars of performance.

With systems programming in particular, inefficiency itself often leads to its own bugs and security compromises farther up the stack. Developers *and* end users naturally route around inconvenient security systems and abstractions.

Heck, we can have drastically improved security and privacy on the Internet right now, if we're just willing to absorb a 5x increase in latency and decrease in throughput (which, if you think about it, we had to suffer with right now), but hardly anyone is willing to make that compromise.

At a higher level, the whole "remember my credit card" feature is an exercise in forgoing the minimal protections of an at least somewhat random and monitored credit card number that they carry with them everywhere for protection from what is almost always not-at-all-random and trivially crackable, not terribly well monitored, memorized password. Ask anyone who works in e-commerce how much more money they make with that feature.

userspace drivers

JanC_ — Sun, 08 Nov 2015 12:43:19 +0000

I don't see how those binary drivers being in userspace would help against an NSA backdoor at all (especially in case of e.g. network drivers)?

And I think you are wrong in case of most Broadcom drivers, which are open source but have to upload a closed source firmware into the network hardware, because they don't have a closed source firmware in ROM/flash like some others do. Both uploaded & saved in ROM/flash firmwares could contain a backdoor, so kernel vs. userspace doesn't even come into play there.

Kernel security: beyond bug fixing

geek — Sat, 07 Nov 2015 18:06:34 +0000

Well, TEX is written in Pascal and is widely held to be the best example of a large, widely used program with NO bugs.

And if it takes Ken Thompson and Dennis Ritchie to write large bug-free programs in C, um, how many programmers like that are there?

Kernel security: beyond bug fixing

PaXTeam — Fri, 06 Nov 2015 19:56:48 +0000

that paper is useless, don't believe anything in it. like i said, go try the code yourself, especially because PCID support was added in 2013 only whereas the paper is from research done in 2010 or so.

Kernel security: beyond bug fixing

patrick_g — Fri, 06 Nov 2015 19:25:28 +0000

> how about you actually try it out instead of speculating about it?

No need to try. There is a usenix paper with perf comparisons here => https://www.usenix.org/system/files/conference/usenixsecu...

The paper is about kGuard but they do perf tests against vanilla and PaX.
For latency in syscalls (in microseconds) they wrote :

> The PaX-protected kernel exhibits a latency ranging between 5.6% and 257% (average 84.5%) on the x86, whereas on x86-64, the latency overhead ranges between 19% and 531% (average 172.2%). Additionally, (..) overhead for process creation (in both architectures) lies between 8.1% to 56.3%.

For sockets and pipes bandwith degradation agains vanilla they wrote :

> PaX’s overhead lies between 19.9% – 58.8% on x86 (average 37%),and 21.7% – 78% on x86-64 (average 42.8%).

But the slowdown is much less noticeable on macro benchmarks. For instance the test to build a vanilla kernel :

> On the x86, the PaX-protected kernel incurs a 1.26% run-time overhead, while on the x86-64 the overhead is 2.89%.

And sql-bench slowdown agains vanilla :

> PaX lies between 1.16% (x86) and 2.67% (x86-64).

Kernel security: beyond legacy

igodard — Fri, 06 Nov 2015 18:53:10 +0000

I am the chief architect of the Mill CPU architecture, whose threat-mitigation aspects were mentioned several times in the comments here. I'd welcome an invitation to meet, formally or informally, with the Linux security community. While it is true that the Mill is paperware as yet, it is a commercial product en route to hardware. There's no better time than now to explore what Linux could be, and should be, when given a clean-slate legacy-free platform underneath. Once the hardware is frozen it will be too late for your concerns - and ideas - to be reflected. You can reach me as ivan at millcomputing dt com.

Kernel security: beyond bug fixing

kees — Fri, 06 Nov 2015 18:50:33 +0000

Yeah, I'm glad to see that working. For people that want to see what their kernels protect against, check out CONFIG_LKDTM, which creates /sys/kernel/debug/provoke-crash/DIRECT. Just echo into it to test various things (like EXEC_USERSPACE to check your SMEP or PXN):

# cat /sys/kernel/debug/provoke-crash/DIRECT
Available crash types:
PANIC
BUG
WARNING
EXCEPTION
LOOP
OVERFLOW
CORRUPT_STACK
UNALIGNED_LOAD_STORE_WRITE
OVERWRITE_ALLOCATION
WRITE_AFTER_FREE
SOFTLOCKUP
HARDLOCKUP
SPINLOCKUP
HUNG_TASK
EXEC_DATA
EXEC_STACK
EXEC_KMALLOC
EXEC_VMALLOC
EXEC_USERSPACE
ACCESS_USERSPACE
WRITE_RO
WRITE_KERN
# echo EXEC_USERSPACE > /sys/kernel/debug/provoke-crash/DIRECT
[2594952.708824] lkdtm: Performing direct entry EXEC_USERSPACE
[2594952.708852] lkdtm: attempting ok execution at ffffffffad5b2422
[2594952.708878] lkdtm: attempting bad execution at 00007f739d328000
[2594952.708907] unable to execute userspace code (SMEP?) (uid: 0)
[2594952.708920] BUG: unable to handle kernel paging request at 00007f739d328000
[2594952.708939] IP: [<00007f739d328000>] 0x7f739d328000
[2594952.708958] PGD 254a3f067 PUD 2732b0067 PMD 255bba067 PTE 248510067
[2594952.708981] Oops: 0011 [#1] SMP

Kernel security: beyond bug fixing

PaXTeam — Fri, 06 Nov 2015 18:19:15 +0000

> That would certainly involve a lot of overhead.

how about you actually try it out instead of speculating about it? PaX/UDEREF/PCID/amd64 at your service.

Kernel security: beyond bug fixing

BenHutchings — Fri, 06 Nov 2015 17:42:44 +0000

Linux 4.3 added emulation of PAN on ARMv7 (CONFIG_CPU_SW_DOMAIN_PAN), again based on memory domains. It's even enabled by default. Unfortunately it's not compatible with LPAE.

Kernel security: beyond bug fixing

ploxiln — Fri, 06 Nov 2015 03:38:50 +0000

Indeed, I did mean "if using a single stack" :) Dual stacks are nifty (but as pointed out by another comment not 100% bulletproof).

Kernel security: beyond bug fixing

fest3er — Fri, 06 Nov 2015 03:25:59 +0000

Seems to me that a lot of what is desired cannot be done efficiently or reliably without hardware support. Like 9-bit memory where that extra bit determines whether or not the other 8 bits are writable. The best we could do today is to leave unwritable page holes in the address space to limit out-of-bounds writing; but this solution still has the resolution of a crayon. And only 64-bit (with 48 bits usable on x86_64) has enough virtual address space to waste.

Kernel security: beyond bug fixing

ortalo — Thu, 05 Nov 2015 14:56:15 +0000

I do not really think the programming language is such a critical concern, neither the language level (Oz/ML vs Python vs C vs assembly language).
Why? Because it really seems possible to write a bad program in any language. And some people seem pretty good at that, whatever the programming environment cleverness.
And also because some of the best programs that never failed [1] were written in a mixture of assembly language and custom languages, but surely finely crafted by pretty good "coders" (who, btw, probably agree with you with respect to high-order languages safety advantage).

The problem when thinking about our children is the fact that you may neglect to assess what our grand-parents did (right).

[1] Look around Apollo or the shuttle flight system for top-class examples. Sorry for not having more recent examples. *That* is annoying I agree.

Kernel security: beyond bug fixing

PaXTeam — Thu, 05 Nov 2015 12:52:03 +0000

there're other defenses against that.

Kernel security: beyond bug fixing

PaXTeam — Thu, 05 Nov 2015 12:46:48 +0000

it seems to me that you're mixing up defense techniques with exploit techniques, which one are you arguing about now? ;) your original post made a blanket statement about something which is demonstrably false, that's all i wanted to point out. how you can make exploit techniques fail with deterministic or probabilistic methods is irrelevant for that discussion. case in point, my simple example may very well be non-exploitable if the compiler places some of those variables into registers which by definition are not subject to memory corruption, but that of course was not the point of the example.

userspace drivers

Zolko — Thu, 05 Nov 2015 12:37:32 +0000

This performance -vs- speed is also the argument to not have drivers in userspace. Which means that there is kernel code running that is completely unknown to the developers. We are not talking about bugs or security flaws, but built-in backdoors by the manufacturer. For example, Broadcom is a US company, who wants to bet that their closed binary drivers contain backdoors requested by the NSA under gagging legislation ?

What use is there for a firewall if the first line of defense is a traitor ? What use is there for sandboxing if the X driver installs keyloggers and then phones home ?

Talking about kernel security with a monolithical kernel and binary drivers is pointless crap (TM Linus) !!!

Kernel security: beyond bug fixing

hummassa — Thu, 05 Nov 2015 12:28:21 +0000

The only defense against THAT would be "only the CALL instruction can write on the return stack".

Kernel security: beyond bug fixing

renox — Thu, 05 Nov 2015 12:25:03 +0000

> the return address is writable by the process and therefore by the arbitrary write primitive too. whether there're defenses that make it harder or not is orthogonal to your original statement 'you cannot use a buffer overflow to override a return address'.

The 'arbitrary write' can overwrite the return address only if the address of the return address is known, which can be quite difficult if there is randomisation.

Also for the Mill CPU(unfortunately paperware only currently) I think that the separated address stack is managed directly by the CPU, so an 'arbitrary write' cannot overwrite a return address.

Kernel security: beyond bug fixing

PaXTeam — Thu, 05 Nov 2015 11:25:29 +0000

the return address is writable by the process and therefore by the arbitrary write primitive too. whether there're defenses that make it harder or not is orthogonal to your original statement 'you cannot use a buffer overflow to override a return address'.

Kernel security: beyond bug fixing

Wol — Thu, 05 Nov 2015 10:52:02 +0000

> It took decades (one generation?) but even BIOS recently dropped assembly, so there is hope. At least for our children.

Which BIOS is that? I understood that at least one major BIOS was written in Forth, and has been that way for what, all of this century?

Cheers,
Wol

Kernel security: beyond bug fixing

renox — Thu, 05 Nov 2015 10:45:54 +0000

Yes, it's able to write anything anywhere the process can, but it doesn't necessarily makes its possible to overwrite the return address, which makes it harder to exploit the flaw (especially if you have w^x and randomisation).

Kernel security: beyond bug fixing

PaXTeam — Thu, 05 Nov 2015 10:36:21 +0000

void foo(char *in, long s)
{
long *p;
char out[8];
memcpy(out, in, 1024);
*p = s;
}

assume the attacker controls the data behind 'in' and that the memcpy overwrites both 'p' and 's' on the stack, the last line will then be able to write anything anywhere. in short, there are many ways a memory corruption bug can be exploited, overwriting the return address of the current frame is just one and perhaps the most popularized textbook example but by far not the only way. this is the reason why having a proper threat model helps avoiding mistakes in devising defenses.

Kernel security: beyond bug fixing

renox — Thu, 05 Nov 2015 09:31:02 +0000

I'm sorry but I didn't understand how your example would work.

Can you explain it again or do you have a link with an article explaining how it could work?
Thanks.

Kernel security: beyond bug fixing

PaXTeam — Wed, 04 Nov 2015 23:47:05 +0000

why not? you first overwrite a local variable of pointer type then let the rest of the function write through that pointer to overwrite the return address on the other stack.

Kernel security: beyond bug fixing

liam — Wed, 04 Nov 2015 21:36:37 +0000

It would be something you could actually mention at conferences:)
More seriously, it would be very much like the rt branch where the intent is to upstream everything that can be upstreamed. In order to do this they'd need to have a good relationship with upstream.
Frankly, starting with the pax/grsec patches may not be a bad idea, but the work would need to be separated out into the smallest, useful components so as to make upstreaming more likely (I haven't examined there patches, so this work may already be in place).

Kernel security: beyond bug fixing

dlang — Wed, 04 Nov 2015 19:45:04 +0000

isn't the grsecurity paxteam stuff an attempt to run a separate kernel security project?

what would be differetn about what you are proposing?

Kernel security: beyond bug fixing

liam — Wed, 04 Nov 2015 19:39:19 +0000

Well, maybe, but hopefully not:)
My thoughts were that having a place where security was the overriding factor would increase the pool of potential contributors, demonstrate the worth (and cost) of said changes (thereby mitigating concerns about performance/bugs rather than having such concerns stop development prematurely), and, in the meantime, act as the upstream for security related work (the later might be useful for folks interested in running such a kernel just as the rt branch is preferred by audio engineers).

Kernel security: beyond bug fixing

renox — Wed, 04 Nov 2015 12:30:58 +0000

> So if bar could overflow a buffer in its stack frame, or in foo's stack frame, no matter how you arrange them,

"No matter how you arrange them" is not correct: if you have separated variable and address stack, you cannot use a buffer overflow to override a return address.

Kernel security: beyond bug fixing

jezuch — Wed, 04 Nov 2015 12:30:01 +0000

> Regarding conservatism: has any consideration been given to creation of a security tree, in similar fashion to the realtime tree?

And then spend a couple of decades trying to merge it back? :)

Kernel security: beyond bug fixing

ploxiln — Wed, 04 Nov 2015 02:02:48 +0000

Sure, if your hardware supports segmentation or something like it, you don't have to swap the page tables (and invalidate the TLB) and performance isn't too bad.

Kees was suggesting swapping the page tables, for each system call or interrupt, when the hardware does not support something like segmentation. That would certainly involve a lot of overhead.

Kernel security: beyond bug fixing

ploxiln — Wed, 04 Nov 2015 01:54:39 +0000

It's very common for a buffer to be allocated in a parent stack frame and then used in a child function. For example

foo() {
char buf[16];
bar(buf, 16);
...
}

So if bar could overflow a buffer in its stack frame, or in foo's stack frame, no matter how you arrange them, one of them could hit the return address.