LWN: Comments on "Ripples from Stack Clash"

Increasingly hard to see how Brad Spengler is/was in the wrong

Wol — Thu, 06 Jul 2017 17:18:49 +0000

> Linus is not much better - which is especially bad because some people think the success of Linux is because of his style and LKML culture - rather than despite of it.

I think you'll find Linus is a *lot* better.

Yes he can have a potty-mouth. But he rarely uses it, and mostly when the other guy has been demonstrably stupid. Also the other guy is usually well-known to Linus.

There's a big difference between a bully picking on everyone, and a playground scrap between friends. Linus is respected for his technical judgement, and his ability to keep out of things while dropping technical bombs into other peoples' conversations. There's two things - being able to tell someone politely that their code is wrong because ..., and also to be able to tell someone that their code "does not feel right". Linus has that ability in spades, while unfortunately Pax et al don't have those people skills ...

Cheers,
Wol

This isn't fixed until the compilers are

Aaron1011 — Tue, 04 Jul 2017 03:15:44 +0000

> If there is a way to crash/corrupt your machine from a sequence of instructions, attackers will use asm() to insert those specific instructions

Stack Clash isn't a vulnerability in the kernel - it's a vulnerability that allows an attacker to gain control over a process that they wouldn't normally be able to (e.g. a guid/suid'd program like 'sudo'). The kernel change simply makes it less likely for this kind of vulnerability to be exploited (though stack probing is needed to truly fix the issue).

Ripples from Stack Clash

BenHutchings — Mon, 03 Jul 2017 18:33:55 +0000

I don't see what nationality has to do with this. (Also, Linus is a naturalised American.)

This isn't fixed until the compilers are

Jandar — Mon, 03 Jul 2017 08:06:00 +0000

If an attacker can write the executed code with asm(), he/she is already past the defense this fix should create.

This isn't fixed until the compilers are

dlang — Mon, 03 Jul 2017 07:29:00 +0000

you can never count on the compilers 'fixing' a problem like this.

If there is a way to crash/corrupt your machine from a sequence of instructions, attackers will use asm() to insert those specific instructions, the fact that the rest of your machine was compiled with a compiler that does something to avoid this bug isn't going to help you.

Ripples from Stack Clash

dtlin — Sun, 02 Jul 2017 17:04:10 +0000

You shouldn't be able to change the segment selector through pointer arithmetic. In a flat address space, you have to somehow check that any pointer offset (that could conceivably be controlled by user input) doesn't cause the pointer to change that bit...

Increasingly hard to see how Brad Spengler is/was in the wrong

jospoortvliet — Sun, 02 Jul 2017 15:34:58 +0000

So the PAX st all team was offered money to upstream their work and was uninterested? That explains their aggression towards the kspp efforts i guess...

Ripples from Stack Clash

josh — Sun, 02 Jul 2017 05:27:10 +0000

Well, yeah. The next best thing to reading *all* of LKML, and every other development mailing list out there.

Ripples from Stack Clash

Cyberax — Sun, 02 Jul 2017 03:39:47 +0000

You can do this with the current architecture. Just use bit 62 of the address to indicate heap/stack and set the mappings accordingly.

Ripples from Stack Clash

immibis — Sun, 02 Jul 2017 03:07:27 +0000

With segmentation, you can have separation between stack and non-stack data pages - such that even if you have an address that points to the heap, if you try to use that address to access the stack, you get a segfault. (A *literal* segfault, not one of those pagefaults that we now call segfaults for historical reasons)

This isn't fixed until the compilers are

areilly — Sun, 02 Jul 2017 00:57:28 +0000

Yes, this is not really an OS issue, although I don't really understand how OSes on 64-bit systems can't help by arranging for swapping death to occur long before the "clash."

This is an application security bug issue, just like every other form of security violation via type violation bug. Everyone has written a program that crashes (eventually) by doing infinite recursion. It's not difficult or clever, it's coding an algorithm bug.

All of the examples in the original Stack Clash report aren't system bugs, they're user-input validation bugs in software that has security implications, and -duh- appropriately unexpected input can tickle a bug that has bad results.

The contiguously-allocated stack is just a convenient data structure that's used for the very good reason that it's efficient. It is just a data structure that's used. Use can include misuse. Some languages and systems trade this efficiency for other structures and other trade-offs: some allocate stack-frames on the heap as a traditional linked structure, because that supports a very large number of threads in a small address space, at the cost of more expensive allocation.

The way this issue is being discussed makes it sound as though Red Hat or Linus or the FreeBSD foundation are responsible for input-validation bugs in Exim. That's simply not the case.

Mail archives

marcH — Sat, 01 Jul 2017 06:12:12 +0000

Since the recent and unfortunate gmane events I saw myself using more and more the services from this small and promising startup:

https://groups.google.com/forum/#!msg/ciekawe-papierki/Ox...

> Sometimes I think we're just going to have to make our own.

Errr... of which subset of the whole universe of lists?

Ripples from Stack Clash

marcH — Sat, 01 Jul 2017 06:03:34 +0000

> "the next best thing to reading LKML",

That's when assuming infinite time, otherwise LWN comes first.

Increasingly hard to see how Brad Spengler is/was in the wrong

raven667 — Fri, 30 Jun 2017 17:13:45 +0000

> let's say Red Hat, pays a sizable annual contract for the services of PaX. In return, PaX works

I don't think you are in any position to negotiate working conditions for GrSecurity developers, they are capable of doing that themselves, and you aren't the first person to propose an arrangement like this, if GrSecurity developers really wanted to work this way this would be a solved problem. Since you are starting with a fantasy assumption the rest of your idea doesn't really matter.

Ripples from Stack Clash

sml — Fri, 30 Jun 2017 11:33:34 +0000

Yep, my thoughts exactly. Thanks!

Ripples from Stack Clash

paulj — Fri, 30 Jun 2017 09:44:37 +0000

I think it's fair to say there's blame on both sides, and both sides feel the unreasonableness of the other makes it impossible to work with them. As a result, I don't think any proposed solution that involves requiring one party to unilaterally accept all blame and apologise to the other will work. Working /around/ assignation of blame and first starting to (re-)build some kind of working relationship (via intermediaries perhaps), while avoiding getting into blame and who is wrong for what, is usually step 1 in conflict resolution.

Find a good intermediary, and finding the resources, would be the first steps, I'd imagine.

Ripples from Stack Clash

itvirta — Fri, 30 Jun 2017 06:48:17 +0000

I thought long mode (64-bit mode) only supports flat segments, that contain the whole memory area
from 0 to 2^64. Not just because segmentation isn't used in the ABI, but just that the hardware doesn't support it.
Even if i386-style segmentation would be available, it would require pointers to have the segment id with them
everywhere, which makes them unnecessarily longer (80-bit pointers? Might be somewhat awkward to handle because
of alignment issues etc.)

Now, hypothetically, reserving one bit out of 64 for a stack/heap indicator would give separation without using that much
space, but we would require dedicated arithmetic instructions for pointers so that the usual arithmetic wouldn't be able to
change the indicator bit... Either that, or just go back to having a large unreserved area between the two areas, which kinda
seems like what the fix we have for the current issue.

Ripples from Stack Clash

flussence — Thu, 29 Jun 2017 23:25:21 +0000

>1. Problem: Linus's belittling and antagonism for years

>Solution: Linus writes a formal, personal apology. Quoting all of his derogatory comments that were later proved false (in a concise way), and saying, "mea culpa", please accept our community apology and come back into the fold and work with us in good faith.

And once Linus has apologised for being Finnish, grsecurity can apologise for being American.

Ripples from Stack Clash

kmeyer — Thu, 29 Jun 2017 22:22:02 +0000

The whole thing gets probed. Why, do you think that particular case is sane and should be optimized for?

This isn't fixed until the compilers are

cesarb — Thu, 29 Jun 2017 18:33:18 +0000

For arbitrary userspace, yes, but if you know how much stack space your userspace can jump over in the worst case, and it's on the initial thread, having a stack gap bigger than that amount can be a fix. For instance, if your worst function allocates a 256K buffer in the stack, having a 260K or bigger stack gap should be enough to prevent it from being exploited.

This isn't fixed until the compilers are

ebiederm — Thu, 29 Jun 2017 17:05:50 +0000

Let's be clear no amount of kernel effort alone will ever fix this. This is not fixed until the compilers are updated and user space is rebuilt.

Mail archives

pjones — Thu, 29 Jun 2017 14:49:27 +0000

I would totally contribute to a kickstarter for funding this :)

Ripples from Stack Clash

felixfix — Thu, 29 Jun 2017 13:55:36 +0000

"one-page-at-a-time expansion is a more sane design"

What if the local var which expands the stack is a multi-page array which is populated randomly?

KSPP

corbet — Thu, 29 Jun 2017 13:44:05 +0000

It is worth remembering that KSPP is not a Linux Foundation project, and there is no "KSPP money" slush fund that the LF could have directed differently.

Mail archives

corbet — Thu, 29 Jun 2017 13:38:37 +0000

There hasn't been a reliable mail archive, especially one that we could link to in an automated way, since gmane went away. Sometimes I think we're just going to have to make our own.

Ripples from Stack Clash

ms-tg — Thu, 29 Jun 2017 13:32:18 +0000

> Further, regardless of prior events, it would still be good to try fix this situation, and find some way to navigate around the social and corporate politics so that spender, et al., can earn a living from their security work. Given that that work has clear value, as there are others being paid to take that work and upstream it.

I am seeing this the same way. So, here's a second modest proposal, one sketch of an approach that could possibly help fix the situation:

1. Problem: Linus's belittling and antagonism for years

Solution: Linus writes a formal, personal apology. Quoting all of his derogatory comments that were later proved false (in a concise way), and saying, "mea culpa", please accept our community apology and come back into the fold and work with us in good faith.

2. Problem: No clear paths forward.

Solution: Ask (don't tell, don't make assumptions) for jointly-designed Next Steps.

In a public forum, Linus in tandem with someone at Red Hat and someone at Linux Foundation or other funding-source formally write a short public letter making clear that they are open to paying, over a number of years, for Brad and Pax Team's time to work diligently with the community to upstream the source. And *ask them* for a suggestion of how that relationship might work. Listen to what they say -- see if there are solutions that meet all interests.

Perhaps there's some reason why this sort of common sense approach cannot work -- would love to know more?

Ripples from Stack Clash

paulj — Thu, 29 Jun 2017 11:35:57 +0000

spender has stated that, but it's never been clear if he stated that _because_ he had been so antagonised/insulted by being left out in the cold, with other people being paid (presumably well) to work on code he wrote.

It is obvious that that would induce a degree of bitterness and even hatred. The chain of cause and effect is not clear though.

Further, regardless of prior events, it would still be good to try fix this situation, and find some way to navigate around the social and corporate politics so that spender, et al., can earn a living from their security work. Given that that work has clear value, as there are others being paid to take that work and upstream it.

Increasingly hard to see how Brad Spengler is/was in the wrong

paulj — Thu, 29 Jun 2017 11:28:36 +0000

That's been done in a way that doesn't benefit spender, PaXteam, etc., at all. Which has made the situation worse. Not really surprising that paying one set of people to work on other's people code, leaving those original authors out in the cold, would make that other set of people even more antagonistic.

Yes, I know there's lots of people factor issues here, but... there is clear value in their security work regardless, and the above situation is highly sub-optimal (for users esp.).

Ripples from Stack Clash

SLi — Thu, 29 Jun 2017 10:53:51 +0000

What's going on with the mail archive links? It seems to me links to mails in LWN articles are broken in maybe 20% of the time, long term. Is it just that there is no non-broken mail archive service?

Increasingly hard to see how Brad Spengler is/was in the wrong

nhippi — Thu, 29 Jun 2017 09:55:59 +0000

When KSPP started to materialize, it would have made sense to Linux Foundation to ask Brad to do it. Even if they didn't, Brad should have seen the writing in the wall and proposed to do it himself. We don't know why exactly Brad is not getting the KSPP money, and instead some other people are paid to upstream Brads code. But chances are "not being nice to your potential future customers" played a big part in it.

Linus is not much better - which is especially bad because some people think the success of Linux is because of his style and LKML culture - rather than despite of it.

Ripples from Stack Clash

farnz — Thu, 29 Jun 2017 09:39:45 +0000

Windows has had a userspace exception mechanism in all Win32 versions. The Windows kernel won't grow stacks automatically; instead, if you want a stack to grow, you set up a guard page, which will trigger a Guard Page Violation exception when touched (instead of the normal Page Fault exception), and then automatically put the page you set up when you configured the guard page into place.

This lets you map in a new guard page, ready for the next step of growth.

Ripples from Stack Clash

jem — Thu, 29 Jun 2017 09:06:24 +0000

I'm genuinely curious how you intend to solve this problem with an ABI change. Memory pages on the x86-64 can be write protected, and they can be marked non-executable, but to my knowledge there is no separation at the hardware level between "stack" pages and "non-stack" data pages.

> It seems weird that we have hardware and software mechanisms all over the place to keep the other layers of an OS from trampling each other, but the stack and heap are allowed to even though nothing good can possibly come of it.

The difference here is that we are not talking about layers of an OS, but memory areas that are internal to a single process. Also, it's a bit inaccurate to talk about "the stack", since there are typically lots of separate stacks, one for each thread.

Ripples from Stack Clash

vegard — Thu, 29 Jun 2017 07:05:21 +0000

> any attempt to access will cause an exception (in userspace; Windows has exceptions even in C)

Are you referring to page fault exceptions, which is a hardware feature different from C++ exceptions? I'm pretty sure it's the same on Linux, and you can even handle those by handling SIGSEGV and/or SIGBUS.

> I doubt it was an intentional security feature on part of the Windows designers; it was probably a side effect of the design of its virtual memory subsystem,

I don't know, I think one-page-at-a-time expansion is a more sane design from the outset. I mean, for sure they didn't have the current Linux userspace exploits in mind, but I don't find it unthinkable that there was at the very least a vaguely security-related concern behind their design.

I don't think we should assume by default that Windows got something right only because of a "lucky design choice"; that's a bit disingenuous.

Ripples from Stack Clash

jhoblitt — Thu, 29 Jun 2017 01:03:32 +0000

Eh, I had no idea. I suppose one wouldn't have to completely throw out the x86_64 ABI -- couldn't a tweaked ABI live in parallel with slightly different ELF metadata?

Ripples from Stack Clash

PaXTeam — Thu, 29 Jun 2017 00:15:46 +0000

> [...] the CII asking him to write a proposal in order to get the funding ball rolling.

the CII did what? that never happened, you must have misunderstood something. i was the one who asked on cii-discuss in august 2015 how this whole thing could work (before investing my free time into writing a proposal) and got no real response, spender was never part of that discussion.

Ripples from Stack Clash

pizza — Wed, 28 Jun 2017 23:59:38 +0000

> no, no such offer was made. the best Kees could offer at the time (about 2 years ago) was to talk to the CII (he said google's decision to compete with us instead of cooperation was made above his pay grade) and they didn't answer my question whether they'd be willing to fund the necessary hours to get this work done. but yeah, don't let the facts stop you from your conspiracy theorizing ;).

Oh, please.

Spender has explicitly stated, here on LWN (and undoubtedly elsewhere) that he would _never_ accept funding from any entity associated with Linux Foundation, including the CII -- in response to the CII asking him to write a proposal in order to get the funding ball rolling.

That sort of response is a good example of what is known as a "self-inflicted career limiting move".

Ripples from Stack Clash

cesarb — Wed, 28 Jun 2017 23:11:58 +0000

I doubt it was an intentional security feature on part of the Windows designers; it was probably a side effect of the design of its virtual memory subsystem, which as far as I could find doesn't have the kind of automatically growing virtual memory area that Linux has. Instead, it allows one to mark a page as a "guard page", where any attempt to access will cause an exception (in userspace; Windows has exceptions even in C), and automatically remove the "guard page" mark. The exception handler has to allocate another page and mark it as the new "guard page".

Since there's only one "guard page" on the stack, and you have to touch it or the stack won't grow, compilers for Windows have to call a function to touch the guard page whenever the stack frame for a function is larger than one page. Since on Linux you don't have to touch the guard page for the stack to grow, compilers for Linux haven't implemented it.

This is not the first time that different design choices happen to prevent a vulnerability. For instance, Linux (X11 actually) was not vulnerable to Shatter attacks, due to not sending function pointers through the same message loop used for inter-application messages.

Ripples from Stack Clash

PaXTeam — Wed, 28 Jun 2017 22:51:37 +0000

> Apparently this offer was made.

no, no such offer was made. the best Kees could offer at the time (about 2 years ago) was to talk to the CII (he said google's decision to compete with us instead of cooperation was made above his pay grade) and they didn't answer my question whether they'd be willing to fund the necessary hours to get this work done. but yeah, don't let the facts stop you from your conspiracy theorizing ;).

Ripples from Stack Clash

nix — Wed, 28 Jun 2017 22:38:24 +0000

The KSPP has been trying to improve the situation, but as ms-tg put it in another comment: hiring Brad and PaXTeam to actually port grsecurity to the mainline would have been the most efficient (at least in a technical point of view) way. Just remind yourself that (at least until last year) Brad was working on grsec _on his spare time_.

Apparently this offer was made. Brad refused (though why is unclear amid all the conspiracy-theorizing). Frankly it seems unlikely this could have ended well: Brad isn't going to be happy until everything he does goes straight into the kernel without review or question, and that's just not how development of anything works. The first code review would lead to a titanic explosion and probably a rapid firing. (Heck, I suspect one wouldn't have to wait that long: the first security bug after he was hired, even in code Brad had nothing to do with, would lead to a mass of snideness, an escalating flamewar...)

Employment requires a degree of treating your colleagues like human beings and considering that their concerns may have value and are not exclusively motivated by hatred and malice. I suspect there is a reason grsecurity is off working on its own...

Ripples from Stack Clash

Trou.fr — Wed, 28 Jun 2017 21:46:23 +0000

> Of course, nobody else posted such a patch either; the community can only
> blame itself for not having fixed this problem. Perhaps LWN shares part of
> that blame for presenting the problem as being fixed when it was not; if so,
> we can only apologize and try to do better in the future. But we might argue
> that the real problem is a lack of people who are focused on the security of
> the kernel itself. There are few developers indeed whose job requires them
> to, for example, examine and address stack-overrun threats. Ensuring that
> this problem was properly fixed was not anybody's job, so nobody did it.

While I'm usually a fan of LWN articles, the recent amount of self-delusion
about security in the Linux kernel has been really annoying.

Maybe you could have reflected on the fact that Linus has been repeatedly
insulting people trying to improve Linux's security for years, which certainly
is a deterent to any contribution on the subject.

And there _are_ people who care about the Linux kernel security: they use grsec
patches.

The KSPP has been trying to improve the situation, but as ms-tg put it in another comment: hiring Brad and PaXTeam to actually port grsecurity to the mainline would have been the most efficient (at least in a technical point of view) way. Just remind yourself that (at least until last year) Brad was working on grsec _on his spare time_.