State of the Kernel Self Protection Project

By Jake Edge
August 31, 2016

At the Linux Security Summit in Toronto, Kees Cook updated attendees on the Kernel Self Protection Project (KSPP), which was announced only nine months ago. KSPP is targeted at adding protections to the kernel to guard against kernel bugs so that those bugs can't lead to a kernel compromise. Progress has been made, but there is lots more to do.

"Security" is a somewhat loaded word, but for the context of the presentation, he is really talking about "self protection", he said. That goes beyond access control (e.g. SELinux), attack surface reduction (e.g. seccomp), bug fixing, protecting user space, and kernel integrity. It is proactive and does rely on kernel integrity, but it is more than that.

Why KSPP?

There are huge numbers of systems using Linux and the vast majority are running old software. Many will never get updates. As we move forward, we need devices protecting themselves because the lifetime of bugs in the field is much longer than even the long bug lifetimes in the kernel. Things will be getting worse with the "Internet of Things" (IoT); while users may replace phones every few years, a door lock or other IoT device may live for fifteen years or more.

Cook has seen a lot of blame shifting. Once a bug is fixed, it is seen as someone else's problem to get it rolled out. But even if every bug was found and fixes were magically sent to all affected devices, bugs live for a long time in the kernel.

In 2010, Jonathan Corbet looked at the lifetime of security flaws in the kernel and found that the average time between introduction and fix was roughly five years. Cook analyzed the Ubuntu bug tracker for the kernel from 2011 through 2016 and found much the same, though the two critical bugs in that time frame averaged 3.3 years between introduction and fix. He put up a graph of the lifetime of all of those bugs and another that focused on just the critical and high severity bugs (as can be seen in his slides [PDF]). Bugs are clearly living for quite some time in the kernel.

One question he gets a lot is: "isn't this all just theoretical?" If kernel developers are not finding the bugs, maybe others aren't either. But that is "demonstrably untrue", Cook said. Attackers are finding these bugs and they are better at it than the kernel community is. Most attackers do not publicly boast about the zero-day flaws they have found. The window of opportunity for exploiting these bugs is large and not at all theoretical.

Kernel developers are fighting bugs, of course, and finding and fixing them. But they are also accelerating the pace of bug introduction. Bugs exist whether they are known or not and an empty bug tracker does not imply there are no bugs. The bugs are there and kernel developers need to act like they are, he said. Whack-a-mole is not a solution long-term; the current bug finding and fixing should continue, but there needs to be more.

At this point, user space is becoming more difficult to attack, which is making the kernel's attack surface more appealing. Containers are also painting a large target on Linux. In addition, it is important to recognize that lives depend on Linux. That statement makes some people uncomfortable, he said, but kernel bugs have been weaponized by organizations like Hacking Team that sell them to governments that use them to target dissidents and activists.

There is some truth to the statement that security bugs are just normal bugs. "Your security bug may not be my security bug", he said. But kernel developers have little knowledge of what bugs attackers use. Those bugs might even be in code that is not upstream, such as a vendor driver. Self protection could make kernels safer, even for that non-upstream code.

Killing bugs and bug classes

So, killing particular bugs is good, but killing bug classes is much better. That can ensure that those kind of bugs can never occur again, even if they are in out-of-tree code. But it is impossible to kill all bug classes.

So, what is best is to kill off the exploitability of bugs. That needs to be done even if the features make development more difficult. So what is needed is eliminating exploitation targets and methods, as well as removing information leaks and anything else that assists attackers. There will be tradeoffs to maintainability and performance, but those need to be judged on their merits, not just avoided at all costs.

Typical exploits are not simply using one flaw; instead they tend to chain together multiple flaws. In order to do that, they need to know where the targets are, inject or build malicious code, and redirect execution to that code.

There is already a lot of code to do self protection available in grsecurity and PaX. There are also academic papers describing ways for kernels to protect themselves. So a lot of these protections don't need to be invented, they just need to get upstream. There is, he said, a large demand for these kinds of features to be added to the kernel.

To address that, he started KSPP in order to gather people interested in self-protection technologies to discuss, test, and document them. There are other people working on excellent technologies for the kernel to protect user space, but he wants to focus on the kernel protecting itself. Currently, there are around ten organizations and five individuals working on roughly twenty separate technologies. It is not a fast process and he intends to take it "slow and steady".

He listed a number of people that are "under the KSPP umbrella". It shows that there are a lot of people paying attention to these ideas, which he is "extremely grateful" for. It is nice to have a deep pool of people to help, he said.

There are numerous areas of interest to the project. His slides detail multiple bug classes, examples of exploits for the class, the relevant mitigations that have already gone into the kernel (which are highlighted in bold), and the work that is either currently being done or is planned for the future.

For example, there have been protections for stack overflows in the kernel for quite some time. 2.6.30 saw the addition of the GCC -fstack-protector flag, while -fstack-protector-strong was added for 3.14. For the future, Andy Lutomirski has been working on virtually mapped stacks that will allow for guard pages around the kernel stack for further protection.

Other classes he covered include integer over and underflow, heap overflow, format string injection, kernel pointer leaks, uninitialized variables, use after free, direct kernel overwrite, function pointer overwrite, user-space execution from the kernel, accessing user-space memory directly from the kernel, and return-oriented programming (ROP).

Progress

He then went through the kernel releases since KSPP was started and noted the protections that had been added in each. For 4.3, the "Privileged Access Never" (PAN) hardware memory protection feature was emulated for the ARM architecture. That helps protect the kernel from accessing user-space memory inappropriately. Ambient capabilities were added to make Linux capabilities behave as most programmers expected them to. In addition, seccomp support was added for PowerPC in 4.3.

The 4.4 kernel saw the addition of the x86_64 configuration option to allow kernels to be built without support for three legacy virtual system calls at fixed kernel addresses. That is a significant attack surface reduction, Cook said. In 4.5, the address-space layout randomization (ASLR) feature added a way for administrators to control the amount of entropy in the random base address. It is a notable user-space protection feature.

There were numerous protection features added to the 4.6 kernel. Most were adding protections to various types of memory for different architectures. Read-only data (RODATA) became the default for ARMv7+ and arm64, while its use became mandatory for x86. The basic infrastructure for marking data as read-only after initialization (__ro_after_init) was added. x86 got execute-only memory and arm64 got support for kernel ASLR (KASLR) on the text and module base addresses.

The MIPS architecture got KASLR support for text and modules in 4.7. In addition, the SLAB memory allocator got support for randomizing its free list. The extended BPF just-in-time (JIT) compiler added constant blinding, which helps prevent exploits from using the JIT code to accomplish its purposes.

In 4.8, Cook expects quite a few features to land. KASLR on x86_64 will be able to use the full range of physical memory and randomization of the kernel memory base address will be added. The SLUB allocator will get free-list randomization. The GCC plugin infrastructure will be added with some example plugins that do not have much security impact. But it will allow grabbing plugins from grsecurity/PaX and using them on the kernel. Hardened usercopy should also make the cut, as will the changes to put ptrace() handling before seccomp handling to avoid a loophole where ptrace() can be used to avoid the seccomp rules.

Looking into his crystal ball, Cook sees several features that seem likely for the 4.9 kernel. The latent_entropy GCC plugin, which will provide more entropy at startup time, is one. Another is the virtually allocated stack work mentioned earlier. Some linked-list hardening patches and PAN emulation for arm64 round out his list.

Challenges

The biggest challenge for the project is the culture on both the kernel and security sides. The kernel side is quite conservative. It took sixteen years (and five or six different developer's attempts) to get a simple, clean solution for symbolic link restrictions upstream. It is going to require kernel developers to pay more attention to these kinds of protections. But those trying to upstream these features need to have patience and understand how kernel development is done.

One of the challenges is explaining the rationale for a particular feature. In order to get it upstream, other developers will have to understand it. Code is not written for the kernel itself, it is written for the other kernel developers, Cook said, so it is critical to make it understandable to them.

As with most projects, people are needed for KSPP. Dedicated developers, testers, reviewers, backporters (to vendor kernels), and so on are all needed. When a set of patches is posted, having them be greeted with silence is the worst outcome. He encouraged those interested to get involved.

[I would like to thank the Linux Foundation for travel support to attend the Linux Security Summit in Toronto.]

Index entries for this article
Security	Hardening
Security	Linux kernel/Hardening
Conference	Linux Security Summit/2016

to post comments

State of the Kernel Self Protection Project

Posted Sep 1, 2016 3:21 UTC (Thu) by spender (guest, #23067) [Link] (56 responses)

Another nice PR piece for the KSPP, funded by the Linux Foundation, this article offers nothing of journalistic interest -- it's merely a paraphrasing of Kees' slides but given a deceiving disinterested third-person spin. Anyone who has read the slides (really, go read them, you'll see this article is a slide-for-slide recitation) will see reproduced in this article the exact same errors and tricks to make the KSPP appear more successful than it is. For instance, anyone who sent a single comment about a KSPP patch is considered a KSPP contributor. Also listing security features that have nothing to do with KSPP as somehow being KSPP success stories -- the ARM software PAN work (a ripoff of my ARM UDEREF research and work from late 2012) was completed well before KSPP was ever even announced. KSPP was announced on Nov 5 2015, Linux 4.3 was released on Nov 1 2015 -- how can KSPP have anything to do with anything from Linux 4.3?

What isn't mentioned (because of the natural bias of the reporting on this site, to only regurgitate upstream kernel developer opinions) is that not one original useful security defense has come out of the KSPP contributors -- it's all ripoffs of grsecurity/PaX and some security theater thrown in (PS: look for yet another generic KASLR defeat this October). In some cases the ripoffs are entire copy+pastes of our work, done with no understanding whatsoever, and in multiple instances Intel's copyright slapped over the entire file without having made any modifications at all. Anything they can't figure out (like the full PAX_USERCOPY code, despite numerous attempts) gets punted on so they can rush out a KSPP success story and get those fancy cargo-culted buzzwords out into articles like this.

If KSPP is in need of a new slogan, I suggest: "KSPP: All the parts of grsecurity/PaX that are simple enough for us to copy+paste and take credit for"

-Brad

State of the Kernel Self Protection Project

Posted Sep 1, 2016 4:44 UTC (Thu) by kris.shannon (subscriber, #45828) [Link] (4 responses)

I have not yet read the slides, but from the article I assumed that the features KSPP has upstreamed were mainly from grsecurity/PAX.

I can understand your annoyance if they have mishandled a feature (as you suggest they have with PAX_USERCOPY) but your constant assumption of bad faith is counterproductive.

When I saw there was just one comment to this I had already guessed it was going to be from you and I was pretty sure of what the tone of it would be.

State of the Kernel Self Protection Project

Posted Sep 1, 2016 4:53 UTC (Thu) by spender (guest, #23067) [Link] (3 responses)

And when I saw a reply to my comment, I knew it would be the old tired tone argument with no technical contribution or rebuttal.

-Brad

State of the Kernel Self Protection Project

Posted Sep 1, 2016 7:43 UTC (Thu) by micka (subscriber, #38720) [Link] (2 responses)

Well, you know, I knew when I saw your name on a post, I knew that it wouldn't amount to anything interesting before reading it.

State of the Kernel Self Protection Project

Posted Sep 1, 2016 8:35 UTC (Thu) by corbet (editor, #1) [Link] (1 responses)

Please, regardless of the quality of the original comment, this kind of thing doesn't make the conversation better. If you disagree with what he wrote, feel free to say how you would correct it, but let's keep the conversation on that level if we can...?

Thanks.

State of the Kernel Self Protection Project

Posted Sep 1, 2016 9:59 UTC (Thu) by micka (subscriber, #38720) [Link]

Yes you're right, I'll just use the blacklist.

State of the Kernel Self Protection Project

Posted Sep 1, 2016 12:32 UTC (Thu) by jake (editor, #205) [Link] (25 responses)

Sorry you didn't like the article (and, evidently, the project), Brad, but this:

> it's merely a paraphrasing of Kees' slides but given a deceiving disinterested
> third-person spin.

is simply nonsense. I am reporting on a talk that Kees gave, for which he used the slides, so *Kees* paraphrased his slides and I reported on that. I am not a disinterested person (there's a reason I attended the Security Summit after all), so I certainly didn't try to spin things that way, nor do I see this supposed spin in the article. I am always open to criticism of my writing so that I can improve, but this is not useful in that regard.

jake

State of the Kernel Self Protection Project

Posted Sep 1, 2016 17:47 UTC (Thu) by chirlu (guest, #89906) [Link]

Just an idea, “disinterested” may not have meant “bored” here, but “neutral” (i.e. without a personal interest, without “a horse in the race”).

State of the Kernel Self Protection Project

Posted Sep 2, 2016 0:03 UTC (Fri) by jschrod (subscriber, #1646) [Link]

Jake, no problem at all. We understand what you did. And you did a good job, we're grateful for it.

spender just wanted to demonstrate publicly why the kernel folks don't want to work with him, that's all. Film at 11.

State of the Kernel Self Protection Project

Posted Sep 2, 2016 2:16 UTC (Fri) by spender (guest, #23067) [Link] (1 responses)

My standards may be too high, but in my mind reporting != repeating verbatim what a particular person said, regardless of the facts. To repeat that "numerous protection features were added" to Linux 4.6 for instance, when that had already been thoroughly debunked here: https://forums.grsecurity.net/viewtopic.php?f=7&t=4476

There is not a whiff of critical thinking present here, it doesn't offer anything that isn't already present in the slides themselves (except, as I mentioned, to bestow a false impartiality to the evaluation of the KSPP as a success).

People come here to be informed and learn something, right? Or is it just to reinforce myths for yes-men?

-Brad

State of the Kernel Self Protection Project

Posted Sep 2, 2016 5:40 UTC (Fri) by karkhaz (subscriber, #99844) [Link]

> My standards may be too high, but in my mind reporting != repeating verbatim what a particular person said

Opinionated, analytical pieces are just one kind of reporting. Sometimes describing the facts without embellishment really is what is called for---for example, when reporting on conference talks. There do exist much more analytical articles on LWN, but this isn't one of them; I don't think it was intended to be. It would be a disservice both to conference speakers and to us readers if the LWN staff added their own editorial salt and pepper when reporting on conference talks, don't you think? The LWN staff do occasionally get opinionated when writing proper editorial pieces that are not based on talks, but I think embellishing other people's talks could get somewhat dangerous. LWN is the only place where most of us get any coverage of these talks, so I'd rather read a neutral summary and make my own mind up about the speaker; and maybe read a proper opinion-piece about the broader issues afterward.

> To repeat that "numerous protection features were added" to Linux 4.6 for instance
> There is not a whiff of critical thinking present here

The whole article is repeating stuff, without thinking too hard about it, that's kind of the point; that's made quite clear by the liberal sprinklings of "Cook said..." throughout the article. You have totally valid reasons for finding some of the claims to be lacking in merit, but they are Cook's claims and I think that it's very unfair to be flaming the messenger about them.

> Another nice PR piece for the KSPP
> It's merely a paraphrasing of Kees' slides but given a deceiving disinterested third-person spin

I don't think that Jake was trying to imbue Cook's talk with any special air of legitimacy by affecting a disinterested (neutral) tone, and especially not pushing an agenda. Cook is surely pushing KSPP's agenda, which may or may not be a questionable one, and it's reasonable to discuss that. But this article cannot be said to be a PR piece; it reported on the talk in an impartial way, and again I think it's unfair to accuse the author of partiality.

State of the Kernel Self Protection Project

Posted Sep 3, 2016 14:11 UTC (Sat) by spender (guest, #23067) [Link] (20 responses)

So let's assume that this article isn't a PR piece and would still totally exist if you weren't funded by the Linux Foundation to attend the conference.

Say I were to fund you to attend Black Hat this year where there were two presentations on generic KASLR defeats (the second not even really being new, it was already blogged about in 2014). I didn't see any articles on here about any of those talks (to actually educate others, here are the slides: https://www.blackhat.com/docs/us-16/materials/us-16-Fogh-... https://www.blackhat.com/docs/us-16/materials/us-16-Jang-...). Would you finally report on that in the same way that you delude your readers into thinking KASLR is useful by repeating slides from Kees' talk?

Just to make it interesting, I'm publicly making that a standing offer for next year for you Jake, at any real security conference that has a presentation relevant to Linux kernel security. If I happen to see one, I'll reach out to you. If you happen to see one, feel free to email me at spender@grsecurity.net and we'll set it up.

-Brad

State of the Kernel Self Protection Project

Posted Sep 5, 2016 16:17 UTC (Mon) by jake (editor, #205) [Link] (19 responses)

> Would you finally report on that in the same way that you delude your readers
> into thinking KASLR is useful by repeating slides from Kees' talk?

I know KASLR is a hot-button issue for you, Brad, but I think you might want to reread the article (and slides) again. I mention some architectures got the feature and that it will be expanded on x86_64 soon ... that's it ...

so exactly what am I deluding readers into believing? (or, in truth, is Kees deluding listeners with in the talk?)

I will go read those papers, though, thanks for the pointer ... I am always happy to get pointers of that sort in email if you (or others) see things you think we should be covering.

I have attended Black Hat once in the past ... an interesting conference to be sure, though the hype levels can only have gotten worse, sadly. I'm open to attending it again or to go to some other security conference of interest down the road, as my schedule permits.

We are certainly open to your offer and would accept travel sponsorship money from grsecurity (or you), with credit given as with others and just as much editorial control on what gets written as others (i.e. none). Thanks!

jake

State of the Kernel Self Protection Project

Posted Sep 5, 2016 17:37 UTC (Mon) by spender (guest, #23067) [Link] (15 responses)

Hi Jake,

Thanks for agreeing to my experiment. What I hope to demonstrate is that even without any kind of editorial control on behalf of the sponsor, LWN's reporting style basically enables sponsors to obtain "free" PR simply by having LWN repeat verbatim whatever a presenter has said, without consideration of any other facts the LWN staff may be aware of at the time. I'll note that my past LSS talk wasn't covered by LWN when the Linux Foundation wasn't sponsoring travel for it. Imagine if we were able to sponsor travel for more presentations than the Linux Foundation -- that would certainly weight the perception of viewpoints the biggest sponsor wants to achieve. Of course, if you were to not report on the security presentations in the same way as these from the Linux Foundation (by interjecting editorial comments), that would be an important bias to expose as well.

Not to mention that this report is a bit more unethical than even what I describe above, as the work itself is funded by the Linux Foundation (a trade association that operates in the interests of its commercial members), and on top of that the Linux Foundation is sponsoring you to report on it. That would be like us paying you to write about grsecurity positively by covering someone's positive talk on grsecurity. I won't ask you to do that as part of this experiment even if that would be a more apples to apples comparison to prove the point.

Totally in agreement about Black Hat being overhyped, but then so too is the presentation you covered :)

-Brad

State of the Kernel Self Protection Project

Posted Sep 5, 2016 17:53 UTC (Mon) by nix (subscriber, #2304) [Link]

LWN's reporting style basically enables sponsors to obtain "free" PR simply by having LWN repeat verbatim whatever a presenter has said, without consideration of any other facts the LWN staff may be aware of at the time.

So, you're saying that if LWN had noted that, say, this entire project existed purely because the kernel developers cannot work with you in particular (well, you and a few other anonymous people) which neatly explains why it is gatekeeping, oh sorry "ripping off" your work into the kernel, which is indeed another fact the LWN staff are demonstrably aware of, that you would have not attacked them straight out of the gate for saying that?

I don't believe you.

(I note that you have in the past attacked the kernel people for not accepting your work: yet here you attack them for doing the opposite, simply because the contribution of changes is coming through an intermediary. Is the real problem that they're not willing to accept your work without review, changes, or alterations of any kind whatsoever? Well, duh, you're not going to be treated like that because nobody is in any free software project worth its salt.)

State of the Kernel Self Protection Project

Posted Sep 5, 2016 18:29 UTC (Mon) by jake (editor, #205) [Link] (13 responses)

> LWN's reporting style basically enables sponsors to obtain "free" PR simply by
> having LWN repeat verbatim whatever a presenter has said, without consideration
> of any other facts the LWN staff may be aware of at the time.

gosh Brad I really don't think you get what we try to do with conference coverage ... we try to accurately reflect what the speaker said, so that we don't put words in their mouth ... if they make an outlandish claim (which happens rarely and did not happen here), we may point to something that contradicts their statement.

In other kinds of articles, we do try (and sometimes even succeed) to get multiple viewpoints into the mix. For example: https://lwn.net/Articles/546686/ ... though it appears to gall you, our sources tend to be public postings on the development mailing lists of interest. The fact that you rarely participate in lkml and similar lists makes it harder to get your thoughts and complaints reflected. You seem to believe there is some grand conspiracy to omit your efforts, but that simply isn't the case -- it is simply a consequence of where we can focus our limited attention. Had the folks who have published the KASLR breakage recently posted to lkml (or similar), I might have noticed and already written about them. I realize you find our efforts not be up to your journalistic standards, but I think we do pretty well within the limits we have. ymmv ...

In addition, I think you aren't seeing the big picture here ... the LF did *not* ask me to do anything ... cover LCNA, ContainerCon, LSS, or anything else. I applied for travel money to go to Toronto -- writing anything at all was kind of implied, but not required at *all* ... In addition, I am planning to write up LSS in its entirety -- Julia's and Kees's talks were just the first two chronologically ... more is coming, plenty of which has no particular LF connection at all ... I'm sorry you find me to be terribly unethical, but I think I am pretty transparent about what I am trying to do here ...

Unlike you, though, I do see KSPP as a success ... getting any of the mitigations, most of which have come from grsecurity/PaX, upstream is a success, however funded ... those mitigations were *demonstrably* not going upstream until recently and KSPP is a big reason why that's changed.

jake

State of the Kernel Self Protection Project

Posted Sep 12, 2016 14:24 UTC (Mon) by PaXTeam (guest, #24616) [Link] (12 responses)

> I do see KSPP as a success ... getting any of the mitigations [...] upstream is a success, however funded

Jake,

i've sat on this one because quite frankly, the level of cruelty and arrogance that you displayed here left me speechless for a (long) moment. i'd like to believe that you're better than this and haven't inadvertantly become full corporate shills so i hope that you will realize the judgement error you made.

let's review some facts before coming to the conclusion, shall we? i've been working on PaX *entirely* on my free time, that is, there's not a single line of code in PaX that was paid for by somebody. out of the last 16 years i spent over a decade in unemployment to make the necessary amount of free time available for the research and development of the world class defenses in PaX. my estimate is that i must have spent upwards of 15k hours of my free time on PaX already.

and what do i get in exchange here from you and some other even more clueless readers? you're basically making demands on me to spend thousands of hours more of my free time on upstreaming my code or else i will be called names and even be *blamed* for not pushing my code upstream. this is insanity, not any sort of success you're cheering about so much, unless of course you're happy about the hobbyist linux developers going away and getting replaced by corporations and their self-congratulating employees. the ends justify the means, don't they?

State of the Kernel Self Protection Project

Posted Sep 12, 2016 15:01 UTC (Mon) by pizza (subscriber, #46) [Link] (10 responses)

> let's review some facts before coming to the conclusion, shall we?

Oooh, yes, let's.

Fact -- Stuff that has been upstreamed is infinitely more useful to general end-users than the stuff that hasn't.
Fact -- You've stated that as-is, PaX is not suitable for upstream and requires a lot of work to do so
Fact -- You've stated that you're not willing to upstream anything.
Fact -- You're complaining that someone else is willing to upstream selected bits or re-implement some of the ideas.

Oh, one more thing:

> there's not a single line of code in PaX that was paid for by somebody. out of the last 16 years i spent over a decade in unemployment to make the necessary amount of free time [...]

Fact -- Your work on PaX has been largely subsidized by other people. Unless you're claiming that you haven't been supported by taxpayers or a spouse/family during this decade of no income?

Conclusions?

If your goal is to repeatedly demonstrate why nobody wants to work with you, you're succeeding admirably.

State of the Kernel Self Protection Project

Posted Sep 12, 2016 15:35 UTC (Mon) by PaXTeam (guest, #24616) [Link] (9 responses)

> Fact -- Stuff that has been upstreamed is infinitely more useful to general end-users than the stuff that hasn't.

what does this have to do with people making demands on my own free time? it's also wrong since being upstream != enabled.

> Fact -- You've stated that as-is, PaX is not suitable for upstream and requires a lot of work to do so

nope, what i said was there were parts of PaX that were implemented to minimize my efforts of maintenance which may or may not be what an upstreamable implementation would look like. ironically, some of the stuff that got upstreamed is evidence of upstream devs not realizing this fact.

> Fact -- You've stated that you're not willing to upstream anything.

not on my free time, correct.

> Fact -- You're complaining that someone else is willing to upstream selected bits or re-implement some of the ideas.

wrong, you should probably read and understand what you're replying to. my complaint is that people make *more* demands on my free time and have the guts to blame me when i refuse to do their bidding.

> Fact -- Your work on PaX has been largely subsidized by other people.

got a proof of that fact or shall we call it what it is, a baseless speculation?

> Unless you're claiming that you haven't been supported by taxpayers or a spouse/family during this decade of no income?

it's not really your business but no, that's not what happened, it was all my own savings.

> Conclusions?

your 'facts' speak for yourself quite well i think: thank you for demonstrating what clueless arrogance looks like to the outside world. yes, that's a fact ;).

State of the Kernel Self Protection Project

Posted Sep 12, 2016 16:18 UTC (Mon) by pizza (subscriber, #46) [Link] (8 responses)

> you're basically making demands on me to spend thousands of hours more of my free time on upstreaming my code

I shouldn't have to point out that something requiring "thousands of hours" to upstream, is, by definition, not something that is currently suitable for upstreaming.

> it's not really your business but no, that's not what happened, it was all my own savings.

Fair enough, I retract my statement to that effect. (I have to say that blowing a decade's worth of savings on a vanity project strikes me as being a rather unwise investment, but it's your time and money to do with what you will)

However, I stand by the rest -- You're saying that you're not willing to put forth the effort to upstream things because it's a lot of work, while simultaneously complaining about those who do, and the efforts to document that ongoing work.

If nothing else, it's incredibly boneheaded PR, but I doubt I'm the only one who's reads what you write and concludes that your public attitude demonstrates why you have such long bouts of unemployement.

But hey, it's your time, money, and life. Have at it, you answer only to yourself, and thus only have yourself to blame for the outcome.

State of the Kernel Self Protection Project

Posted Sep 12, 2016 17:22 UTC (Mon) by PaXTeam (guest, #24616) [Link] (4 responses)

> I shouldn't have to point out that something requiring "thousands of hours" to upstream, is, by definition, not something that is currently suitable for upstreaming.

you're wrong, there're many features in the kernel that i'm sure required even more time to get in shape (e.g., -rt, file systems, etc), second, this estimate isn't for a single particular feature but the ones i've heard people express an interest for.

> on a vanity project...

... that fundamentally influenced the entire industry to the point that you've been running code implementing my ideas for over a decade now. as for (not) being an investment, i'm not sure you realize but you're arguing against yourself and saying that i should have asked money from the get go (e.g., via patents, etc) instead of keeping it as a hobby that i gave away for free (gratis, in addition to libre).

> You're saying that you're not willing to put forth the effort to upstream things because it's a lot of work,

no, that's not what i'm saying. let me quote myself back, perhaps it'll sink the second time: not on my free time.

> while simultaneously complaining about those who do, and the efforts to document that ongoing work.

wait, are you saying that i should stop having an opinion just because it happens to be about my own code?

State of the Kernel Self Protection Project

Posted Sep 12, 2016 17:53 UTC (Mon) by pizza (subscriber, #46) [Link] (3 responses)

> wait, are you saying that i should stop having an opinion just because it happens to be about my own code?

As the saying goes, "Opinions are like a**holes. Everybody's got one and everyone thinks everyone else's stinks."

State of the Kernel Self Protection Project

Posted Sep 13, 2016 10:25 UTC (Tue) by sdalley (subscriber, #18550) [Link] (2 responses)

pizza, such remarks are unworthy of you.

The opinion of one who has, by the sweat of his brow, produced good working code, (for example, code in wide use by those who care about robust security-hardened kernels), will always be worth a lot more than the opinions of those who snipe from the sidelines.

Any of us would find continual criticism demoralizing. Why not try constructive praise?

State of the Kernel Self Protection Project

Posted Sep 13, 2016 11:35 UTC (Tue) by jubal (guest, #67202) [Link] (1 responses)

The opinion of one who has, by the sweat of his brow, produced good working code, (for example, code in wide use by those who care about robust security-hardened kernels), will always be worth a lot more than the opinions of those who snipe from the sidelines.

Perhaps you should address this to the lovely and gentle folks of the PaX fame when they again decide that it's time to talk down the whole kernel community. You might also want to remind them, that they are not the sole authors of the whole kernel, and their contribution, useful and needed as it is, is by far and large not the most important.

State of the Kernel Self Protection Project

Posted Sep 13, 2016 12:49 UTC (Tue) by PaXTeam (guest, #24616) [Link]

jubal, in general, if you have something to tell us/me, you can always address us/me directly.

> Perhaps you should address this to the lovely and gentle folks of the PaX fame when they again decide
> that it's time to talk down the whole kernel community.

first of all, you're trying to judge characters based on a very small sample of interactions which i guess speaks volumes more about yourself, than us. hint: we've worked with various kernel and other developers just fine for as long as our projects existed. you can find the evidence on lkml, bugzillas, distro lists, etc. as far as i recall, the only topic that ended up in flamewars was about high level policy decisions (about the handling of security related issues), which represents a small fraction of the interaction of all participating sides.

as for 'talking down' the whole kernel community, it's of course nonsense (present the evidence if you think otherwise). we did and do criticize people who we find do something stupid and we don't mince words about that either, not unlike certain kernel (and non-kernel) developers by the way. you just have to learn to deal with it.

> You might also want to remind them, that they are not the sole authors of the whole kernel, and their contribution,
> useful and needed as it is, is by far and large not the most important.

and why do we need to be reminded of this again? or was it just a strawman?

State of the Kernel Self Protection Project

Posted Sep 13, 2016 9:23 UTC (Tue) by paulj (subscriber, #341) [Link] (2 responses)

Not every technically capable person is adept at soft skills.

Maybe PaXTeam hasn't done the right things to secure funding. Maybe they're not capable of it. However, doesn't the fact we're discussing the work of a paid group to unpick PaXTeam's code and upstream suggest that it might be a good idea to also fund PaXTeam too?

State of the Kernel Self Protection Project

Posted Sep 13, 2016 12:40 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

> However, doesn't the fact we're discussing the work of a paid group to unpick PaXTeam's code and upstream suggest that it might be a good idea to also fund PaXTeam too?

A commercial organization is far more likely to fund developers who have a history of pushing code upstream and are not antagonistic with that community of developers. Linux kernel developers have historically not been very welcoming to contributions that harden the kernel and omitting info on vulnerabilities they know about, so it isn't surprising to see that has been a constant source of friction but here we are now.

State of the Kernel Self Protection Project

Posted Sep 13, 2016 16:09 UTC (Tue) by paulj (subscriber, #341) [Link]

Well, of course. However, that doesn't change the fact that in an ideal world some way would be found to fund the developer who did the core work despite that, given that those unpicking that work are being paid.

State of the Kernel Self Protection Project

Posted Sep 13, 2016 20:15 UTC (Tue) by flussence (guest, #85566) [Link]

>you're basically making demands on me
No he isn't. The rest of your words have a similarly loose grip on reality.

State of the Kernel Self Protection Project

Posted Sep 5, 2016 17:56 UTC (Mon) by spender (guest, #23067) [Link] (2 responses)

Regarding KASLR and the article/slides, I read them just fine. What matters here (and where the spin occurs) is not in what's said, but in what isn't. That KASLR is being extended is just more cargo cult security and a distraction. What won't ever be mentioned is that there are generic local defeats circulating in public that none of these extensions deal with whatsoever. It's a way to avoid admitting it's a nonsense mitigation to begin with, if attackers even in the most strict sandboxes can defeat it. Upstream won't do anything to deal with it either likely because it's not clear if anything can be done about it. KSPP wants the image of security progress, so you can't expect Kees to admit the uselessness of KASLR in his talk, nor could you expect him to announce himself KSPP is bumbling along, when he needs to give the impression that it's a successful effort in order to attract more free work for the Linux Foundation.

-Brad

State of the Kernel Self Protection Project

Posted Sep 10, 2016 11:21 UTC (Sat) by spender (guest, #23067) [Link] (1 responses)

Hot off the presses, another KASLR bypass:

> We implement our attacks on a real system with Haswell CPU and recent Linux kernel and show that kernel-level ASLR can be recovered in about 60 milliseconds.

http://www.cs.binghamton.edu/~dima/micro16.pdf

-Brad

State of the Kernel Self Protection Project

Posted Sep 10, 2016 11:31 UTC (Sat) by tao (subscriber, #17563) [Link]

Ahhhh, Schadenfreude, sweet Schadenfreude.

State of the Kernel Self Protection Project

Posted Sep 3, 2016 7:01 UTC (Sat) by mb (subscriber, #50428) [Link] (21 responses)

>it's all ripoffs of grsecurity/PaX

The proper way to avoid "ripoffs" would be to upstream your code by yourself.
You don't do that? Well, in the Open Source world somebody else will possibly do it then.

State of the Kernel Self Protection Project

Posted Sep 3, 2016 13:09 UTC (Sat) by PaXTeam (guest, #24616) [Link] (20 responses)

the proper way to upstream our code would be for somebody to pay for that time.

State of the Kernel Self Protection Project

Posted Sep 3, 2016 13:59 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link] (7 responses)

> the proper way to upstream our code would be for somebody to pay for that time.

That's the rub. People are getting paid to do that. It just so happens to be people who have a history of doing it.

State of the Kernel Self Protection Project

Posted Sep 3, 2016 14:32 UTC (Sat) by PaXTeam (guest, #24616) [Link] (2 responses)

no, the rub is that while they're supposedly paid to upstream our code instead they end up ripping random stuff out without understanding what it does, how it works, how it was designed, etc. the end result is various ways and levels of brokeness which among others means that i still have to patch these ripped out parts in PaX to make them work properly (it's not a new phenomenon by the way, i've had to do this ever since NX/ASLR appeared upstream). i think that's not a history one would be proud of but then they're not wasting my money at least ;).

State of the Kernel Self Protection Project

Posted Sep 3, 2016 15:04 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

> no, the rub is that while they're supposedly paid to upstream our code instead they end up ripping random stuff out

You aren't pushing for that code to be directly upstreamed. That isn't necessarily a pleasant process and it is time consuming but since you aren't interested in doing that, you do end up losing control over how it is done.

State of the Kernel Self Protection Project

Posted Sep 3, 2016 15:12 UTC (Sat) by PaXTeam (guest, #24616) [Link]

> You aren't pushing for that code to be directly upstreamed.

i guess you've successfully concluded an infinite loop now: https://lwn.net/Articles/699300/

State of the Kernel Self Protection Project

Posted Sep 3, 2016 15:36 UTC (Sat) by Lionel_Debroux (subscriber, #30014) [Link] (3 responses)

The entry barrier for _significant_ security improvements to the mainline kernel - as opposed to e.g. KASLR, repeatedly shown to be easily defeatable - is very high.

As mentioned previously on LWN, e.g. https://lwn.net/Articles/662907/ , Linus himself repeatedly refuses some of the most powerful PaX defenses. Not only the hardware protections of recent high-end x86, x86_64 and ARMv7 processors he's mentioning aren't as powerful as PaX's (partially) software protections - see e.g. https://lwn.net/Articles/617945/ - but also, MEMORY_UDEREF and KERNEXEC work on processors about a decade older, and less expensive parts, which means that they could protect the vast majority of existing computers, rather than a tiny minority.
Emese Revfy once spent time, probably a significant amount thereof, making a large patch series constifying lots of needlessly writable, and therefore ripe for abuse, mostly "ops" structures in the kernel. The fact is that relatively few maintainers picked up the pieces.

Fortunately, the number of distros packaging grsec kernels is growing, and programs are slowly being improved / fixed - depending on one's POV - to work on PaX/grsec kernels (e.g. Docker wrt. grsec chroot hardening), which means that it's getting easier for users to get most PaX/grsec benefits. RANDSTRUCT is nullified by shared builds, but the other benefits, spanning a wide range of defenses, remain.

State of the Kernel Self Protection Project

Posted Sep 3, 2016 15:49 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link] (2 responses)

> The entry barrier for _significant_ security improvements to the mainline kernel - as opposed to e.g. KASLR, repeatedly shown to be easily defeatable - is very high.

True but the politics here is unhelpful. The developers involved are not exactly pleasant folks to work with.

State of the Kernel Self Protection Project

Posted Sep 3, 2016 16:25 UTC (Sat) by spender (guest, #23067) [Link] (1 responses)

Are we talking about Linus and Greg KH?

-Brad

State of the Kernel Self Protection Project

Posted Sep 4, 2016 0:59 UTC (Sun) by rahulsundaram (subscriber, #21946) [Link]

> Are we talking about Linus and Greg KH?

On either sides.

State of the Kernel Self Protection Project

Posted Sep 4, 2016 15:14 UTC (Sun) by mina86 (guest, #68442) [Link] (11 responses)

> the proper way to upstream our code would be for somebody to pay for that time.

It's free software, anyone can upstream it in any way they want regardless of being paid or not.

State of the Kernel Self Protection Project

Posted Sep 4, 2016 16:10 UTC (Sun) by PaXTeam (guest, #24616) [Link] (10 responses)

you're wrong, not anyone can upstream this much code without getting paid, you must be an early retiree with a few millions in your bank account to be able to work the thousands of hours 'for free'. but hey, prove me wrong, give it a try yourself and report back on your progress in a few years (or more realistically, never ;).

State of the Kernel Self Protection Project

Posted Sep 4, 2016 18:01 UTC (Sun) by ssmith32 (subscriber, #72404) [Link] (1 responses)

Confused. Is upstreaming the code without demanding payment such a unique and valuble ability that it should be respected and applauded, or so easy and simple that anyone could do it? You seem to be taking both positions?

State of the Kernel Self Protection Project

Posted Sep 4, 2016 18:42 UTC (Sun) by PaXTeam (guest, #24616) [Link]

you're confused indeed as i took neither of those positions.

State of the Kernel Self Protection Project

Posted Sep 5, 2016 0:43 UTC (Mon) by mina86 (guest, #68442) [Link] (7 responses)

I’m not commenting on practicalities.

But notice that in the past, people developing free software in their free time without being paid for it was the norm. To this day this in not out of ordinary. IIRC unaffiliated contributors are still single group involved in Linux.

Lastly, as far as I understand, a lot of the hard work has already been done (and PaX and grsecurity teams are to be thanked for that) so now pushing the code upstream is the easiest part.

There may be political reasons why it’s hard for the PaX and grsecurity teams to do that, but complaining about someone who puts up with Linus et al. to get the code into Linux is just childish.

State of the Kernel Self Protection Project

Posted Sep 5, 2016 17:39 UTC (Mon) by PaXTeam (guest, #24616) [Link] (5 responses)

i think you simply have no experience with upstreaming code of this amount and complexity and so don't understand the amount of effort needed for it. the thousands of hours i mentioned wasn't hyperbole, it's my estimate based on observing past progress (and often lack thereof) on various attempts. of course you can call me names all you want but that doesn't change the physical reality in which i simply don't have those thousands of hours of free time to spend (dare i say, waste) on upstreaming this code and clearly noone else does as all the real progress so far was made on company time.

State of the Kernel Self Protection Project

Posted Sep 5, 2016 18:16 UTC (Mon) by mina86 (guest, #68442) [Link] (4 responses)

I don’t understand your point then:

> the proper way to upstream our code would be for somebody to pay for that time.

This sounds to me that you’re claiming current upstreaming efforts are not proper because no one is paying for the time. Is that what you’re saying?

State of the Kernel Self Protection Project

Posted Sep 5, 2016 19:15 UTC (Mon) by PaXTeam (guest, #24616) [Link] (3 responses)

perhaps try not to take stuff out of context and also read the comment i replied that to :).

State of the Kernel Self Protection Project

Posted Sep 5, 2016 21:11 UTC (Mon) by nix (subscriber, #2304) [Link] (2 responses)

So... reading that, your complaint is that nobody is paying *you* in particular to upstream it?

This is sounding less and less high-minded by the minute.

State of the Kernel Self Protection Project

Posted Sep 5, 2016 21:15 UTC (Mon) by PaXTeam (guest, #24616) [Link]

not quite, try reading what i responded to again.

State of the Kernel Self Protection Project

Posted Sep 6, 2016 10:13 UTC (Tue) by paulj (subscriber, #341) [Link]

To be fair, even if "No one paid me to do it" is what PaX intended as a criticism, that's still quite acceptable. They spent a lot of effort on that work, and really it would be good if those who work on Linux get something towards food, shelter, etc. You can understand why he'd be miffed if others are now being supported to pick at their work and upstream bits.

E.g., every seems to be agreed PaX et al have done some good work on security. Wouldn't it be nice if some of the money going into hardening the kernel helped that work?

The question is whether the inter-personal issues at play can be overcome to make that possible. Whether PaX et al can co-operate, tackle the nits and implement review comments (which sometimes do no more than help give the reviewers a shared sense of participation and value - but we socio-insensitive techies often don't perceive the soft, socio-politics, sigh), etc. Whether the other side(s) could get over their hostility, etc. ?

Shame really for everyone, inc. the users.

State of the Kernel Self Protection Project

Posted Sep 9, 2016 14:13 UTC (Fri) by thestinger (guest, #91827) [Link]

> IIRC unaffiliated contributors are still single group involved in Linux

Lack of known affiliation doesn't mean they aren't being paid to do the work. It can still be part of their full-time work or a contract. They may also be self-employed in a way that they earn money from landing code upstream. A subset are certainly volunteers but there aren't meaningful numbers on that.

State of the Kernel Self Protection Project

Posted Sep 4, 2016 15:12 UTC (Sun) by mina86 (guest, #68442) [Link] (2 responses)

> it's all ripoffs of grsecurity/PaX

I think you're confused about what free software is.

State of the Kernel Self Protection Project

Posted Sep 5, 2016 5:20 UTC (Mon) by citypw (guest, #82661) [Link]

Free software will be ok with giving the credits to those who deserves. PaX/Grsecurity has been contributing significant mitigation technology for linux kernel and meanwhile the whole FLOSS community gets benefit from it. I couldn't imagine the world without PaX/Grsecurity especially in those 0ld horrible days. KSPP is a very good starting point to improve vanilla kernel security. But I have to admit that what most work( until now) of KSPP is try to re-build( development from scratch or backport some features from PaX/Grsecurity directly) what PaX/Grsecurity has already done for years:

https://github.com/hardenedlinux/grsecurity-101-tutorials...

Linux foundation may not care about real sense of security. The "work", which can do PR might become the benchmark to them;-)

State of the Kernel Self Protection Project

Posted Sep 9, 2016 14:15 UTC (Fri) by thestinger (guest, #91827) [Link]

You can't strip copyright notices from GPL software, or substantially use the code as a reference without leaving the copyright notice intact.