An important set of stable kernel updates

That's the problem I think -- if there are all these voices that agree, I certainly haven't heard them, and definitely not on this site. You can go back to around 2005 or so I think when we first started pressing the issue about the disconnect between the "full disclosure" policy mentioned in Documentation/SecurityBugs vs the obfuscation practices of mainly Linus and Greg (which then became an example for other upstream developers) after Chris Wright stopped being involved in the stable kernels. I don't know how to explain the responses we got to what we were saying at the time, people were incredulous, kept coming up with excuse theories of their own as to why this information wasn't being provided instead of the facts staring them in the face, or resorting to some strawman about how we were demanding info on vulns upstream knows nothing about, and how that would destroy development speed. You can see one example of that in the link I posted actually, and the same stuff was repeated over and over again. At some point, after Linus admitted in public he's intentionally obfuscating commit messages, people needed some way to deal with the cognitive dissonance, but instead of admitting being wrong, they bought into this naive "a bug is a bug" mantra to justify the behavior. Most others simply didn't speak up at all, minus a handful of people I recall every now and then -- anyone remotely involved in upstream work (I recall Willy and Eugene) basically shut up about it after Linus and Greg repeated their views a few more times in an authoritative way (or maybe they got tired of complaining to a brick wall). I think with people not having the courage to stand up to Greg and others who in nearly every facet are very much about asserting the status quo (Linus' comments on security at every conference in the past decade or so are basically a repeating of the same few sentences), upstream becomes convinced their view is right, that the suppressed dissent is tacit assent.

-Brad

Well, is there any point in talking to Linus, and his like-minded friends, until they fix their counter-productive views of security ? The arguments told years ago by PaXTeam, spender and others from their side still stand.

Let's examine the mainlining of the five most powerful defenses from PaX/grsecurity against kernel exploitation, which stop most PoCs discussed time and again on LWN and elsewhere, at every step of their execution:
* KERNEXEC (over a decade old, preventing the kernel from blissfully executing payloads in user-space): principle refused by Linus and friends because "the hardware does it". Right, but only for a minority of very recent hardware from the 2010s featuring x86/x86_64 SMEP or ARM/AArch64 PXN, as opposed to protecting most CPUs from this millenium (and not just on x86/x86_64 or ARM/AArch64) ! Plus the GCC plugin infrastructure it uses was only integrated in 4.8, several months ago.
* MEMORY_UDEREF (over a decade old, preventing the kernel from blissfully dereferencing pointers to payloads in user-space): ditto, with "the hardware does it" being even more of a lie, as x86/x86_64 SMAP and ARM PAN are even scarcer in the real world than SMEP / PXN. And there have already been easy SMAP bypasses in mainline;
* RAP (powerful anti-ROP measure): no way, because per-CPU PGDs, another PaX change on which RAP is based, is refused, too;
* CONSTIFY (making const all those needlessly writable "ops" structs, often overwritten by PoCs as a first step to kernel exec): not refused, it's just that nobody seriously does it in mainline (years ago, I contributed several patches extracted from PaX for manually constifying structs, before they switched to a GCC plugin; the situation hasn't changed much);
* RANDSTRUCT (foiling binary exploits by randomizing struct layout): possible now that the GCC plugin infrastructure is in. But the real-world protection brought by this feature is limited, given that pretty much everybody is using publicly downloadable distro kernels, nullifying the effect of RANDSTRUCT.

The three most powerful defenses against kernel exploits are flat out refused. Until this counter-productive stance is reversed, the KSPP efforts will remain of very limited usefulness - and trickling in at a slow pace, what's more: look at how little got in mainline over the past year.
In the meantime, I'm keeping grsec kernels (which are being packaged by a growing number of distros, for very good reason !) on my own computers, using them on a growing number of computers at work, and slowly educating my workmates on how powerful grsecurity is / how lackluster mainline Linux security is / how it is hardly improving.

> Go do a service to the community (as this site does) by explaining how the bug works. Publish things. By all means, talk about it. Set up a service which tries to assess each bug fix to the Linux kernel.

Because the lead developer of the most comprehensive security-related Kernel patchset with an amazing track record doesn't add anything to the community? Wtf? His arguments are well known but fail on deaf ears. And one person only can do so much, Spender clearly is on the upper end of that scale or beyond...

If it was just splurgy nonsense then you would have a point. Sometimes important things can't fit into the space of a twitter post and this is one of those cases.

No, what I'm saying is "if I'm in the line of work of providing 45K instances of essentially the same system, it behoves me to plan for unexpected problems coming along that could impact all of them at once."

The details of that "unexpected problem" don't actually matter. You only know that *something* will go wrong, and try to architect mitigations into your systems and processes to handle it.

As already mentioned elsewhere, if you find the current status quo so unacceptable, why weren't you already running grsecurity/PaX/etc on those 45K instances? But even grsecurtiy/etc have flaws, to say nothing of the actual hardware it's all running on..

> an avoidable wide scale security issue like this

How was this avoidable? I thought even Grsecurity was vulnerable to this bug?

Unfortunately, the other *nix options aren't that good when it comes to security either. spender has been lambasting them for years as well, especially OpenBSD, which prides itself about security and tries to make people believe it's so great, but isn't...

Let's just look at ASLR. It was invented by the PaX Team in 2001. ASLR broke a number of direct exploits based on predictable memory addresses. Even nowadays, it still raises the cost for attackers, who have to use infoleaks to get around it, so I'd say it's still an effective component of defense in depth. PaX has a plugin for sanitizing the kernel stack, as well as better allocated / freed memory sanitization, thereby removing a whole class of infoleaks, BTW.
* Linux grew PIE support in 2003 and other ASLR aspects in 2005. Even KASLR in 2014, but that barely provides any security in return for its complexity, as predicted by PaXTeam and spender from the get go, and as shown by the many KASLR bypasses of the past two years;
* OpenBSD implemented some ASLR features in 2003, but didn't implement PIE until 2008;
* NetBSD didn't enable ASLR by default until 2016 (!);
* FreeBSD still doesn't have ASLR because some of its maintainers refuse it because "it's not really effective". The HardenedBSD derivative has it, for good reason, though;
* for completeness, though it's not an option for your 45K Linux servers (cost): Solaris recently grew ASLR support, but it wasn't there in the discontinued OpenSolaris, and AFAICT, its derivatives don't have it;
* not an option either for you (special hardware requirements, cost): MacOS X (and iOS) got ASLR after Linux did.

In 2016, OpenBSD was trounced by kernel fuzzers, just like Linux, finding easy DoS and LPE.
OpenBSD reinvents a subset of PaX/grsecurity's wheels in NIH mode: look at what the PaX Team and spender wrote and said about W^X.

Overall, deploying grsec-patched Linux kernels with MEMORY_UDEREF, KERNEXEC, RAP, CONSTIFY, RANDSTRUCT and the dozens of PaX/grsec goodies onto some of your many servers is pretty much the best you can do to protect them. None of the other *nix options come anywhere close to the level of protection offered by grsec. Obviously, there will be a performance hit.
You should also consider buying (having management buy, I mean...) commercial support from grsecurity. For a corporation which has 45K Linux servers, it's a tiny drop in the ocean. All the more you'll save money (besides saving your sleep and sanity) every time you don't _have_ to reboot your servers *absolutely right now* (but can choose to do so tomorrow or on Monday) to fix yet another critical issue, thanks to the PaX/grsecurity defenses making an issue completely unexploitable, e.g. because the corresponding ops structures targeted by the exploit were constified, because the refcount overflow protection kicks in, or for other reasons.

If you find the kernel development practices are unacceptable to you and also believe that it is not possible to take your business elsewhere then you, by your own admission, have no business continuity plan.

That's an unpleasant situation to be in. Good luck in finding a path to a better place.

Or you could look honestly at why it might be so difficult to find an alternative. And look at how changing kernel development practices might have negative impacts on your business that, in the long-term, outweigh the admittedly high short-term impact of this 'fire-drill'.

Look at the multi-billion value of the Linux kernel [1], which is likely why you find it difficult to find an alternative. Linus Torvalds, so far (25 years), has led a coalition of divergent interests to invest that value. He has demonstrated unswerving commitments to users of the kernel, with all their conflicting interests (even yours). Examples inlcude:
- his insistence on maintaining the external API and ABI of the kernel, even at the expense of developers' short-term interests; and
- his willingness to take on ideas, functionality and code for things/areas that he has no personal interest in;
- his insistence that all bugs, whether functional, performance or security are important (and that many apparently non-security related bugs turn out to have security implications).

Even Microsoft, who have vastly improved their development practices, and yet still release security patches every month, often for issues that already are being exploited in the wild. Large organisations that pay serious money to Microsoft get hit by this. And still call out their IT staff to patch thousands of servers, risking that line-of-business applications will stop working.

[1] from 2008 - https://www.linuxfoundation.org/news-media/announcements/...